Network May Be Monitored By A Third Party (a.k.a. Android KitKat scary warning message part 3)

I guess people care about this annoying warning message that Android has thrown on us.   No other blog post here has generated anywhere near the number of hits that this issue has generated.  So, to all of your that have found my blog and read what I have to say, I thank you!

Unfortunately, Google doesn't seem to care about this issue.   (Newsflash!  Google doesn't care about making certificates easy to use on Android.   And, by extension, they don't care to make wireless authentication on Android usable either!   I could write volumes on my experiences trying to get things improved.  But, I fear that it would inspire depression in anyone that read it.)   So, we need to do something to try to GET them to care!

There have been several bugs opened about this problem, and most of those bugs have been closed with various reasons that make no sense.  (Which is what I base my claim that they don't care on.)  Some of the other people on the Internet that agree with me that this is a problem have continued to open new bugs to continue to push the problem and make Google listen.   If you care about this problem, PLEASE go to the Google bug tracker, and at least star the issue.   If you have the time, you might consider posting a comment to the bug that outlines why this issue causes you grief, and what changes you would like to see made in order to improve the situation.   (If you are only going to complain, do us all a favor and just star the issue.   If you want to complain, and offer suggestions on how to improve things, please post a comment!)

I'll even make it easy for you.   The URL (with link!) to the current version of the bug is at : https://code.google.com/p/android/issues/detail?id=82036 .


As you make your feelings known, please consider this.   There is actual value in having a warning like this, if you are the kind of person that values privacy and is against network operators using man-in-the-middle tactics to monitor what you are doing.   And, thanks to stupid legislation here in the US, if you are attending a K-12 school that provides Internet access, you SHOULD care about this very much!   US law requires the schools to snoop on students as a condition to continue to receive funding!    I'm lazy, so I am not going to track down the exact law that requires this, but I know it is true because of my day job.   (Not going to say any more than that because my opinion might get me in trouble.)

Because I would prefer that people being snooped on are given any warning that it could be happening, I am actually in favor of the warning message that was added.   However, I have problems with how it was implemented.   I think there are a few tweaks that could be made to the implementation that would take this from being a pointlessly scary warning message to a valuable warning message, and an opportunity to educate at least a few people on some of the security issues that are associated with installing new root CA certificates on any device or OS.   Also, as I have stated before, I don't like people that complain about a problem without offering solutions!

So, here are my suggestions on how to transform the annoying mess that is the "scary certificate warning message" in to something useful and significantly less annoying :

• Don't show the message (ever) if the certificate is used with 802.1X on a Wi-Fi connection -- Starting with Android 4.3, the "Credential Store" (a.k.a.  Certificate store, or key store) has been split in to two parts.  One part holds the certificates that will be used for authentication on Wi-Fi networks.   The other part holds certificates that are used for VPN connections, web site connections, and everything else.   Since you *HAVE* to install a certificate in order to securely use a Wi-Fi connection, showing this warning message is pointless.  The other alternative is not to use a certificate to validate the Wi-Fi network, which leaves you WIDE open to anyone doing a man-in-the-middle.   So, the scary warning actually has the potential to make users LESS secure by making them remove the certificates necessary to secure their networks!   Google must recognize this on some level, because they don't show the warning if an app uses the Wifi API to install certificates.
• When you tap the warning, give useful information --  Right now, when you tap the warning, you are taken to the security settings screen.   This is COMPLETELY useless.   The user is expecting to be given more information on what this warning is all about.   Take them to a screen that explains what the potential issues are with installing a new CA certificate.   Explain clearly what the problem is, and don't try to scare them in to submission.  Try to give the user enough information to allow them to make a good decision about leaving the certificate installed.   Also, provide links to more detailed information so that those that are really curious can really dig in and understand the problems.
• Allow users to get rid of the warning -- Right now, you can dismiss the warning, but it comes back any time you install another CA cert, or anytime you reboot your device.   This behavior is annoying.   Tell me once, let me decide what to do, and then leave me alone!  Anything else is a bad user experience.   When the user taps the warning to get more information about what it is about, they should also be given two check boxes.   One check box should allow them to dismiss the warning just for the current certificate, the other should allow them to block ever showing it again.   There really are two types of users where this warning is concerned.  There are those that already understand the risks, and those that don't care.   The purpose of the warning to to try to get some of those that don't care to actually start to care.  But, annoying the crap out of them is only going to make them hate the issue so much that they will never care.

If my three suggestions above were implemented, I suspect most people would no longer hate the warning message as much as they do now.   It also gives the power back to the user, which is what I always thought Android was supposed to be about.

I understand that in addition to warning users that they may be causing themselves problems, they are probably also trying to cover their butts should a bunch of users get hacked and complain in the media, or worse, try to sue.   However, I really believe that showing the warning once is more than enough for Google to cover their butts on this.

In the unlikely event that someone from Google reads this, and wants me to put my money where my mouth is, I will offer to submit patches to the AOSP project to implement all of the things above.  HOWEVER, given my previous experiences with submitting patches to AOSP, I will be expecting a promise that the code will eventually be included.   By "eventually", I mean that someone from Google will be assigned to review my code, and comment on it until the code is in the right state to be fully included in AOSP and eventually the main Android builds.   (As an aside, while I have tried not to give out any information on this blog about who I am, I am sure that Google has the ability to look up my e-mail address based on this blog being hosted on Blogger.   There is also someone on the Android dev team that knows me, and someone on the Android security team that knows me.  So, it should be easy enough to verify that I know what I am talking about, and it should be easy to reach out to me.)

How does this wireless authentication stuff work anyway?

My last couple of posts have been calling out some of the issues with wireless authentication on Android.   But, in my day job I deal with authentication on the most common operating systems you would see in the wild.  If you were only to read my blog posts this far, you would probably believe that Android is the worst device for wireless authentication there is.   The truth is, every operating system I have come across does something stupid when it comes to wireless authentication.

But, perhaps the worst component of authentication on wireless networks is the complete lack of understanding that most people have of how it works.   Now, you can argue that you don't need to know how your engine works in order to drive a car, so you don't really need to know how wireless authentication works in order to use your computer.  And, in general, I would agree with you.   However, most people would agree with me that knowing more about how their engine works results in an engine that generally works better because those people know what to look for and what types of behaviors can cause them trouble down the line.   With wireless authentication, it is also the same thing.   However, if you destroy the engine on your car, you will be out a lot of money to fix it or replace the car.   If you screw up with your authentication the damage can be far worse, and far more expensive.   So, do yourself a favor and learn a bit more about how the authentication on your wireless networks work.

My goal in this blog entry is to try to boil down the authentication process in to plain English, while mapping some of the technical jargon in to something that an average person can understand.  As a result, this blog entry will be very long.  But, hang in there with me.  I think it will be worth it.

We are going to look specifically at user name and password based authentication on networks.  This type of authentication is commonly used on enterprise wireless networks.  However, it can also be used on wired networks.  In fact, the method of authentication used in these networks was designed originally to be used on wired networks.  It was adapted to work with wireless networks, and that is where it seems to have found the most success.  This standard is called IEEE 802.1X.  Some people may now be yelling at their screen, "Nuh uh!  WPA-Enterprise or WPA2-Enterprise is used on wireless networks, not this 802.1X stuff!"  Those people are both right and wrong at the same time.   The WPA specifications outline how to use the encryption keys that are generated during an authentication with 802.1X.   So, we will be focusing on 802.1X, and leaving the encryption key stuff for another time.

There are two ways that an authentication starts.   One is that the network recognizes that you have connected to it and sends a request to start the authentication.  The other is the client machine (i.e. your laptop or phone) sends a message to the network requesting that it start the authentication process.  Both methods result in the network sending an identity request to the client station.   The station responds to this request with a message that contains some form of identification.   Because of the way that 802.1X has evolved, this identification is often referred to as the "outer identity" or "anonymous identity".  This terminology is technically questionable, but seems to be what the industry has settled on.  Probably because the "proper" terminology would just be confusing to anyone that doesn't understand how the various authentication methods works.   So, we will use the "outer identity" or "anonymous identity" to reference this through the rest of this blog post.

Following this identity request, the network will send a challenge message to start the authentication.   If you have ever configured authentication for a wireless network, and seen the choices such as (EAP-)PEAP, (EAP-)TTLS, (EAP-)TLS, this is where those settings come in to play.  The challenge message that the network sends includes an identifier that tells the client what type of authentication the network wants to use.  If the client is configured to allow that type of authentication, the process proceeds.   If not, then the client will respond to the network asking for a different type of authentication.   If the network allows the type of authentication the client requests then it sends another challenge, this time identifying the authentication method requested by the client.

At this point, the details of the methods used for authentication diverge wildly depending on which authentication system you use.   But, the basic ideas are all the same.   So, rather than dive in to the details of the various authentication protocols, we will discuss what the goals of the authentication are, and then work backwards.

The network, the client, and the user each have specific goals when they enter the authentication.  Those goals are :

• The networks wants to make sure the client is someone that is allowed to use the network.
• The client wants to make sure that it doesn't provide its credentials to anyone other than the networks that it believes it can trust.
• The user just wants to get on the network, everything else be damned.

So basically, the client and the network want to make sure that both the network and the user are protected.  But, as usual, the user is the weak link as they will probably do just about anything to gain access to the network.  (Facebook statuses won't update themselves!)     This is one of the biggest failures in the design of the various authentication protocols.   For methods such as PEAP or TTLS, the user has the ability to gut the security of the connection as an easy way of gaining access to the network.   

So, if we ignore the desires of the user and focus on the security aspects of the authentication, how do the goals of the network and the client get met?

The first step is for the network to identify itself to the client.   Currently, this is done using a word that strikes terror in to most network administrators.  "Certificates".   Certificates really aren't as scary as most admins think they are, but that is a topic for another blog entry.   For the purposes of this discussion, the network provides a certificate to the client that the client can run some checks on to verify that the network is who it claims to be.  (Side note : There are a lot of potential security issues with certificates and this type of authentication.  But, that is beyond the scope of this post.)   If the client decides it doesn't trust the certificate it will either send a message to the network indicating it doesn't like it, or simply disconnect from the network.   Assuming it does trust the certificate, various pieces of information are used to generate cartographic keys that allow all of the future messages to be encrypted when they cross the network.  This layer of encryption is usually referred to as the "inner tunnel" or sometimes just "the tunnel".

At this point, if the client believes it can trust the network, it will fulfill the desires of the network by proving it should be allowed access to the network.   This is usually done by using a second authentication method "inside the tunnel".   This portion of the authentication is known in most documentation as "phase 2" or "the inner phase".   At this point of the authentication, the client sends the user name and password back to the server using the encryption keys that were generated by the certificate exchange.   It is believed that the encryption set up by the certificate exchange is pretty secure, since it is basically the same type of encryption that is used when you purchase something from a secure web site.   

Taking a quick side trip back to the beginning of the authentication, it should now be more clear why the initial identity the network requests is called the "outer identity".   However, it isn't any more clear why it might be called the "anonymous identity".   The reason it can be called that is because the specifications for PEAP and TTLS explicitly state that the first identity request can be answered with an identity of "anonymous" because the users actual user name will be passed to the network using encryption later in the process.   The idea is that by returning "anonymous" for the first identity it makes it a little harder for a bad guy to use that identity to target a specific user, since the first identity is sent over the network with no encryption.  (Side note : The first identity isn't always sent unencrypted.   There are situations where it is encrypted, but the working assumption in the standards is that it is unencrypted.)

Once the user name and password are sent to the network, the network can decide if it wants to allow the client to access the network.  If the client is allowed, then the network sends a success message and opens the network up to be used.   If the client is not allowed, then the network sends a failure message and blocks the client from using the network.

While I don't want to get in to many of the security aspects of this type of authentication, I do want to point out the weakest link.   There is an assumption that the client will validate the certificate the server provides to it.   There are a lot of places that can go wrong, but by far the biggest one is the user disabling the checking of that certificate at all.   Sadly, some devices like the Nook tablets and Windows RT both default to not checking the certificate, or in the case of the Nook doesn't even allow the user to configure checking the certificate!  (Note : I have not looked at the latest Nooks, so it is possible this issue has been resolved.)  If the client doesn't verify the certificate, then the client will send the user name and password to any device that pretends to be the wireless network the user wants to connect to.   Depending on the authentication method used, this could mean that the password is sent to the bad guy with no encryption, or only lightly hashed.

Hopefully someone finds this post useful.   802.1X authentication can quickly go from something that seems simple enough to a wildly complex set of decisions with potentially huge negative consequences.   In the coming months, I plan to go in to more of a "deep dive" on the different authentication methods and what the known risks are with each of those methods.

Scary Warning Message on KitKat Part 2

Update 1/26/2015 - Google has refused to take this issue seriously.  They have closed several bugs posted on the Android bug tracker.   Fortunately, there are some other people out there that agree that the current implementation is broken.   Unfortunately, Google doesn't seem to think that it is worth fixing.   Since I know a lot of people find this page while looking for information on this annoying warning, I would encourage you to go to the current bug on the Android bug tracker, and star it to show Google that you want this resolved!   Perhaps if enough of us star it, Google will start to pay attention.   You can find the current open bug here.

It always amazes me which posts of mine tend to generate the most traffic.   Not that my blog gets a ton of traffic, but once in a while I get spikes of traffic to a specific post and it surprises me a little.   My previous post on KitKat seems to be one of those that has caused a bit of a spike.   So, I figured a follow up might be interesting.

A couple of hours ago, a coworker sent me a link to Android issue 62076.   Gotta say, I was quite surprised to see a link to my blog as supporting evidence!   I was equally surprised to see that the bug was marked "Works as Intended" and closed.   I read through the statements on how it is intended to work, and came to the conclusion that if it does indeed work that way, this may not be as big a deal as I thought.   But, what was being claimed didn't seem to match up with my experience using KitKat.

For those who don't want to sift through the bug chatter, here is a snip of the response from one of the Android team members :

The "User" portion of the trusted credential store is non-system CA certificates that have been installed and are trusted by the browser and other things that use the system Trusted Certificate Store. This warning is about protecting the user of the device.

Note that EAP-TLS and other Wi-Fi modes do not need to install a CA certificate to the Trusted Certificate Store. If you include your CA certificate in a PKCS#12 bundle when installing the credentials and select "Wi-Fi" as the destination for those credentials, you will not get this warning. You can also create a program that adds Wi-Fi credentials and configurations programmatically using the new WifiEnterpriseConfig API (see http://developer.android.com/reference/android/net/wifi/WifiEnterpriseConfig.html).

Asking for exceptions does not make sense. There is a process for getting a CA into the trusted list of browsers and operating systems. Please see http://www.mozilla.org/projects/security/certs/policy/ for an example of the process that needs to be followed.

This all seems pretty reasonable, although arguing that you can write an app to install certificates in a way that doesn't cause scary messages seems to be a bit of a stretch for most people.   But, lets run with the claim for the PKCS#12 bundle for installing the certificates for a second.

Lets start with a little test.   First, go download a root CA certificate from a known CA provider.  You might need to search around a bit to find one, or you can just use the GeoTrust certificate that I will be using as I run through this test.   I used the first certificate on the list, which is the "Equifax Secure Certificate Authority" certificate.   Next, lets convert that certificate to a PKCS#12 bundle so that we can attempt to install it.   You can either try to figure out the magical incantations to do it using OpenSSL, or you could just hop over to the SSL Shopper.com converter that I will be using.

Whoops.  You mean we have to have a private key to create the bundle that we need to install a root CA in a way that doesn't trigger that scary warning message?!   Awww, c'mon!

A little bit of searching will quickly turn up that a PKCS#12 bundle is intended to store a user certificate and any supporting CA certificates it may need.   Further, even if you could get a PKCS#12 file with just the CA certificates in it, the code in the Android certificate installer would prevent you from using it.  (I have spent FAR too much time digging around in that code.  So you may have to trust me on this one.)

Okay, so lets assume for a second that we are using an EAP-TLS network, and so we can create a PKCS#12 bundle that contains a user certificate and private key.   According to the statement from the bug, we should be able to do this, and as long as we select the "Wi-Fi" store, we should be good to go.   As I happened to have a small home grown CA set up on a Linux box, I tried this method.   After going through the install, I still saw the scary icon in the shade.   But, how could that be?   If the bug was closed because it works as intended, and someone with an @android.com e-mail address says the way it is intended to work shouldn't be showing that scary icon, we must have done something wrong.   Did I maybe forget to select "Wi-Fi" for the target store?   So, I decided to try again.   The first thing I would do is go an clear the "Credential Store" by going to Settings->Security->Clear Credentials.   Then, I tried it again, and again got the icon for the scary warning.   But this time, I made extra sure that I had selected the "Wi-Fi" store like the bug resolution said to.  Interesting...

While poking around with KitKat, I remembered coming across another location that I could use to install certificates.   If you go to Settings->WiFi, select the menu button at the bottom and select "Advanced" there is another option in that menu to install certificates.   But, before we do that, we need to clear out the certificates we had already installed.  So, we go to Settings->Security->Clear Credentials, but the option to clear the credentials isn't available.   Does this mean that the certificates weren't installed?  If you tap on "Trusted Credentials" and then select the "User" tab, you would find that the certificates you had installed are actually listed.   So, either the "Clear Credentials" option has a bug that prevents it from being used when a user has installed certificates to the "Wi-Fi" store, or the certificates are installed someplace other than the "normal" user key store.   The response to the bug indicated they were installed elsewhere, so lets assume that is the case.  (A quick aside.  They are actually stored in the same place as they always were, but they are flagged with a different user ID.   So, it could easily be argued that we are dealing with a bug.  Exactly what the bug is would depend on your view of how things should work, but the fact that historically using "Clear Credentials" has cleared all of the certificates that were installed on the "User" tab would seem to indicate that the same should continue to be true.)

Okay, so the only way to get rid of them appears to be to tap each certificate listed, scroll down and tap the "Remove" button.   Fair enough, lets do that.   Now, we are ready to install certificates using the installed option located in the advanced section of the WiFi settings.   Doing so acts almost the same as installing them through the security menu.  The only difference is that you are not given the option to install to the "Wi-Fi" store.   So, I guess we have to assume that it is going there by default when you install this way.  (Another aside, testing this on the emulator shows that it does go in to the "Wi-Fi store".)

But, look at that, the scary warning is showing up again!

Well crap.   Now what?   Lets try the other option outlined.   Let's write our own code.  But first, lets take a look at this bug.   Uh oh.   Looks like that isn't an option either.   And I won't even bother to point out that the API doesn't allow for certificate chaining.   So, even if it DID work, most people would be outta luck using it anyway.

So there you have it.  This is not a bug.  It works as intended.   Which means that you have *NO* way to access an authenticated wireless network without seeing a scary warning that someone might be monitoring your traffic.


But, before I go, let me relate a story that happened to me a few years ago that, while tangentially related seems like it makes a good warning.   Several years ago, I was on a conference call with a large mobile phone provider from the UK, and higher-ups at a phone manufacturer that had significant market share at the time.  I'm leaving out names because I suspect neither party would be overly happy to be identified publicly.  During this call, the mobile phone provider was asking the manufacturer for APIs to support a product they wanted to deploy.   No matter how they asked, or how they threatened, the manufacturer kept telling the provider that they didn't need those APIs because the product they wanted to use would be obsoleted by something they were working on.   After getting off the call, I talked to my boss (who was also on the call) and said, "Ignoring what your customers want seems to be a good way to kill a company."   In the years since that call, the manufacturer ignored more of what its customers wanted, and its market share has declined to a point that is shameful.   It doesn't matter how big your company is, or how your market share is today, if you ignore what your customers want, you will eventually die.

Since there seems to be some interest in how the wireless authentication all fits together on Android, I plan to post additional information here.   If you want to know far more about the certificate internals of Android than any reasonable person would, you might want to check back.

Android KitKat : Network may be monitored by an unknown third party

Update 1/26/2015 - Google has refused to take this issue seriously.  They have closed several bugs posted on the Android bug tracker.   Fortunately, there are some other people out there that agree that the current implementation is broken.   Unfortunately, Google doesn't seem to think that it is worth fixing.   Since I know a lot of people find this page while looking for information on this annoying warning, I would encourage you to go to the current bug on the Android bug tracker, and star it to show Google that you want this resolved!   Perhaps if enough of us star it, Google will start to pay attention.   You can find the current open bug here.

Let me start by making the disclaimer that these observations are based on an AOSP build of Android KitKat on a Galaxy Nexus.   (My Nexus 5 isn't due for two more days.)   However, things like this that exist in AOSP tend to also exist in the release builds.

Along with all of the great new features in KitKat, Google has introduced what is probably the WORST security hole possible.   A ham-fisted implementation of certificate pinning.

Certificate pinning itself is a good idea.   It verifies that when you visit a web site, it provides you with the same certificate every time.   In general, certificates shouldn't be changing on web sites, and if one does you should be made aware of it.

So, how could certificate pinning be a security hole then?   By creating a situation where a harmless certificate creates a scary, and unnecessary, warning message.  When you install any certificate in to the key store, you get a warning icon in the bar at the top of the screen.   Pulling down the shade presents the following screen :

Let's consider the average user at this point.   Given the revelations about all of the snooping by various governments, it seems that a warning like this would make uninformed users very concerned.   (And, lets face it, there are very few informed users when it comes to 802.1X on Android.)   But, okay, maybe I am freaking out over nothing.  What happens when we tap the warning?

Hmm... Maybe not.   So, the assumption that an uninformed user is to draw based on this is that having any form of third party "trusted credential" installed means that a 3rd party will probably be monitoring my traffic.

I'm going to skip over the obvious irony here that something from Google is warning me that a 3rd party might be monitoring the web sites I visit, and reading my e-mail.  (If the irony is lost on you, you may want to do some research.)

Google seems to be trying to argue that the only safe type of certificate is one that comes pre-installed on your device.   Which is a downright silly argument no matter how you slice it.   But, lets go ahead and let that one slide.   Being the security minded individual that I am, I make sure that all of my network connections are as secure as possible.   So, I make the (probably bad) assumption that purchasing a certificate for my RADIUS server from a public CA will provide me what I need in order to have a secure wireless network.

Once I get the network setup, I try to connect my Android device to the network.   Now, being that we have paid even a little attention to the security issues around wireless networks, we know that we need to validate the server certificate in order to have a secure connection.   No problem, we purchased from a public CA, so we will just select that in the configuration settings.   But, you can't.   The pre-installed certificates on Android can't be used with 802.1X authentication.   Okay, no problem.  I'll just install the CA certificate on my device and then use that.   Oh, what is this scary message?

Now, those of us that understand the meaning of this message will just dismiss it.   But, let's assume that you aren't a techno savvy individual that has enough time to spend learning about security.   This warning is going to freak you out!   If this conversation hasn't happened on a message board yet, it will soon :

"Hey, I got the upgrade to KitKat, and it is great!  But, now I get this warning saying that someone is monitoring my network connection!   How do I make that go away!?  I don't want someone monitoring my network connection!"

"Getting rid of that warning is easy.   Go to Settings->Security->Clear credentials.   After that, the warning will go away."

"Thanks!  The warning is gone.  But now I can't connect to my wifi network!  HELP!"

"Not a problem.  Go in to the configuration for your wifi network and change the 'CA certificate' setting to '(undefined)'.  Problem solved!"

"Perfect!  That solved my problem!  Thanks!"



And somehow Google either didn't consider this case, or they really want to decrease the security of wireless networks.

Anyway, if you happen across this post while looking for how to make this scary looking warning go away, and you use a secure wifi network, please just swipe the warning out of the shade.   It is really nothing to worry about.   (I'll do a follow-up post in the next week or so about how authentication on wifi works and why you should care.)

Edit 11/26/2013 : A coworker pointed out that my blog post was referenced in a bug posted to the Android bug tracker.   The same bug post had a response from someone using an @android.com e-mail address outlining why this issue isn't a problem.   I tested their solutions and wrote about it here.

My Insane Arcade System Build (Part 8)

Dunno how many parts this will get up to, but it will probably be quite a few as I slowly lumber toward the completion of this "little" project.

But, back to the gear.   Let's talk joysticks.   Since I want everything to light up, the sources I had for sticks was pretty limited.  Basically, it looks like Paradise Arcade Shop was pretty much my only choice.   This isn't necessarily a bad thing as they send candy when they fill an order!

As you can see, the box was a tad hammered when I got it :

Of course, this was not likely the fault of Paradise, but of the postal system.   But, it did make me a tad nervous about how the sticks inside would look.

Fortunately, nothing appeared off when I opened the box :

Inside, there was the previously mentioned candy (top right), and 5 individually wrapped sticks with RGB LED ball tops.

For a better look :

The micro switches in the sticks seem to click pretty loud, which hopefully will be a non-issue once the sticks are nicely mounted inside their wooden control panel box.  (Spoiler Alert : They are fine once they are mounted.)

I wired up each of the sticks to an Ultimarc LED controller and a mini-PAC to make sure that they worked.  Of the 5 sticks I bought, 4 of them worked perfect.  One stick had a burned out red LED, which gave me the opportunity to find out how customer service is with Paradise Arcade.

I sent an e-mail and got a quick response asking if I had used resistors in line with the LEDs to avoid burning them out.   According to the documentation for the PACLED64 boards from Ultimarc, there is no need for resistors in line, so I responded letting them know which controller I was using and that the documentation specifically said I didn't need resistors.  (I took High School digital electronics, so I knew that resistors were usually used.  So I made sure to check the docs.)

After that, I didn't get a response for the better part of the week.   In frustration I posted to the Arcade Controls forum asking if anyone had any issues with Paradise in the past.   A couple people said that they had, but calling them on the phone solved the problems quickly.  (As an aside, most people had not had any issues.)   I called and left a message asking for a status update on getting a new set of LEDs for that stick, and within an hour had an e-mail indicating that the LED was being shipped along with an apology for it taking so long to respond.   Perhaps they were on vacation or something?   Whatever it was, I did post back to the forum that the problem was resolved to my satisfaction, and I would be happy to order from them again.  (As I am writing this after the control panel is all put together, I can safely say that I actually did order from them again.  More on that in a later post.)

One thing that I thought was really cool about these sticks is the restrictor plate comes built in to them.  As you can see in the image below, the plate fits three ways.  The default way allows for fully 8-way movement.  Flip it one way and the stick becomes a 4-way stick.   Flip it the other way, and it appears to become a 2-way stick!  (I didn't plan on using a 2-way stick, so I can't comment on how well that works.)  As I was shopping for sticks, it seemed that most of them required the restrictor plate be purchased in addition to the stick, so this seemed like a pretty good value.

The sticks themselves seem to be pretty well built.   I am FAR from an expert in this area, but they feel like they could easily handle the kind of moderate beating they are likely to get from guests at my house.

So, in short, I would recommend these sticks.

My Insane Arcade System Build (Part 7)

Among all of the stuff that has shown up, are the light guns.   I took a lot of pictures of these because the pictures on the ArcadeGuns.com web site left me with some questions about button layout.

But, lets get to the pictures.  The guns showed up in a pretty standard USPS priority mail box.  It shipped two day priority from the ArcadeGuns.com folks.

Inside, the guns were wrapped in bubble wrap and surrounded with packing peanuts.  As usual, the packing peanuts got everywhere when I opened the box, which was a pain.  But, beyond that, everything seemed to be packed snugly in the box.

As you can see, the guns were each wrapped in bubble wrap, as was the IR bar that goes along with them.  Because the IR bar will end up going behind the plexi-glass of the cabinet, I saved myself a few bucks and ordered it without the case.

The IR bar itself really doesn't have much to it.  It is made up of 6 IR LEDs and a few resistors.  One nice feature is that the cable can easily be disconnected from the IR bar.   I suspect this will make things a bit easier when it comes time to install this in the cabinet.

The guns themselves are where I had a few questions while looking at the ArcadeGuns.com site.   With some digging, I was able to find pictures that lead me to believe that there were two buttons on the guns.  One is where the hammer would be on a revolver, the other is in the center of the grip.

I am not sure how much I'll like having the button on the grip.  It seems like it would be easy to hit.  But, I guess it can always be programmed not to do anything.  (Or to reload, which would make accidentally hitting it a bonus!)

The mold of the gun is also interesting :

If you look at the Ultimarc web store, you can find the kit to build your own light gun.  It is really small.  So, obviously most of the casing of the gun is to make it comfortable in your hands.   The really interesting piece to me is the section right in front of the trigger.  It seems unlikely that piece holds any kind of circuitry in the gun.  Perhaps ArcadeGuns.com is thinking of adding a recoil feature in the future?   Or, I guess it could also be just to even out the weight of the gun in your hand.

I have not had a chance yet to try the guns out on a game.   But, one other thing I was curious about was how the buttons feel.   I remember from my younger days, playing "Operation Wolf" on the 8-bit Nintendo with the zapper.  The trigger on the zapper was really loud and and made a snapping sound each time you pulled it.   This didn't seem like a big deal for games like Duck Hunt where you only pulled the trigger once in a while, but on games like Operation Wolf, where you pull the trigger rapidly, it was really noisy, and felt as though you were going to break the trigger mechanism in the gun!  Fortunately, these guns are nothing like that.

Each of the switches on this gun seem to be fairly soft.  They make a rubberized "squishing/clicking" sound when you press them.   Very similar to the buttons on remote unlock systems for cars, but probably a little softer.  Overall, I expect they will be pretty quiet when in use.  And, while I am unsure how much I am going to like having a trigger that soft, there does seem to be enough forward force on the trigger so that you could easily tell when you have pulled the trigger fully.   Also, while the trigger looks rather large, it doesn't seem to need to be pulled back very far before you hear the clicking of the switch inside.   I am guessing that for rapid fire games like Operation Wolf, these guns will be comfortable, and not feel like they are going to break like the old NES zapper did.

Finally, as you can probably tell from the picture above, there is quite a bit of USB cord on these guns.  I am very hopeful that there is enough that I can string it through the cabinet and get to the USB ports on the PC inside.  But, I won't know that until the cabinet is complete, which is probably still a couple of months away.

My Insane Arcade System Build (Part 6)

Boy, oh boy!  Orders have been working their way to my house!   Probably enough for a few more posts!

Lets start with a few simple things.  I don't have panels that make up the control panel from North Coast yet.  Those will probably be here in the next week or so.  However, I figured it is never too early to start looking at things to make it pretty.   So, while looking around for T-molding, I came across t-molding.com.  T-molding.com will send you some samples to look at, so I took them up on it and ordered samples of the 5/8" molding to get a feel for how the colors looked.  Here is a picture of the samples :

The one on the farthest left looks like an issue with the camera, but it is actually black with a silver stripe.  It looks really cool, but I am not sure it will look good on a cabinet that is already black.   I'm leaning toward the light blue right now, but have to wait until the first parts of the cabinet show up to be sure.


My order from AllElectronics.com came packed in a pretty typical shipping package from the USPS.

Inside was the goodies that I will need to make the connections to the sticks and buttons.   Which basically boils down to quick connect pieces, a few DSub connectors, and some wire.  It all came packed in a large plastic bag wrapped with some brown packing paper.  When it is unpacked, it looks like this :

I elected to get a wiring that matches the colors used on a standard molex power connector on a computer.  This is because the PACLED64s need power and come with a molex connector.   It seems it will be easiest to keep the colors all the same.

While not the most exciting stuff, it is all necessary to complete the project.  And, since I wanted to document everything involved in it, you get this exciting post!  Enjoy!