Network May Be Monitored By A Third Party (a.k.a. Android KitKat scary warning message part 3)

I guess people care about this annoying warning message that Android has thrown on us.   No other blog post here has generated anywhere near the number of hits that this issue has generated.  So, to all of your that have found my blog and read what I have to say, I thank you!

Unfortunately, Google doesn't seem to care about this issue.   (Newsflash!  Google doesn't care about making certificates easy to use on Android.   And, by extension, they don't care to make wireless authentication on Android usable either!   I could write volumes on my experiences trying to get things improved.  But, I fear that it would inspire depression in anyone that read it.)   So, we need to do something to try to GET them to care!

There have been several bugs opened about this problem, and most of those bugs have been closed with various reasons that make no sense.  (Which is what I base my claim that they don't care on.)  Some of the other people on the Internet that agree with me that this is a problem have continued to open new bugs to continue to push the problem and make Google listen.   If you care about this problem, PLEASE go to the Google bug tracker, and at least star the issue.   If you have the time, you might consider posting a comment to the bug that outlines why this issue causes you grief, and what changes you would like to see made in order to improve the situation.   (If you are only going to complain, do us all a favor and just star the issue.   If you want to complain, and offer suggestions on how to improve things, please post a comment!)

I'll even make it easy for you.   The URL (with link!) to the current version of the bug is at : https://code.google.com/p/android/issues/detail?id=82036 .


As you make your feelings known, please consider this.   There is actual value in having a warning like this, if you are the kind of person that values privacy and is against network operators using man-in-the-middle tactics to monitor what you are doing.   And, thanks to stupid legislation here in the US, if you are attending a K-12 school that provides Internet access, you SHOULD care about this very much!   US law requires the schools to snoop on students as a condition to continue to receive funding!    I'm lazy, so I am not going to track down the exact law that requires this, but I know it is true because of my day job.   (Not going to say any more than that because my opinion might get me in trouble.)

Because I would prefer that people being snooped on are given any warning that it could be happening, I am actually in favor of the warning message that was added.   However, I have problems with how it was implemented.   I think there are a few tweaks that could be made to the implementation that would take this from being a pointlessly scary warning message to a valuable warning message, and an opportunity to educate at least a few people on some of the security issues that are associated with installing new root CA certificates on any device or OS.   Also, as I have stated before, I don't like people that complain about a problem without offering solutions!

So, here are my suggestions on how to transform the annoying mess that is the "scary certificate warning message" in to something useful and significantly less annoying :

• Don't show the message (ever) if the certificate is used with 802.1X on a Wi-Fi connection -- Starting with Android 4.3, the "Credential Store" (a.k.a.  Certificate store, or key store) has been split in to two parts.  One part holds the certificates that will be used for authentication on Wi-Fi networks.   The other part holds certificates that are used for VPN connections, web site connections, and everything else.   Since you *HAVE* to install a certificate in order to securely use a Wi-Fi connection, showing this warning message is pointless.  The other alternative is not to use a certificate to validate the Wi-Fi network, which leaves you WIDE open to anyone doing a man-in-the-middle.   So, the scary warning actually has the potential to make users LESS secure by making them remove the certificates necessary to secure their networks!   Google must recognize this on some level, because they don't show the warning if an app uses the Wifi API to install certificates.
• When you tap the warning, give useful information --  Right now, when you tap the warning, you are taken to the security settings screen.   This is COMPLETELY useless.   The user is expecting to be given more information on what this warning is all about.   Take them to a screen that explains what the potential issues are with installing a new CA certificate.   Explain clearly what the problem is, and don't try to scare them in to submission.  Try to give the user enough information to allow them to make a good decision about leaving the certificate installed.   Also, provide links to more detailed information so that those that are really curious can really dig in and understand the problems.
• Allow users to get rid of the warning -- Right now, you can dismiss the warning, but it comes back any time you install another CA cert, or anytime you reboot your device.   This behavior is annoying.   Tell me once, let me decide what to do, and then leave me alone!  Anything else is a bad user experience.   When the user taps the warning to get more information about what it is about, they should also be given two check boxes.   One check box should allow them to dismiss the warning just for the current certificate, the other should allow them to block ever showing it again.   There really are two types of users where this warning is concerned.  There are those that already understand the risks, and those that don't care.   The purpose of the warning to to try to get some of those that don't care to actually start to care.  But, annoying the crap out of them is only going to make them hate the issue so much that they will never care.

If my three suggestions above were implemented, I suspect most people would no longer hate the warning message as much as they do now.   It also gives the power back to the user, which is what I always thought Android was supposed to be about.

I understand that in addition to warning users that they may be causing themselves problems, they are probably also trying to cover their butts should a bunch of users get hacked and complain in the media, or worse, try to sue.   However, I really believe that showing the warning once is more than enough for Google to cover their butts on this.

In the unlikely event that someone from Google reads this, and wants me to put my money where my mouth is, I will offer to submit patches to the AOSP project to implement all of the things above.  HOWEVER, given my previous experiences with submitting patches to AOSP, I will be expecting a promise that the code will eventually be included.   By "eventually", I mean that someone from Google will be assigned to review my code, and comment on it until the code is in the right state to be fully included in AOSP and eventually the main Android builds.   (As an aside, while I have tried not to give out any information on this blog about who I am, I am sure that Google has the ability to look up my e-mail address based on this blog being hosted on Blogger.   There is also someone on the Android dev team that knows me, and someone on the Android security team that knows me.  So, it should be easy enough to verify that I know what I am talking about, and it should be easy to reach out to me.)

How does this wireless authentication stuff work anyway?

My last couple of posts have been calling out some of the issues with wireless authentication on Android.   But, in my day job I deal with authentication on the most common operating systems you would see in the wild.  If you were only to read my blog posts this far, you would probably believe that Android is the worst device for wireless authentication there is.   The truth is, every operating system I have come across does something stupid when it comes to wireless authentication.

But, perhaps the worst component of authentication on wireless networks is the complete lack of understanding that most people have of how it works.   Now, you can argue that you don't need to know how your engine works in order to drive a car, so you don't really need to know how wireless authentication works in order to use your computer.  And, in general, I would agree with you.   However, most people would agree with me that knowing more about how their engine works results in an engine that generally works better because those people know what to look for and what types of behaviors can cause them trouble down the line.   With wireless authentication, it is also the same thing.   However, if you destroy the engine on your car, you will be out a lot of money to fix it or replace the car.   If you screw up with your authentication the damage can be far worse, and far more expensive.   So, do yourself a favor and learn a bit more about how the authentication on your wireless networks work.

My goal in this blog entry is to try to boil down the authentication process in to plain English, while mapping some of the technical jargon in to something that an average person can understand.  As a result, this blog entry will be very long.  But, hang in there with me.  I think it will be worth it.

We are going to look specifically at user name and password based authentication on networks.  This type of authentication is commonly used on enterprise wireless networks.  However, it can also be used on wired networks.  In fact, the method of authentication used in these networks was designed originally to be used on wired networks.  It was adapted to work with wireless networks, and that is where it seems to have found the most success.  This standard is called IEEE 802.1X.  Some people may now be yelling at their screen, "Nuh uh!  WPA-Enterprise or WPA2-Enterprise is used on wireless networks, not this 802.1X stuff!"  Those people are both right and wrong at the same time.   The WPA specifications outline how to use the encryption keys that are generated during an authentication with 802.1X.   So, we will be focusing on 802.1X, and leaving the encryption key stuff for another time.

There are two ways that an authentication starts.   One is that the network recognizes that you have connected to it and sends a request to start the authentication.  The other is the client machine (i.e. your laptop or phone) sends a message to the network requesting that it start the authentication process.  Both methods result in the network sending an identity request to the client station.   The station responds to this request with a message that contains some form of identification.   Because of the way that 802.1X has evolved, this identification is often referred to as the "outer identity" or "anonymous identity".  This terminology is technically questionable, but seems to be what the industry has settled on.  Probably because the "proper" terminology would just be confusing to anyone that doesn't understand how the various authentication methods works.   So, we will use the "outer identity" or "anonymous identity" to reference this through the rest of this blog post.

Following this identity request, the network will send a challenge message to start the authentication.   If you have ever configured authentication for a wireless network, and seen the choices such as (EAP-)PEAP, (EAP-)TTLS, (EAP-)TLS, this is where those settings come in to play.  The challenge message that the network sends includes an identifier that tells the client what type of authentication the network wants to use.  If the client is configured to allow that type of authentication, the process proceeds.   If not, then the client will respond to the network asking for a different type of authentication.   If the network allows the type of authentication the client requests then it sends another challenge, this time identifying the authentication method requested by the client.

At this point, the details of the methods used for authentication diverge wildly depending on which authentication system you use.   But, the basic ideas are all the same.   So, rather than dive in to the details of the various authentication protocols, we will discuss what the goals of the authentication are, and then work backwards.

The network, the client, and the user each have specific goals when they enter the authentication.  Those goals are :

• The networks wants to make sure the client is someone that is allowed to use the network.
• The client wants to make sure that it doesn't provide its credentials to anyone other than the networks that it believes it can trust.
• The user just wants to get on the network, everything else be damned.

So basically, the client and the network want to make sure that both the network and the user are protected.  But, as usual, the user is the weak link as they will probably do just about anything to gain access to the network.  (Facebook statuses won't update themselves!)     This is one of the biggest failures in the design of the various authentication protocols.   For methods such as PEAP or TTLS, the user has the ability to gut the security of the connection as an easy way of gaining access to the network.   

So, if we ignore the desires of the user and focus on the security aspects of the authentication, how do the goals of the network and the client get met?

The first step is for the network to identify itself to the client.   Currently, this is done using a word that strikes terror in to most network administrators.  "Certificates".   Certificates really aren't as scary as most admins think they are, but that is a topic for another blog entry.   For the purposes of this discussion, the network provides a certificate to the client that the client can run some checks on to verify that the network is who it claims to be.  (Side note : There are a lot of potential security issues with certificates and this type of authentication.  But, that is beyond the scope of this post.)   If the client decides it doesn't trust the certificate it will either send a message to the network indicating it doesn't like it, or simply disconnect from the network.   Assuming it does trust the certificate, various pieces of information are used to generate cartographic keys that allow all of the future messages to be encrypted when they cross the network.  This layer of encryption is usually referred to as the "inner tunnel" or sometimes just "the tunnel".

At this point, if the client believes it can trust the network, it will fulfill the desires of the network by proving it should be allowed access to the network.   This is usually done by using a second authentication method "inside the tunnel".   This portion of the authentication is known in most documentation as "phase 2" or "the inner phase".   At this point of the authentication, the client sends the user name and password back to the server using the encryption keys that were generated by the certificate exchange.   It is believed that the encryption set up by the certificate exchange is pretty secure, since it is basically the same type of encryption that is used when you purchase something from a secure web site.   

Taking a quick side trip back to the beginning of the authentication, it should now be more clear why the initial identity the network requests is called the "outer identity".   However, it isn't any more clear why it might be called the "anonymous identity".   The reason it can be called that is because the specifications for PEAP and TTLS explicitly state that the first identity request can be answered with an identity of "anonymous" because the users actual user name will be passed to the network using encryption later in the process.   The idea is that by returning "anonymous" for the first identity it makes it a little harder for a bad guy to use that identity to target a specific user, since the first identity is sent over the network with no encryption.  (Side note : The first identity isn't always sent unencrypted.   There are situations where it is encrypted, but the working assumption in the standards is that it is unencrypted.)

Once the user name and password are sent to the network, the network can decide if it wants to allow the client to access the network.  If the client is allowed, then the network sends a success message and opens the network up to be used.   If the client is not allowed, then the network sends a failure message and blocks the client from using the network.

While I don't want to get in to many of the security aspects of this type of authentication, I do want to point out the weakest link.   There is an assumption that the client will validate the certificate the server provides to it.   There are a lot of places that can go wrong, but by far the biggest one is the user disabling the checking of that certificate at all.   Sadly, some devices like the Nook tablets and Windows RT both default to not checking the certificate, or in the case of the Nook doesn't even allow the user to configure checking the certificate!  (Note : I have not looked at the latest Nooks, so it is possible this issue has been resolved.)  If the client doesn't verify the certificate, then the client will send the user name and password to any device that pretends to be the wireless network the user wants to connect to.   Depending on the authentication method used, this could mean that the password is sent to the bad guy with no encryption, or only lightly hashed.

Hopefully someone finds this post useful.   802.1X authentication can quickly go from something that seems simple enough to a wildly complex set of decisions with potentially huge negative consequences.   In the coming months, I plan to go in to more of a "deep dive" on the different authentication methods and what the known risks are with each of those methods.

Android KitKat : Network may be monitored by an unknown third party

Update 1/26/2015 - Google has refused to take this issue seriously.  They have closed several bugs posted on the Android bug tracker.   Fortunately, there are some other people out there that agree that the current implementation is broken.   Unfortunately, Google doesn't seem to think that it is worth fixing.   Since I know a lot of people find this page while looking for information on this annoying warning, I would encourage you to go to the current bug on the Android bug tracker, and star it to show Google that you want this resolved!   Perhaps if enough of us star it, Google will start to pay attention.   You can find the current open bug here.

Let me start by making the disclaimer that these observations are based on an AOSP build of Android KitKat on a Galaxy Nexus.   (My Nexus 5 isn't due for two more days.)   However, things like this that exist in AOSP tend to also exist in the release builds.

Along with all of the great new features in KitKat, Google has introduced what is probably the WORST security hole possible.   A ham-fisted implementation of certificate pinning.

Certificate pinning itself is a good idea.   It verifies that when you visit a web site, it provides you with the same certificate every time.   In general, certificates shouldn't be changing on web sites, and if one does you should be made aware of it.

So, how could certificate pinning be a security hole then?   By creating a situation where a harmless certificate creates a scary, and unnecessary, warning message.  When you install any certificate in to the key store, you get a warning icon in the bar at the top of the screen.   Pulling down the shade presents the following screen :

Let's consider the average user at this point.   Given the revelations about all of the snooping by various governments, it seems that a warning like this would make uninformed users very concerned.   (And, lets face it, there are very few informed users when it comes to 802.1X on Android.)   But, okay, maybe I am freaking out over nothing.  What happens when we tap the warning?

Hmm... Maybe not.   So, the assumption that an uninformed user is to draw based on this is that having any form of third party "trusted credential" installed means that a 3rd party will probably be monitoring my traffic.

I'm going to skip over the obvious irony here that something from Google is warning me that a 3rd party might be monitoring the web sites I visit, and reading my e-mail.  (If the irony is lost on you, you may want to do some research.)

Google seems to be trying to argue that the only safe type of certificate is one that comes pre-installed on your device.   Which is a downright silly argument no matter how you slice it.   But, lets go ahead and let that one slide.   Being the security minded individual that I am, I make sure that all of my network connections are as secure as possible.   So, I make the (probably bad) assumption that purchasing a certificate for my RADIUS server from a public CA will provide me what I need in order to have a secure wireless network.

Once I get the network setup, I try to connect my Android device to the network.   Now, being that we have paid even a little attention to the security issues around wireless networks, we know that we need to validate the server certificate in order to have a secure connection.   No problem, we purchased from a public CA, so we will just select that in the configuration settings.   But, you can't.   The pre-installed certificates on Android can't be used with 802.1X authentication.   Okay, no problem.  I'll just install the CA certificate on my device and then use that.   Oh, what is this scary message?

Now, those of us that understand the meaning of this message will just dismiss it.   But, let's assume that you aren't a techno savvy individual that has enough time to spend learning about security.   This warning is going to freak you out!   If this conversation hasn't happened on a message board yet, it will soon :

"Hey, I got the upgrade to KitKat, and it is great!  But, now I get this warning saying that someone is monitoring my network connection!   How do I make that go away!?  I don't want someone monitoring my network connection!"

"Getting rid of that warning is easy.   Go to Settings->Security->Clear credentials.   After that, the warning will go away."

"Thanks!  The warning is gone.  But now I can't connect to my wifi network!  HELP!"

"Not a problem.  Go in to the configuration for your wifi network and change the 'CA certificate' setting to '(undefined)'.  Problem solved!"

"Perfect!  That solved my problem!  Thanks!"



And somehow Google either didn't consider this case, or they really want to decrease the security of wireless networks.

Anyway, if you happen across this post while looking for how to make this scary looking warning go away, and you use a secure wifi network, please just swipe the warning out of the shade.   It is really nothing to worry about.   (I'll do a follow-up post in the next week or so about how authentication on wifi works and why you should care.)

Edit 11/26/2013 : A coworker pointed out that my blog post was referenced in a bug posted to the Android bug tracker.   The same bug post had a response from someone using an @android.com e-mail address outlining why this issue isn't a problem.   I tested their solutions and wrote about it here.

Meru configuration, part 1

As promised, I am going to give the QuickPhones phone a shot with the Meru gear. But first, I need to get the gear set up.

A quick rundown of what my configuration will look like :
1 - Meru MC 500 controller running version 3.4-103 software
1 - Meru MN-AP320
1 - Meru MN-AP201RH

I powered up the unit, and plugged in to it via a USB->Serial dongle. As the instructions indicated, I set up the serial port to 115200,8,N,1. I watched as the controller booted some flavor of Linux, and then hung on the "Starting Meru 3.4-103 wireless LAN services ...". After sitting for a while, I figured something was wrong, so I power cycled the box to see if it would come up a little quicker. (It didn't.)

I figured there was something wrong with my unit, so I pulled up a text editor to start writing my blog entry about how much I would have loved to write the entry about the QuickPhones with Meru. While I was getting everything together, the console spit out an error message, and then prompted me for a login.

I went ahead and put the default admin/admin username and password and got the console prompt back. At the console prompt, I ran the "setup" process. After answering the normal questions (ip address, gateway, netmask, etc.) the unit asked if it could reboot. (So I let it. Shocking, huh?)

Fortunately, it rebooted a lot faster than the initial boot. After a minute, I was able to log in and was back at the command prompt.

Normally, I am the kind of person that hates to configure gear with the web interface. But, in this case I decided to make an exception since the goal is not to evaluate the Meru gear, but rather to see if the phone works better on it. I fired up Firefox, and hit the web page. At this point, I was informed that I needed to use Internet Expoliter to access the UI on the box. (As an aside, I am sure that the version of code I am using is older. But, I still have to say, REALLY!? *sigh*)

Since I am already inflicting punishment on myself, I figured I would give IE 8 x64 a shot. When I hit the web site, I was first told that the certificate on the unit was suspicious (which is normal on gear like this), and after telling IE it was okay to continue, I was prompted for the username and password.

The UI that was displayed is reasonably clean, but doesn't show any useful information initially. (Probably because I don't have any APs it can talk to.) So, I pulled out the AP201RH. The first thing I noticed is that the AP appeared to want power over ethernet (PoE). So, I tried plugging it in to port 2 on the controller in hopes that it would power on. But, it seems that either port 2 doesn't do PoE, or there is some configuration setting that I need to enable before it will do it. Again, since I really want to get to the QuickPhones testing, I just took the AP in to another room that had a PoE port that I knew would work.

I managed to get the AP201RH up and running easily enough. So I plugged in the AP-320 so I could get it configured and ready to rock. Unfortunately, the AP-320 is running a different version of code, and try as I might, I am unable to upgrade, or downgrade it. Since the controller sees the AP as an AP-100, I am guessing this is a good thing.

Unfortunately, this is a dead end for this effort. I'll have to see if I can track down my friend at Meru to see if I can get some help. (The gear I have is test gear, so I don't have a maintenance contract to get the latest versions of code.)

I hope to have more in the coming weeks, assuming I can reconnect with an old friend.

The final(?) word on the QuickPhones phone.

It has been a while, but a few weeks ago I go another update from the QuickPhones guys about the multiple AP issue that I ran in to. I was extremely disappointed to hear that they were unable to find a fix for the multiple AP issue, and have given up trying.

So, what can I say? I will still tell you that this phone has the best battery life of any of the lower priced wifi phones out there. (I have not had a chance to try the more expensive ones. Feel free to contact me if you want to send me one!) When I didn't have it associated to a network, and just sitting on my desk the battery lasted about a month. When connected to an AP, the battery lasted about a week.

Now, just to be clear, it is possible that this phone would work perfectly on wireless gear that "pretends" to be a single huge AP. The one that comes to mind is Meru networks. I have some Meru gear that I am going to install, and will post a follow-up on this phone once I get it installed.

I have, however, discovered a bit of a workaround to the multi-AP issue. If you happen to have a single AP around that can broadcast another SSID, you can still use the phone in an area with multiple APs on one SSID. To do this, you need to shut down all of the APs that are broadcasting the same SSID so that the phone can pick up the AP that is single.

That didn't make a lot of sense, so let me expand on it. Lets say you have 5 "brand X" APs, all broadcasting an SSID called "CorporateFoo". While those APs are all broadcasting the SSID, the phone will be unable to consistently see any SSIDs around you. So, to get the phone working, you need to run to your local electronics store and purchase an inexpensive "brand Y" AP. Set that AP up and name it something like "IWishQuickPhonesWorkedOnMultiAPNets". (yes, I know that is too long for an SSID.. work with me here.) Now, shut down all of the APs that are broadcasting the "CorporateFoo" SSID, and reboot your QuickPhone. When the phone comes back up, it should be able to see your single AP that is broadcasting the "IWishQuickPhonesWorkedOnMultiAPNets" SSID. Configure the phone to use this SSID, make sure it connects and can talk to your SIP server. Once you verify it is working, you can turn your other APs back on. Now that you have the configuration set in the phone, it should always talk to the single AP. When it is powered on, it might take a while to find the AP, but it should eventually connect to it.

So, in a nutshell, if battery life is king for you, and you either only have a single AP, or don't mind jumping through hoops to get your phone working, the QuickPhone is what you want. If having a phone off the charger and on standby for more than 24 hours isn't important, then I recommend the UniData phone I reviewed a few months ago. It is a solid phone that has been working perfectly in my environment.