Recently, I have been looking in to tools in Android that would help me improve code quality and find bugs before putting the product out. I picked up a copy of
Diego Torres Milano's "Android Application Testing Guide" as a good place to get me started. Since there are a lot of tools out there, and not all of them support Android, it was a useful guide to tools that should be working.
One of the tools I was looking in to is code coverage. Over the years, EMMA has been integrated with the Android build tools, which is supposed to make it easier to use. Unfortunately, if you don't use them just the right way they can end up being a horrid time suck trying to figure out why your instrumented build just isn't working.
For my testing, I was using the latest tools available as of this writing. (That would be ADT 20) Lots of the information available on the Internet is for older versions of tools that don't use the same commands as ADT 20. I also started with the most basic app I could, and set it up so that I had an easy way to intentionally miss some code, or make sure I hit it all. The code looked something like this :
@Override
public void onCreate(Bundle savedInstanceState) {
String appname = "unknown.app";
super.onCreate(savedInstanceState); setContentView(R.layout.activity_main);
}
public void testme() {
System.out.println("This is a test!");
}
A pretty simple "app" that mostly comes out of the Android tools when you create a new project. The "testme()" method is there because it will allow us to explicitly call some code (or not) to let us see that code coverage is really working.
Next, I created a test app using the Android tools and created a new test case. The test case extends ActivityInstrumentationTestCase2, as it will start up the activity we want to test in a state that is easy to use. The test class looked something like this :
public class MyTest extends ActivityInstrumentationTestCase2 {
public MyTest()
{
super(TestActivity.class);
}
public void testCodeCoverage()
{
}
The "TestActivity" referenced in the above code is the name of the Activity that we created earlier. Also, note that our test doesn't actually do anything. For now, this is okay. When the MyTest class is run, ActivityInstrumentationTestCase2 will create the activity, which will result in the onCreate() method being called. So, once we have our code coverage running, we will have a block of code that was covered, and a block that was not.
At this point, we are going to use the command line. We also need to make sure that we have a fairly current version of Apache Ant ready to use. For my tests I was using 1.8.2.
At the command line, we want to convert our apps to use Ant for building. To do this, we need to use the "android" command that is included in the SDK. (On Windows, it is "android.bat") We need to run two commands, one to convert the "TestActivity" app and one to convert the "MyTest" test project. The command will look something like this :
android update project --path /path/to/TestActivityProject --name TestActivityProject --target android-16
The "--name" parameter should be set to whatever you named your project, and "--target" should be set to the API level that you want to test with. The "--path" parameter requires the FULL path name to your project. So, if you are on windows it might be something like "c:\users\foo\workspace\TestActivityProject" I tested with an Android 4.1 VM, so I chose "android-16". The version you use isn't too important, unless you plan to use Ant to make your final builds.
Next, to convert the test project :
android update test-project --main ../TestActivityProject --path /path/to/MyTestProject
The "--main" parameter confused me a bit at first. It should be a relative path from the directory you store "MyTestProject" in, to the path of "TestActivityProject". If you are using Eclipse and putting your code in a workspace, you should normally need to use ../
.
Finally, we are ready to build and run our test. Go in to the directory for "TestActivityProject" and run "ant". (If ant isn't in your path, you may need to use a full path to ant instead.)
But, whats this? We get a help screen like the following :
Android Ant Build. Available targets:
help: Displays this help.
clean: Removes output files created by other targets.
The 'all' target can be used to clean dependencies
(tested projects and libraries)at the same time
using: 'ant all clean'
debug: Builds the application and signs it with a debug key.
The 'nodeps' target can be used to only build the
current project and ignore the libraries using:
'ant nodeps debug'
release: Builds the application. The generated apk file must be
signed before it is published.
The 'nodeps' target can be used to only build the
current project and ignore the libraries using:
'ant nodeps release'
instrument:Builds an instrumented package and signs it with a
debug key.
test: Runs the tests. Project must be a test project and
must have been built. Typical usage would be:
ant [emma] debug install test
emma: Transiently enables code coverage for subsequent
targets.
install: Installs the newly build package. Must either be used
in conjunction with a build target (debug/release/
instrument) or with the proper suffix indicating
which package to install (see below).
If the application was previously installed, the
application is reinstalled if the signature matches.
installd: Installs (only) the debug package.
installr: Installs (only) the release package.
installi: Installs (only) the instrumented package.
installt: Installs (only) the test and tested packages (unless
nodeps is used as well.
uninstall: Uninstalls the application from a running emulator or
device. Also uninstall tested package if applicable
unless 'nodeps' is used as well.
This is where I really screwed up and spent a lot of time scratching my head. Since we want to use EMMA, we need an instrumented build, right? So the command that we need is "ant emma instrument install test" to get a build, install it, and run the test in it.
While this seems like the most intuitive answer, you will quickly find out that it doesn't work. Everything will compile and install fine, and we can assume that the resulting apps are instrumented. However, when it gets down to running the test, you will be greeted with something like this :
[echo] Running tests ...
[exec] INSTRUMENTATION_RESULT: shortMsg=java.lang.IllegalAccessError
[exec] INSTRUMENTATION_RESULT: longMsg=java.lang.IllegalAccessError: Class ref in pre-verified class resolved to unexpected implementation [exec] INSTRUMENTATION_CODE: 0
Hmm.. Something didn't work. This is where I spent a ton of time trying different combinations and searching to understand what was going on. Finally, as I was giving up, I noticed a browser page that I had already opened that said for EMMA code coverage, we need to run "ant emma debug install test". I didn't think it would work, but I gave it a shot. Crazy as it is, it DID work! (I literally spent 7 hours trying to figure out what the above error message was all about.)
Once your instrumented build is run, the necessary coverage files will be downloaded, and a coverage.html file will be created in your test project's "bin" directory. Opening that up should show that your onCreate() method ran all of its code, but the testme() method wasn't touched at all. If you go in to the testCodeCoverage() method in the test app, you should be able to add the line "getActivity().testme()" and then run the instrumented build again. This time, you should get 100% code coverage.
Hopefully, this post saves someone out there a significant portion of their time getting started with code coverage!
