Introduction

This is the first of a series of posts aimed at showing and discussing best practices on TDD and Android. The title is obviously an homage to the GOOSGBT book, as it shares the same purpose of showing how a real application grows guided by tests. The project I’m working on is a Github Android client, and I’m trying to make every commit as self-descriptive as possible, so that you can follow a logical series of action jumping from commit to commit. These blog posts are a way to expand those commit messages, giving more insight on why certain choices are made, discussing alternative approaches and gathering feedback from other developers.

While GOOSGBT is still a great reference for TDD and Java, Android developers face particular challenges while testing their code, mainly because the lifecycle of key objects like Activities and Fragments is managed by the framework. Thankfully, Android testing has gone a long way since its inception, and now we have an interesting array of tools tat our disposal. We can use Robotium to drive our integration tests and Robolectric to unit-test objects that depend on Android OS. We can use Injection to mock singletons and inject dependencies on the Activities and REST clients like Retrofit to abstract over a third-party service.

Of course, this application is mainly a pet project (see Breakable Toy), where I can try out new ideas and verify alternative strategies that are hard to setup in production code. Sometimes you’ll find things that will make a TDD purist cry, sometimes I’ll skip a few steps just to include an interesting library. That’s ok, if you see something that you don’t like you’re welcome to make a pull request.

But enough for the introduction, let’s jump right into the code. Every paragraph will show the link to the commit I’m discussing, so that you can read the post and the code side by side.

47cc8d54bb Initial Commit

Define the project structure as follows: * github-android contains the actual Android Application and the unit tests for its declared objects. It depends on github-android-core. * github-android-core contains plain java objects used to exchange information with external services (e.g. an api server) plus the associated unit tests. * github-android-it contains the integration tests used to verify the overall behaviour of the application. It depends on github-android. * github-android-smoke contains test fixtures used to verify the expected behaviour of external services. It depends on github-android-core only, so that the fixtures can run on a plain JVM.

We start with a project configuration that is an extension of the plain maven release archetype for android. What is crucial in this step, and I can’t stress it enough, is that you want, you really want to mock the external services that your app uses.

Just as in GOOSBT, you may build your application against a service that you can’t access, and you have to rely on its documentation. But more often, you find yourself in a situation where the external service is half-completed, with some features already implemented, some missing, and some that will change in the future. Having a dummy service at hand is a lifesaver in these situations, because you can quickly swap the original service for your mock, test and implement the required features, and merge them back to production when the apis are stable. The worst-case scenario is when the service is being implemented at the same time as your application: in this case not having a dummy service abstraction is just like digging your own grave.

But even if you’re working against a reliable service (as we are, with our Github application), I think you should go the extra mile and use this configuration anyway. This is because it has several advantages:

• 1) It makes writing tests easier
  Knowing the response in advance is a huge advantage because you can predict what you should look for in your tests.

• 2) It makes running tests easier
  No network activity means faster and reliable tests. If a build fails you know that it’s your fault. If your build starts failing because of random connection timeouts, server outage, or something else you can’t control, you tend to ignore your “Build Failed” messages, and your CI become useless.

• 3) It pushes you to write more flexible code
  Your core package is the bridge between the service and your application. Here is where you validate your models, where you abstract over the service interface and where you try not to let external concepts leak in your application. This is your first line of defence against external changes, and in a perfect world, if something changes in the external services implementation you would only need to touch you core package, making it transparent for the rest of the application. As a minor plus, being a pure Java project means that this module is potentially reusable in other applications (e.g. Desktop or J2EE) that interact with the same service.

• 4) It pushes you to write tests for the service you’re mocking
  If you mock an existing service, you want to be sure that your predictions are correct. In order to ensure that, for every expectation you have you write a test. This is a good way to understand the service and it’s code that you want to write anyway because it is necessary in order to place the correct calls. Moreover, you can catch unexpected or wrong behaviour early on, before it crashes your application and you have to do an intense debug session just to discover that it’s not your fault. You will find those test useful even if you just want to double check a particular call: it is really handy to have a named test method for that, instead of copying and pasting urls in your browser, or using external scripts.

• 5) You should have some cached responses for testing anyway
  When you’re writing your POJO deserializers you need a proof that the deserialization is always successful, right?

• 6) It is always nice to run your app in mock mode
  The service is down for the rest of the day. Or
  You have to show the app to the client and want to avoid random crashes. Or
  You need to verify the behaviour in a particular edge case (a network timeout, a particular response, a user with too many objects in its basket etc.). Or
  The service is rolling api v2 today breaking retro compatibility with your application. Or
  You’re stuck in some weird place with no network connectivity.

Bottom line: once you have a mock mode ready at your disposal, you’ll find more and more uses for it.

Introduction

This is the first post in a series about debugging Android memory issues - subsequent posts will deal with using Eclipse Memory Analyzer to inspect heap dumps, as well as looking at common Android-specific memory consumption pitfalls.

The contents of this post ought to be useful to developers new to the Android platform (regardless of Java experience) as it details Android-specific APIs and the use of Android SDK utilities to successfully interoperate with the standard JVM toolchain (e.g. hprof-conv which converts Dalvik heap dumps into the J2SE HPROF format, expected by Eclipse Memory Analyzer; MAT). Once this has been accomplished, the process of debugging leaks is more or less identical for Android applications and standard Java code running on a traditional JVM.

What’s a Heap?

All we really need to know about the heap for the purposes of this post is that it’s a slab of VM-managed memory into which (most) Java objects are allocated - certainly all of the objects we’re likely to care about. When we’re talking about the heap size of a particular object - how much of the heap it occupies - we talk of Shallow Size (or Shallow Heap) and Retained Size (or Retained Heap). The shallow size is the amount required for an object itself, while the retained size is the amount required for an object and all of the objects it refers to (i.e. via instance attributes).

A heap dump is a portable (i.e. file-based) representation of a VM’s heap at a particular point in time.

Obtaining Heap Dumps On Demand

Interactively getting an Android process to dump its heap is trivial with DDMS (there’s a Dump HPROF File button). That’s not our concern; we’re focused on obtaining dumps at particular points in a program’s execution by using the android.os.Debug API, which is an approach we might want to take if pathological memory consumption is peaking at points which aren’t easily identified through interactive use.

That said, our example is fairly contrived - an in-app Button which triggers a heap dump is isomorphic to the Dump HPROF File DDMS feature - figuring out when to trigger the dumping is entirely application-specific - the code samples are to illustrate how to do it, using a triggering mechanism likely to be familiar to anybody who’s written an interactive Android application.

Here’s an example of a View.OnClickListener which accepts a String data directory, and writes a heap dump within it, when the onClick method is invoked:

Below is an excerpt of an Activity which attaches the above listener to a Button, and passes in a sensible value for the path prefix, so that the dump ends up somewhere useful (note that this method will overwrite an existing file at the supplied path):

When the Dump Heap button is activated, we’ll be able to retrieve the file from the Android device/emulator by executing the following command on the host machine (assuming $ANDROID_HOME/platform-tools is in $PATH):

% adb pull /data/data/com.novoda.example.MemoryLeaker/MemoryLeaker.dalvik-hprof
226 KB/s (9160365 bytes in 17.973s)

If we plan to do anything with the heap dump, we’ll need to convert it to the J2SE format, as outlined above, using the hprof-conv binary, in $ANDROID_HOME/tools (which ought to be in $PATH):

% hprof-conv MemoryLeaker.dalvik-hprof MemoryLeaker.hprof
% ls -lh MemoryLeaker.hprof
-rw-r--r--  1 moe  staff   9.2M 18 Apr 15:33 MemoryLeaker.hprof

Grabbing a Heap Dump When Things Go Wrong

Alternatively, if we’re dealing with a worst-case scenario and have absolutely no idea what’s causing the leakage, or lack the patience to babysit the application until it falls apart, we can install a handler which’ll catch an OutOfMemoryError and try to write a snapshot of heap use at that point. Alternatives would be using either Activity.onLowMemory or Application.onLowMemory, which have vaguely defined semantics, and are not as definitively catastrophic as an uncaught OOM (from Application’s documentation: “this is called when the overall system is running low on memory, and would like actively running process to try to tighten their belt. While the exact point at which this will be called is not defined…”).

Back to Uncaught exception handlers: these are per-Thread, which is clear from the entrypoint - Thread’s setUncaughtExceptionHandler instance method. Below is an example of a Thread.UncaughtExceptionHandler suitable for passing in:

And the code to set this up at Application startup:

Next Steps

The next post in this series will detail how to use MAT to find likely culprits in a heap dump which has been fetched and converted using the above method.

ZeroMQ vs. JeroMQ

While researching possible technologies for implementing a publish/subscribe-based task distribution system for long-running, semi-autonomous/intelligent agents (with Android as a supported platform), ZeroMQ emerged as a sensible choice of message transport, given the exploding complexity of AMQP (Pieter Hintjens’ August 2008 steering comittee missive sums up my misgivings).

The ZeroMQ/jzmq Android build instructions look pretty onerous, particularly for pre-3.0 versions (due to a libuuid dependency) - JeroMQ (a pure Java implemention) - seemed like a natural choice. I haven’t benchmarked anything like a real-world use case, though there are some promising numbers published by the JeroMQ authors.

Since the JeroMQ and jzmq (0.2.0 series) API structures are functionally identical, there shouldn’t be any real cost associated with making a choice for exploratory reasons (barring showstopping bugs). The code snippets below adumbrate a minimal working example of a ZMQ client/server in request/response mode, running in a single process (an Android application). There are a dearth of Android-specific ZMQ examples around - I’ve endeavoured to use Android-specific concurrency abstractions where they seem idiomatic.

Note that the purpose of this example/tutorial is to provide examples of how to issue a request and process a response using a trivial message format with ZMQ - while all of the work is happening in a single process, the server & client threads can be viewed as separate applications; trivial adaptations of the Hello World request/response example applications (hwclient & hwserver) which are ubiquitous in ZMQ tutorials.

Code

Let’s start with a Handler which takes an instance (conforming to a trivial interface) to which it dispatches message events, and a key name identifying the String datum within the corresponding Message’s Bundle which holds the message payload.

Predictably, the interface isn’t particularly jazzy.

The long-running server, which blocks on reads and instantly responds with a reversed version of the message payload is defined as a Runnable, taking a Handler which is used to communicate the content of incoming messages to interested parties.

The client/request-side is implemented as an AsyncTask which re-does all of its setup when run. AsyncTask.execute takes a String to use as the message body, and the constructor, as above, takes a Handler to which received messages are dispatched (after being bundled).

UI

The UI is incredibly simple, and the layout XML doesn’t bear repeating. The three pieces of information required to make the Activity code intelligible are:

• text_message is the EditText at the top.
• button_send_message is the Button beside it.
• text_console is the large TextView console below.

Wiring Everything Together

Notes:

• The static methods in Util aren’t particularly useful - the implementations have been omitted to reduce clutter.
• No Activity-lifecycle contingencies have been made in the above code.
• From the AsyncTask documentation: “When first introduced, AsyncTasks were executed serially on a single background thread. Starting with DONUT, this was changed to a pool of threads allowing multiple tasks to operate in parallel. Starting with HONEYCOMB, tasks are executed on a single thread to avoid common application errors caused by parallel execution.” Both seems like strange choices to me. In any case, this example doesn’t use THREAD_POOL_EXECUTOR, so client tasks won’t run concurrently.
• Your application’s manifest (AndroidManifest.xml) will have to request android.permissions.INTERNET if you’re using the TCP transport, as in this example.

Alternatives

MQTT makes sense on a couple of axes, though it may be a slightly curious choice given the heterogeneity of clients in this use-case (it’d be easier to defend if clients were exclusively Android, or mobile-only). It may well be the subject of a follow-up blog-post.

ZeroMQ is used more in the wild, and it may be more flexible in case any strange requirements emerge (for example, if we were to develop an in-browser Javascript task consumer which required the proxying of ZMQ messages over HTTP, then the existence of middleware-factories like Mule might make this easier). There’s an example of what looks something like ZMQ over HTTP via Mule although I haven’t looked at it (Mule, nor that example) in depth. The word “middleware” makes me a little queasy.

“Are you still there?!? YES .. well no but YES!”

At Novoda we use Google Hangouts for communication between our London and Berlin office and also for connecting people when working remotely. We aim to always have our hangout in the London office live so anyone can connect and see what’s going on. The problem comes because if you leave a Google Hangout running for ~2 hours it brings up a dialog asking “Are you still there?”. If you don’t click ‘ok’ it will disconnect you from the hangout.

Solution! (Chrome variant) There is a nice Chrome plugin called ‘Ninja Script’ you can get it here. This allows you to run javascript when a specific URL is loaded. i.e. when you connect to the Hangout, you can run some javascript that will click the ‘ok’ button when it appears, yay!

After you have installed the plugin, click the ‘blue cross’ on your Chrome bar. Press the ‘new’ button at the bottom. Give it a name and description, paste the javascript from below. Finally press the ‘Autorun’ button, here you can paste https://plus.google.com/hangouts and then this script will run automatically when you start or join a new hangout. Done! One caveat, this plugin is a bit flaky in it’s UI (but once setup runs great). If you find you cannot click some of the buttons go into your Chrome settings and disable and re-enable the plugin.

Here is a script to solve the problem! Thanks to spencerelliott on userscripts.org userscripts.org

For convenience, the script is below (v1.0.8)

(function()
{

    function addJQuery(callback)
    {
        var script = document.createElement("script");
        script.setAttribute("src","https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js");
        script.addEventListener('load', function()
        {
            var script = document.createElement("script");
            script.textContent = "(" + callback.toString() +")();";
            document.body.appendChild(script);
        }, false);
        document.body.appendChild(script);
    }

    function checkForPrompt()
    {
        function simulate(target, evtName)
        {
            evt = document.createEvent("MouseEvents");
            evt.initMouseEvent(evtName, true, true,document.defaultView, 0, 0, 0, 0, 0, false, false,false, false, 0, target);
            target.dispatchEvent(evt);
        }

        function simulateClick(target)
        {
            simulate(target, "mouseover");
            simulate(target, "mousedown");
            simulate(target, "mouseup");
            simulate(target, "mouseout");
        }

        $('div[role="button"]').each(function(idx, item) // For each div with attribute role = "button"
        {
            if ($(item).html().indexOf("Yes") >= 0) // Correct button found
            {
                simulateClick(item);
            }
        });

        setTimeout(checkForPrompt, 6000); // Repeat every 6 seconds
    }

    function init()
    {
        addJQuery(checkForPrompt);
    }

    setTimeout(init, 6000); // Start after 6 seconds
    alert("Keep Alive Script Running");

})();

Here is the original script .