Symphonious

Ant Dependencies Running Twice

2013-05-10T08:42:25Z

Most people believe that ant will only ever execute a target once per execution of ant. So if we have targets A, B, C and D, where A depends on B and C and C and D both depend on D.

<target name="A" dependencies="B, C"/>
<target name="B" dependencies="D"/>
<target name="C" dependencies="D"/>
<target name="D"/>

When we run ‘ant A’, we expect each task to execute once, D first, then B and C, ending with A. Since there is no dependency between B and C, they may execute in any order, so we might also get D, C, B, A. All pretty straight forward.

Similarly, if we run ‘ant B’ we expect (and get) D then B and ‘ant C’ gives D then C. Most people expect that if you run ‘ant B C’, the execution path would be D, B, C but that’s not what happens. Instead we get D, B, D, C. Each target passed to the command line creates a new dependency chain and runs all targets in that chain.

The other case that catches people out is when using antcall. The antcall task actually sets up essentially a new execution of ant (by default inheriting any properties that have already been set). It never inherits the dependency chain or list of targets that have already been executed. So if our targets were:

<target name="A">
  <antcall target="B"/>
  <antcall target="C"/>
</target>
<target name="B" dependencies="D"/>
<target name="C" dependencies="D"/>
<target name="D"/>

Then the execution order for ‘ant A’ would be A,D,B,D,C.

The cost of running targets twice is reduced because most ant tasks do nothing if their inputs haven’t changed (so javac only compiles files once), but for a large project there can still be a significant cost in checking over all the files to determine if anything needs to be done. So whenever possible, use actual dependencies to build up a chain of targets instead of antcall or specifying multiple targets on the command line.

Doctypes, Compatibility Modes, Charsets and Fonts

2013-04-26T01:08:22Z

This information is all covered in much more detail elsewhere on the web but for my own future reference, here’s a primer on doctypes, compatibility modes, charsets and fonts which was required to explain why certain Chinese characters weren’t showing up in IE8. Of course the best answer is that you need to have the East Asian Font pack installed and then it just works (usually) but this tends to be useful background and saves “server side” folks from a number of gotchas.

Doctypes and Compatibility Modes

IE 7 and above has an insane array of compatibility modes which are out to get you. The most common gotcha is that it will use compatibility mode (emulating IE7) if the website is in the “intranet zone”. There’s an option to disable this somewhere in the preferences dialogs. You wind up in the intranet zone if you’re accessing a site via any domain name that doesn’t look like a real one (e.g. http://dog/ is in the intranet zone).
If you can avoid falling into compatibility mode, any pages including the DOCTYPE as <!DOCTYPE html> or <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> will render in standards compliant mode (where the world is as sane as it gets in web development). Go with the shorter version unless you have a reason not to.
http://hsivonen.iki.fi/doctype/ is the gold standard for information about browser modes.

Character Sets and Fonts

If you’re tracking down problems with foreign languages there are two major categories of problems – encoding corruption (where characters come out garbled or as ?) and missing glyphs in fonts (where characters come out as little boxes).
Corruption is fixed by specifying the same character encoding everywhere. It is a security issue if any webpage is missing a meta tag defining the character set (smallest variant is <meta charset="UTF-8">). It must be the first tag in the <head> of the document.
Little square boxes mean that either the font currently in use doesn’t have a glyph for that particular character and the font fallback routine was unable to find any font on the system which supports that character.
Browsers have a default stylesheet which is automatically applied to every page which commonly sets a specific font-family and font-size for text input elements, so adding the style body { font-family: 'Arial Unicode MS' } may get some Asian characters working in the main content but not in text boxes unless you also add input { font-family: inherit; }.

The security issue mentioned above is that any page which doesn’t define a character set but includes any form of user supplied content is vulnerable to a cross site scripting injection attack – even if the user supplied content is escaped properly, because the content may include a character that causes the browser to incorrectly switch to UCS-7 or other weird character sets and drastically change the meaning of the content on the page (hence the user content is no longer correctly escaped). There have been steps taken by modern browsers to remove this risk (including removing support for UCS-7 I believe) but its good practice to specify your charset explicitly anyway.

Media Release or Bug Report?

2012-12-10T23:58:20Z

Ah Apple maps, ever the source of a good sensationalist headline. This time the Victorian police have warned people not to use Apple Maps to get to Mildura. This is definitely a bug with Apple maps, no question it should be and has been fixed. What’s interesting though is that the Victorian police thought it would be easier to attempt to notify every iOS 6 user about the problem via the media and get them to use an alternate mapping application than it would be to call Apple and get them to fix the source data.

Users complaining publicly instead of being constructive and reporting the problem to the manufacturer certainly isn’t new, but in the age of web services its more self-defeating than ever. Apple can fix the problem in one place and its fixed for every user, surely the Victorian police have a better way of getting in touch with Apple than through the media?

And lets not even start asking why anyone would blindly follow GPS directions into the Australia bush despite the fact that they are looking at a satellite photo that clearly shows there isn’t a town there.

The power of user feedback and cloud hosted data was very clear with my recent experience of using Apple maps – on the way to my destination was some road works which diverted the road and replaced a round about with a set of lights. Despite the fact that the roadworks were actually still in progress, Apple maps gave exactly correct instructions for the new layout. Meanwhile my TomTom GPS will still think there’s a round about there for up to another year and then require me to pay nearly $100 to get the updated maps. It wasn’t too long ago that I’d be looking up a book of maps to get my outdated driving instructions (and then have to memorise it or stop regularly to have another look).

Makes you realise what an incredible time we live in.

Brawling on a Plane

2012-09-05T04:49:07Z

From Straights Times:

A flier aboard Sunday’s Swiss airline flight – a 57-year-old Chinese man [...] felt disturbed during his meal when the passenger in front of him reclined his chair.[...]
“The older of the two felt disturbed during his dinner. When the younger did not respond to his protests, he hit him on the head with the flat of his hand. It was a real slap,” said the guide, Ms Valerie Sprenger.
A fight then broke out between the two men, who rolled in the plane’s aisle. A crew member and a well-muscled passenger restrained the aggressor, bound his hands and placed him at the back of the plane, where he shouted for an hour, the newspaper said.

I can sympathise. In fact I’d have probably tied up the douchebag who put his seat back during meal service.

Demystifying Doubles: Consistent Inaccuracy

2012-08-29T03:36:19Z

Of all the data types, double is probably one of the most misunderstood. A huge amount of folk lore has been built up around it to help protect developers from falling into its many pitfalls. Lately I’ve done a lot of work replacing usage of BigDecimal with double and learnt a lot about where those pitfalls are and how the folk lore can be misleading.

The great challenge with double is that it has a degree of inaccuracy because of the way the number is actually stored. For example, the number 2983792734924.2293 actually winds up being stored as 2983792734924.2295, a tiny difference but with far reaching consequences. All of the folk lore around doubles is designed to deal with this potential for inaccuracy.

One piece of folk lore is that you should never do a strict comparison of double values (e.g. with == or !=). Due to the inaccuracy of doubles you should always compare them with a level of tolerance:

abs(doubleA - doubleB) < tolerance

It’s important to realise however, that this inaccuracy isn’t random, it’s perfectly consistent and predictable. The number 2983792734924.2293 always comes out as 2983792734924.2295. The same applies if you parse it from a string: parseDouble(“2983792734924.2293″) always comes out as 2983792734924.2295. As a result the following are consistently true:

2983792734924.2293 == 2983792734924.2293
parseDouble("2983792734924.2293") == 2983792734924.2293
parseDouble("2983792734924.2293") == parseDouble("2983792734924.2293")

Introducing tolerance to the comparison can lead to subtle bugs. For example, if we were updating our model with user input we might have something like:

// In our model:
public void setAmount(double amount) {
  this.amount = amount;
  fireChangeEvent();
}
// In our UI code:
model.setAmount(parseDouble(widget.getText()));

So far so good, but what if we want to avoid firing the change event if the value hadn’t really changed? We might change the model code to:

public void setAmount(double amount) {
  if (abs(amount - this.amount) > 0.01) {
    this.amount = amount;
    fireChangeEvent();
  }
}

It follows the double folk lore, but also introduces a bug. Let’s say the user first enters the amount ’1′. Then they change the amount to ’1.001′. The second amount is within the tolerance so it’s not considered an actual change. The user is now looking at a text field that says 1.0001, but the model thinks the value is still 1. You can make the tolerance smaller but that just means the user has to enter more zeros to trigger the problem.

The right answer is to take advantage of the fact that inaccuracy with doubles is consistent and do the comparison using strict comparison:

public void setAmount(double amount) {
  if (about != this.amount) {
    this.amount = amount;
    fireChangeEvent();
  }
}

So when doesn’t this work? When the two values are calculated in different ways. So:

(1491896367462.1147 * 29.837927349242293) / 2.3982983923
= 18561116318075.207
1491896367462.1147 * (29.837927349242293 / 2.3982983923)
= 18561116318075.203

The only difference between the two equations is the order of evaluation – they should give the same answer, but because doubles have a degree of inaccuracy the intermediate result is rounded to something that will fit in a double, so the order of evaluation matters because it affects what rounding is done to the intermediate result. Repeating the exact same calculation will always give the exact same result, but if there’s any chance that the values were calculated in different ways, comparisons must be done with tolerance.

So the folk lore about needing to compare doubles with some tolerance is perfectly valid, good advice, but be aware that it introduces its own inaccuracy which may cause problems. You should consider what the impact of falsely deciding that the values are different vs falsely deciding that the values are the same; which side should you err on?

Sometimes you really do want a strict comparison, even between doubles.

The Myth of Measurable Productivity

2012-08-14T01:04:02Z

Farnam Street Blog holds up a coding competition as evidence that:

If you want to make your computer programmers and engineers more effective give them “privacy, personal space, control over their physical environments, and freedom from interruption.”

The only trouble is, the coding competition is fatally flawed as a measure of normal developer productivity. It’s setup such that developers work on their own:

Each participant was also assigned a partner from the same company. The partners worked separately, however, without any communication, a feature of the games that turned out to be critical.

So what we actually learn is that if you’re working by yourself without interacting with anyone else, you will perform better if you don’t have anyone else around – that’s not overly surprising.

What happens in the real world however is that a team of developers work together to build software. The research is unable to shed any light on the conditions that benefit that mode of development because it measured something completely different.

And this isn’t the only example – experiments are constantly being put forward as proving or disproving things that affect productivity, but all of them are fatally flawed in one or both of two key ways:

The activity being performed in the experiment is not the same as what we’re trying to optimise in the real world.
The way they measure productivity is flawed – usually because it only measures a subset of tasks and attributes that would matter in the real world over the long term.

This shouldn’t be surprising, any repeated task is best optimised by automating it and removing humans altogether. The remaining work then is dealing with exceptions and is inherently unique (if the same exception kept happening we’d automate it). More importantly though, the idea that productivity of modern office workers can be measured is a myth. Any metrics you put around productivity are inevitably incomplete and lead to people gaming the system, working only on the measured parts of productivity and to the detriment of anything that isn’t measured. Measured productivity goes up, but those results aren’t reflected in the business’s bottom line – they’re fictional.

The only viable option is to experiment in your own workplace with your actual work and see what works for you. Use measurements as a guide but also follow some gut feel to account for the unmeasurable elements. Maybe individual offices are best for your workers, doing your work or maybe open plan is. Unless you’re in the business of completing abstract coding competitions, you probably should work that out for yourself instead of believing the research.

Continuous Integration Requires Integrating Continuously

2012-07-31T03:39:44Z

Sarah Goff-Dupont has a post on the Atlassian blog about how Atlassian is using feature branches and still doing continuous integration:

In the not-so-recent past, continuous integration in the context of story branching was considered so impractical as to be outright incompatible. Who has time to manually configure a CI scheme for dozens of active branches that will only live a couple of days? Nobody –that’s who. And story branch-loving teams would frequently encounter *ahem* “surprises” when merging work onto the main code line –”master”, in Git-speak. But recent versions of Bamboo have essentially removed that overhead and allowed these two frenenmies to become genuinely harmonious partners in our development lifecycle.

Unfortunately, what Sarah describes is not continuous integration. At best, it’s regular integration but really it’s just automated testing. The key principal of continuous integration is that all developers are merging all changes, all the time. That way you see the messy combinatorial effects of concurrent changes and can fix them before you waste a lot of time going down incompatible paths. Anything that reduces the frequency of doing that integration is a step away from continuous integration, regardless of how often automated tests are run.

Let’s take a look at the basic process they’re using:

Next, we go into development and testing cycles on the story branch. The developer is making local commits on her workstation. [...] When she reaches a point where she wants to run tests against her work, the developer pushes those local commits up to the repository –still on the story branch, of course. Bamboo sees that there’s been a change, and starts building the corresponding Plan branch/es.

This is not continuous integration. The developer’s changes are being kept away from the main team and not integrated while they’re being developed, while they sitting just in the local git repository and while they’re in the story branch. Even though Bamboo runs tests against the code, there has been no integration done here. This is just automated testing, not continuous integration.

It seems that Atlassian have discovered this lack of integration and have developed ways of combating it:

To help find conflict before merging to master, the developer can choose to have Bamboo merge master into her story branch each time she pushes changes to the repo, then run through the CI Plan against the merged code. If the Plan passes, Bamboo will commit the merged code to her branch.

Finally we get some integration happening, but only one way. Since everyone’s off working on feature branches this is really only integrating with completed stories. All the work going on in parallel on other stories is still not integrated so we have no idea if they will work when combined.

Some teams are taking this workflow a step further and requiring developers to merge to an integration branch during development, before finally merging the completed story to master. This practice ferrets out integration issues even faster. Developers get to mash their in-progress stories together several times a day, rather than waiting until one has merged a completed story to master (which may happen only every 2-3 days) and the other has pulled code from master onto their story branch.

The key lesson here is that if we integrate more often, it’s easier to ferret out integration issues. It’s like continuous integration but with the extra fun of managing branches…

Of course, Atlassian aren’t alone in the use of feature branches – it’s become all the rage since DVCS supposedly made branching cheap and merging painless. Unfortunately while DVCS has eliminated many of the basic mechanics of merging, they haven’t done anything to reduce the risk of different developers push the codebase in incompatible directions at a structural level. The only way to avoid that is to integrate all changes as often as possible – that’s continuous integration.

The only way that feature branching and continuous integration can be at all compatible is if the build system automatically builds and tests every possible combination of branches, all the time – a massive waste of build system resources. Anything less leaves potential combinations untested until they are finally merged on to master and all the problems are discovered.

Or you know, we could just do continuous integration…

Golden Rule of Dependency Management

2012-06-19T01:54:48Z

There’s a huge amount of complaining and problem solving for various dependency management solutions (particularly maven, but ivy and friends certainly aren’t immune). Problems range from having optional dependencies unnecessarily included to having duplicate classes or class conflicts and the solutions tend to be complex and involve a lot of trade offs. All these problems stem from breaking the golden rule of dependency management:

Own your own repository
– Sutton’s golden rule of dependency management

The vast majority, if not all, problems with dependency management comes from having incorrect, conflicting or imprecise meta-data in the repository of dependencies. Maintaining a public repository of perfectly accurate, precise and flexible dependency metadata is next to impossible – there are just too many libraries and the interrelationships are too complex. Fortunately, even extremely large companies only use a tiny subset of these libraries. With the scope reduced it’s much easier to ensure the metadata is correct and consistent.

Any time you need to introduce a new dependency, very carefully review the metadata associated it and correct any errors or inconsistencies before importing it into the repository you administer and control.

You don’t let anyone commit to your source code repository, don’t let anyone commit to your dependency repository either.

Minimise Configuration

2012-06-05T05:27:47Z

Having configuration differences between development and production is largely unavoidable, but it’s important to keep the number of differences to a minimum to avoid unexpected bugs appearing in production that don’t occur in development mode. More than that though, it’s important to minimise configuration.

Configuration is Code

Often things are put in configuration files so that they are easy to change later, often without a full redeployment of the software, but that also implies it can be changed without going through the full testing that a release would usually be subject to. It’s rare for automated tests to cover more than one possible setting for these configuration options so changing them in production is equivalent to deploying completely untested code. What’s worse, it’s quick and easy to do with no audit trail of what happened and when.

These settings should be avoided altogether and the values hard coded in the built product and deployed along with it. That may be as simple as embedding the existing config file in the built software – the aim isn’t necessarily to mix configuration and code, only to remove the options to change the configuration outside of the normal development and deployment process.

For settings that are designed to be changed by the user as a normal part of using the software, make sure that the full range of values is tested, just like any other system input should be.

Soft Coding

The most extreme form of configuration is “soft coding” where the system becomes so configurable its essentially a poor-man’s programming language. The Daily WTF has an excellent explanation of soft coding and how even the best of intentions can lead to it.

Dealing with Production Problems

One of the biggest reason people make options configurable by administrators is so that they can quickly reconfigure things to resolve issues in production. Sadly, the worst time you could be making untested changes in production is when things are already behaving in unexpected ways. Instead, make the build, test and deployment process for your software so fast and so seamless that you can make the change in code, fully test it and get it out to production fast enough to resolve the issue. Doing so isn’t easy, but it delivers immense value beyond just resolving production issues. It allows you to more reliably and more rapidly deploy any code change into production. That means spending less time wrangling releases and more time improving the product.

Default to Development Settings

2012-05-28T06:17:19Z

Most systems have multiple configuration profiles – one for production, one for development and often other profiles for staging, testing etc. Minimising differences between these configurations is critical but there are inevitably some things that just have to be different. This then leaves the question, what should the default settings be?

There are three main options:

Default to production values
Default to development values
Refuse to start unless an environment has been explicitly chosen

My preference is to default to development values. Why?

Development values should be “safe” in terms of any external integrations. So a developer isn’t going to accidentally start sending real buy or sell instructions to your stock broker.

There are more developers than production systems. If you default to production systems, every developer needs to remember to switch to development mode whenever they setup a new checkout. Defaulting to development mode means it just works for the most common case.

Checking authentication credentials for external systems into your source control system is generally considered a bad security practice, so the default values are unlikely to actually work in production anyway.

The down side with defaulting to development is it’s possible to accidentally deploy to production with development values causing an outage. This can be pretty easily prevented with automated deployments or using tools like RPM where files can be marked as config and thus avoid overwriting them when doing updates.

Refusing to start is the worst of all worlds – every developer has to specify a configuration mode and you still risk production outages by not specifying a mode.