I will start with statistics regarding the adoption of agile. The Forrester / Dr. Dobb's Q3 2010 Global Technographics Survey, as referenced in the Forrester paper Water-Scrum-Fall is the Reality of Agile for Most Organizations Today, found that 55% of projects where using some form of agile or iterative method. This statistic is now almost 2 years old. The Project Management Institute (PMI) reports that the use of Agile has tripled from 2008 to 2011 and states that Gartner predicts 80% of software development projects will be using Agile by the end of 2012.

Agile has often been perceived as developer-centric, so I have found it quite revealing that over the last year other I.T. roles have formally recognized the importance of Agile. The PMI has come out with its Agile Certified Practitioner certification and the International Institute of Business Analysis (IIBA) is finalizing their Agile Extension to the Business Analysis Body of Knowledge.

This information about the adoption of agile is interesting but I do not view it as sufficient evidence on its own to adopt agile. What I do find compelling is the benefits that Agile brings. This is clearly summarized by the 2011 CHAOS Manifesto from the Standish Group which states that "software applications developed through the agile process have three times the success rate of the traditional waterfall method and a much lower percentage of time and cost overruns".

I will conclude this article with one final reference that I believe perfectly captures the point of my article. Gartner's 2012 Planning Guide for Application Delivery Strategies has a section that is simply titled "Say goodbye to waterfall".

I'm giving a presentation on this topic Thursday, May 17th, 2012 at noon that aims to answer these questions and provide guidance based on a number of case studies involving Agile government projects. Along the way we will cover various perspectives on what Agile is, shatter some common misconceptions, and see some compelling evidence for why your next software development effort should use Agile.

The presentation is for Agile Edmonton at a new location - Startup Edmonton which is located on the 3rd floor of the historic Mercer Warehouse at 10363 – 104 Street (on the corner of 104st and 104 ave). Attendance is open to the public, so please drop by to attend the talk.

The role that I want to share today is the architect as a scout in the military sense. A definition from Google is "a person sent out ahead of a main force so as to gather information about the enemy's position, strength, or movements." I like the military metaphor because software development is a team activity - the architect as scout is serving the rest of the team with the objective of helping keep the team moving ahead at a steady pace. This means being aware of what lies ahead, mapping the path forward, avoiding or eliminating obstacles, and being an initial target for hostile fire.

Awareness of what lies ahead

Architects need to be aware of what lies ahead in terms of both technical and business changes. The more accurate their picture of the future, the better their designs and decisions can be.

Awareness of the technical domain includes staying current on new libraries, frameworks, and platforms (e.g. mobile), learning about new software development practices (e.g. DevOps), and knowing what new infrastructure might become available to the team (e.g. middleware upgrades, cloud).

Awareness of the business domain includes having an idea of what major new functionality the business is thinking of introducing (e.g. need for multidimensional data analysis), and also having an idea of what business events or changes might significantly impact the system (e.g. significant variations in business usage of the system from how it was originally modeled, resulting in a significant change in the load and/or data volumes).

As an example, I recently recommended planning to do some refactoring to eliminate some duplicate code in the generation logic for two reports. Since there was a lot of this repeated code and it was clearly identified and isolated making it an easy, low risk refactoring, I though this was an obvious decision. Unfortunately, after making the recommendation, I learned that the business was expecting to eliminate one of the two reports in the near future. This completely changed the value proposition of the refactoring, actually making it potentially harmful in that it might increase the work required to rip out the one report entirely down the road. As a result, I reversed my recommendation.

Mapping the path forward

Architects determine how to use new technologies and how to solution new requirements to map out a clear path forward for the rest of the team. By tackling the risky or unknown items early, architects help the team maintain consistent progress.

Architects can use spikes, experiments, and pilots to try out and validate new technologies, solutions, and practices. Sometimes architects cannot do more than a basic evaluation in complete isolation. One approach I have used is to then do a pilot with a single team while closely monitoring and providing guidance, and then evaluate the suitability for rolling the change out more broadly within the organization. In such a case the entire team doing the pilot becomes a scouting force for the entire organization, led by the architect.

Solutioning of major new requirements can involve build-versus-buy decisions, determining major new components or infrastructure required, and architectural-level design. The common objective is to clarify what the team will need to do and allow for advance planning for activities needing more lead time (e.g. provisioning of new infrastructure, installing new software, training on new technologies).

For agile teams mapping the path forward typically means working an iteration earlier, ahead of the majority of the team, to ensure that business requirements are ready for the team to work on in the coming iteration and to ensure that there are minimal surprises with any new technologies or solutions. Architects can also be a source of technical requirements that are part of the product backlog. These activities help ensure that the team maintains a consistent, predictable velocity in delivering business value. Some agile teams might not have someone named explicitly in the role of architect, but all teams at least occasionally need someone operating as a scout to chart the waters ahead.

Dealing with obstacles

Architects need to identify and then avoid or eliminate obstacles that will potentially prevent successful delivery of the software or impede the team's progress.

Sometimes these obstacles are tricky technical issues that the team is having trouble resolving. As an example, one time I became involved in solving a performance problem with a batch process that took ever-increasing amounts of time to process individual records as the batch process ran. The problem was hard to find because it was not due to any particular piece of application code, but to how Hibernate (the underlying object-relational mapping library) was being used. To fix the problem I had to adjust the overall design of the batch process to reduce caching within Hibernate (follow the link to read more about the technical details).

Obstacles sometimes arise when the business raises requirements that are overly complex or extremely challenging to implement, especially in terms of meeting non-functional requirements such as performance or scalability. Architects sometimes need to push back on these requirements, or at least raise the alarm regarding the higher risk to ensure that sufficient time and resources are allocated to dealing with it.

Obstacles are not just with the software being developed or its requirements - they can also arise with respect to the project or external stakeholder expectations. For example, if management imposes an unrealistic delivery schedule on the team then architects are responsible for raising the alarm and helping determine ways to mitigate the risk.

Target for hostile fire

Architects serve as lightning rods for the team to initially bear the brunt of external problems or threats, especially those technical in nature. In this role architects act as a buffer to avoid having the team distracted or demotivated by these types of issues.

While the term 'hostile fire' is a metaphor, sometimes there are people in the organization who really are hostile. This may be due to political fallout from unpopular decisions, or stakeholder unhappiness with the software (e.g. due to poor performance or crippling defects). When the issues are technical the architect is the first contact - hearing the complaints and taking responsibility for what the team has produced. This is seldom pleasant, but is one of the responsibilities that comes with the role.

Sometimes the hostile fire really is a metaphor for tools or technologies that are too bleeding-edge or otherwise do not work as advertised. The architect then must confront the pain points and determine whether it is manageable or whether the tool must be scrapped and an alternative selected. Hostility may arise as a result of these issues or the resulting decisions, either from the team itself if they discovered these issues using a product proposed by the architect, or from parties within the organization outside of the team that are promoting or mandating the use of the problematic tool.

One example of this is years ago when new version control software was being introduced to replace CVS. I had not been involved with the initial pilot, but had heard disturbing things about the tool despite the decision to move ahead with its adoption. So I volunteered my team to be one of the first adopters and basically did my own evaluation. Shortly after transitioning to the new tool, I discovered a major design flaw that made it unusable. Efforts to work with the vendor to get the problem rectified failed, so I presented the situation to management and the decision to abandon the tool was made.

In conclusion, if you are an architect or functioning as a scout for your team ensure you are aware of what lies ahead, map out the path forward, deal with obstacles, and be the first contact for hostile fire.

To be clear, I am not talking about process certifications for individuals like the popular Certified ScrumMaster and Project Management Professional. I view these certifications as recognition of a person achieving a certain level of knowledge with a particular process - what the person does with this information is entirely up to them. (The value of these certifications is a completely separate debate I do not want to get into today.)

So what is my key issue with organizational process certification? It stems from my passion for continuous improvement. In my mind there is a fundamental mismatch between a true culture of continuous improvement, such as I have read about regarding Toyota in books like The Toyota Way by Jeffrey Liker, and the requirement to comply with a particular process to maintain certification. Continuous improvement may identify activities or aspects of the process that can be altered or eliminated due to lack of value (waste, in lean terms), but if these aspects are required by the certification, then the certification itself becomes an impediment to change.

One example comes from software development teams starting with Scrum and then inspecting and adapting their process as they proceed (which is a key aspect of Scrum). Some teams end up with an arguably different Agile process such as Kanban that can no longer be called Scrum, and other teams end up with some hybrid version of Scrum that may or may not be as effective, but may be more achievable given the limitations of the organizational context. (Jeff Sutherland, co-creator of Scrum, derisively labels limited adoptions as "ScrumButt".) The frequent debates that occur as to whether such teams are really doing Scrum are lively but largely irrelevant to the more important question of whether such teams are as effective and successful as they can be. This dynamic would look completely different if a team or organization was required to use Scrum via formal certification (something, fortunately, I have not heard of happening.) Then the question of effectiveness would take a back seat to the question of compliance.

The Asian philosophical concept of yin-yang seems to be the source of inspiration for the Japanese attitude that avoids this mismatch between process compliance and continuous improvement. This viewpoint holds that a process represents the best way known at this point in time to accomplish an objective, but to always search for a better process via continuous improvement. In my readings regarding Toyota and lean thinking, it appears that continuous improvement is the ultimate priority which all processes are subordinate to.

Some standards claim to be process-neutral in the sense that they do not require a specific process. For example CMMI specifies many process areas with objectives that must be met (the what), but allows you to specify the process that meets the objective (the how). However, the requirement to meet the objectives and to produce the evidence necessary to demonstrate that the objectives have been met both introduce significant constraints on what your processes are. In addition, in my experience the CMMI appraisers / consultants seem to have a bias towards particular processes being used. As a result pursuing such standards tends to push organizations towards a certain set of practices that may or may not be appropriate.

Just because I do not like process certifications does not mean I do not like the underlying standards. There are a lot of good ideas embedded in standards like ITIL and CMMI. But there are also aspects I think are often inappropriate depending on the context. Certification prevents you from intelligently picking and choosing - the full standard is forced upon you.

Visualizing package dependencies can be useful for a number of reasons:

• Reverse engineering the architecture of an application when documentation is limited, out-of-date, or non-existent.
• Automatically generating diagrams, saving the time required to draw them manually.
• To understand impacts when doing large-scale refactorings or changing implementations of components.
• To determine how to extract functionality for reuse.
• To support good architecture / design as espoused by Robert C. Martin in his book Agile Software Development, Principles, Patterns, and Practices, such as by eliminating cyclical dependencies or unwanted dependencies.

Here is an example package dependency diagram generated by Graphviz:

The source code provided below that was used to generate this example diagram is available from my Java Examples project on GitHub or as a packaged bundle on my Software page. The code can easily be reused: change the directory of compiled code scanned by JDepend ("dist/classes"), and change the root package used to limit the scope of the diagram ("com.basilv.examples.packagediagram").

package com.basilv.examples.packagediagram;
import java.io.*;
import java.util.*;
import jdepend.framework.*;

public class PackageDiagramCreatorApp {

  public static void main(String[] args) {
    createPackageDependencyDiagram();
    System.exit(0);
  }

  public static void createPackageDependencyDiagram() {
    Collection packages = analyzePackages();
    StringBuilder builder = generateGraph(packages);
    generateImage("packages", builder.toString());
  }

  @SuppressWarnings("unchecked")
  private static Collection analyzePackages() {
    JDepend jdepend = new JDepend();
    try {
      jdepend.addDirectory("dist/classes");
    } catch (IOException e) {
      throw new RuntimeException("Error adding directory for JDepend to analyze.", e);
    }
    Collection packages = jdepend.analyze();
    return packages;
  }

  private static StringBuilder generateGraph(
    Collection packages) {
    StringBuilder builder = new StringBuilder();
    builder.append("digraph packages {").append("\n");
    builder.append("node [shape=box];").append("\n");
    builder.append("rankdir=BT;").append("\n");
    Set drawnDependencies = new HashSet();
    for (JavaPackage javaPackage : packages) {
      String packageNodeName = getGraphVizNodeForPackage(javaPackage);
      if (packageNodeName == null) {
        continue;
      }
      builder.append(packageNodeName).append("\n");

      @SuppressWarnings("unchecked")
      Collection dependencies = javaPackage.getEfferents();

      for (JavaPackage dependency : dependencies) {
        String dependencyNodeName = getGraphVizNodeForPackage(dependency);
        if (dependencyNodeName == null
          || packageNodeName.equals(dependencyNodeName)) {
          continue;
        }
        String dependencyKey = packageNodeName + "->"
          + dependencyNodeName;
        if (drawnDependencies.contains(dependencyKey)) {
          continue;
        }
        builder.append(packageNodeName).append(" -> ").append(
          dependencyNodeName).append(" [weight=4]").append("\n");
        drawnDependencies.add(dependencyKey);
      }
    }
    builder.append("}\n");
    return builder;
  }

  private static String getGraphVizNodeForPackage(
    JavaPackage javaPackage) {

    String rootPackage = "com.basilv.examples.packagediagram";
    String packageName = javaPackage.getName();
    if (!packageName.startsWith(rootPackage)) {
      return null;
    }

    return packageName.replace(".", "_");
  }

  private static void generateImage(String fileName,
    String graphVizDotFormattedGraph) {
    try {
      File graphFile = createFileWithContents(fileName
        + ".txt", graphVizDotFormattedGraph);

      // This requires the Graphviz software to be installed -
      // see http://graphviz.org/
      String imageFileLocation = fileName + ".png";
      Runtime.getRuntime().exec(
        "dot -v -Tpng " + graphFile.getName() + " -o "
          + imageFileLocation);

      System.out.println("Image file available at "
        + new File(imageFileLocation).getAbsolutePath());
    } catch (IOException e) {
      throw new RuntimeException("Error generating image " + fileName, e);
    }
  }

  private static File createFileWithContents(
    String fileName, String graphVizDotFormattedGraph)
    throws IOException {
    File graphFile = new File(fileName);
    FileWriter writer = new FileWriter(graphFile, false);
    try {
      writer.append(graphVizDotFormattedGraph);
    } finally {
      writer.close();
    }
    return graphFile;
  }

}

I have faced this problem yet again recently and wanted to share two approaches that I have found useful: swarming and reserves.

Swarming

The basic idea behind swarming is to throw extra people at the outstanding work necessary to meet the milestone. In project management terms this typically involves sacrificing effort (budget) to save time (schedule). Effort is typically sacrificed due to two factors. First, more than the optimal number of people are working on one piece of work and some effort is lost through extra coordination. Second, people get involved doing work outside their normal expertise and thus require more time than normal.

An example of swarming is a scrum team pulling together at the end of an iteration to finish their committed stories. Developers could pair up to more quickly resolve the remaining defects. Once done testing their own functionality testers could jump in to help test other features they are unfamiliar with. Business analysts can delay working on future requirements to help test.

There are a couple of guidelines to swarming effectively. The first is to focus on the critical path - the activities that are predicted to take the longest and drive what the final completion date will be. This is typically accomplished by pulling people off of non-critical-path activities, but you have to be careful that you do not inadvertently slow down something else so much that it then becomes critical path.

The second guideline is similar: focus on work necessary for completing the milestone instead of non-milestone-related work. This may seem obvious, but in practice it is surprising how people will tend to choose the more interesting tasks over more tedious but necessary tasks (for example, updating end-user documentation prior to release). One way of achieving this focus is to clearly define what is part of the milestone versus what is not. One team I know uses a task board with post-it notes colored by release to visualize this.

Swarming only works when you have people on the team who are available to swarm - i.e. not all busy on critical path and/or milestone work. This leads us to the topic of reserves.

Reserves

The basic idea behind reserves is to hold back a portion of some type of resource so that it is available if an unforeseen situation arises - usually a problem, or sometimes an opportunity.

One very common approach is to set aside a period of time in the schedule as contingency. This may be an extra iteration per release, an end-of-release iteration to finalize the software for release, or simply extending the original estimated schedule by a certain percentage.

Another approach is to ensure that the team members are working normal hours - i.e. no overtime. If a time crunch does occur, then a short burst of overtime acts as a reserve. You have to be careful using this approach, as too much overtime is dangerous. You need to allow time for recovery afterwards, once the milestone is met, or risk disengagement, reduction in productivity, burnout, and loss of personnel.

A more advanced version of this approach is to ensure that each person is not normally 100% busy, as espoused by the book Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency by Tom Demarco. It is suspected that this is one of the reasons that companies such as Google provides 20% time for each employee to pursue personal projects. The first benefit of slack is that this minimizes time to completion by reducing delays due to work waiting in queues (as per lean theory). This thus reduces the chance of needing to struggle to meet a milestone in the first place. The second benefit is that, like overtime, people have the capacity to increase their efforts if a milestone is threatened

A related approach that has worked well in my own experience is to have some project members normally be assigned part time with flexibility in how they spend their non-project time. During crunch periods they can then easily increase the percentage of time they spend on project work.

I would like to hear your experiences regarding swarming and maintaining reserves - please share by leaving a comment.

Time

The first key is to make time to write. Ideally this should be a decent period of time each day during which you are undisturbed and at peak mental capacity.

The minimum period of time that I find effective is usually around 15 - 30 minutes. This is because it sometimes takes time to get into a good flow of writing. The maximum period of time that I find effective ranged from 45 minutes to 1.5 hours. This maximum is based on how well I can maintain my flow of writing and my motivation / discipline. If I get stuck, especially on a new section for which I do not have a clear outline, I tend to stop and let it percolate in the back of my mind. The longer the writing session goes, the more likely I am to run out of steam (motivation / discipline) - e.g. by getting distracted by email or my RSS reader. I find that practically this means that I am generally unable to continue writing for more than 1.5 hours.

So how do I find the time to write? I get up early each morning, before the rest of my family, and use the time to write. I sometimes write in the evening, but I find I often do not have the mental energy for this after a long day at work. So I tend to save the evenings for other activities, such as exercising, and dedicate the mornings to writing. Successfully doing this day after day takes motivation and discipline.

Motivation

The second aspect of my system is maintaining a high level of motivation. I have found that if I am not motivated to write regarding a particular topic or section then my progress is usually quite slow and I am easily distracted.

One way I keep my motivation high is extremely simple: I primarily choose topics that I am motivated to write about. When a new idea comes along, motivation to write about it is usually quite high initially and decays over time. So the ideal is to start writing about such ideas immediately. Other times I am wrestling with a topic at work or researching it on my own and gain some insights that I feel like sharing.

My motivation in these cases is linked to the goal of helping others learn and grow, which is part of my professional vision. In some cases, as I have started writing about an idea, I realized that I only had a sentence or two of meaningful content, which by my standards is not enough to share via an article. This typically rapidly deflates my motivation, and I move on to a different idea.

This concept of linking the act of writing to the achievement of a larger, more personally-meaningful goal is another valuable technique for boosting motivation. In the last year I have started to tackle some larger projects (which is one reason for the decrease in my posting frequency), and I find that having this linkage is extremely powerful in maintaining motivation over the long term. Having this motivation helps in dedicating the time to write.

Motivation is powerful, but it tends to fluctuate. When motivation fades, discipline is needed to maintain consistency of purpose.

Discipline

The final key to my system of writing is discipline - consistently applying deliberately-designed techniques and practices that help me write. Discipline has also helped me discover these methods through ongoing reflection and improvement on what is working versus what is not

A large part of discipline is consistency, which helps build up productive practices into positive habits. One example is my practice of getting up early each morning to write - I do this every morning, weekends as well as workdays.

I have found that there is an initial hurdle to overcome when sitting down to write, especially if I am struggling with a section. This difficulty often caused me to switch to email or the web as a distraction from which I tended not to emerge, resulting in me making no progress for that block of time. One practice I have developed to overcome this is to have a firm plan in mind the night before of what specifically I am going to write the next morning. As I get up in the morning, I immediately begin recalling my plan as a way to help prime my mind for action. Not only does this help me stay focused, but I also find it reduces the start-up time necessary to get into the flow, thus allowing me to be productive with even a short block of time.

Outlines are another practice I regularly use, even for short pieces of writing such as this article. These outlines are often quite informal, focusing on the main points I plan to write about. The purpose is not to identify all the content, or how it is organized - I often add more points that come to mind as I write, or restructure as the material comes together. Instead, I use outlines to help eliminate one of the causes of getting stuck - not knowing what content to write about. (It is still possible, unfortunately, to get stuck regarding how to write the content.) Outlines are also helpful in dividing the writing into sections: each section acts as a mini-milestone that I can aim to complete during a writing session, thus helping me remain motivated to write.

I have a technique for maintaining the flow while writing and minimizing distractions that involves the use of 'todo' labels. The general idea is that when I identify something to do or consider that falls outside my current focus, I document it with a 'todo' prefix. This keeps it preserved for later action, which helps to get it out of my mind now. Here are some common examples of how I employ this technique:

• When I want to link to an article on my website or elsewhere, I leave a 'todo'. Finding the link is often quite quick, but I do this to avoid distractions.
• When I have an idea for a piece of content for a future, unwritten section, I jot down a brief note with a 'todo' in that section, then return to what I was writing.
• When I get quite stuck on a particular paragraph or section, I leave a 'todo' comment and switch to another area. This is especially common when I have a long list of points outlined: I start writing the points for which I have the most motivation, leaving 'todo' flags on ones I skip. I then use the momentum I have built writing these more motivating points combined with the promise of almost being done them all to return to these few remaining points and grind them out.
• When I have an outline consisting of sections, like this article does, I start by writing down the section headings with 'todo' labels on each. This serves as a road-map for me to follow while writing. It also helps me see the overall structure of what I am writing.

This is my system for writing - it may not work for you. But I do suggest at least experimenting with some of these techniques and practices if you have not tried them before. I encourage you to leave a comment describing what practices or techniques you have found helpful.

• Upgrade to Java 5 compatiblity. This version of EnvGen will no longer work with Java 1.4.
• Migrate source code repository to GitHub at basilv/EnvGen. All known defects are now listed there.
• Fixed defect: Allow comments without commas in first row of csv file.
• Upgraded to FreeMarker 2.3.18.

EnvGen is an Ant task for generating different versions of the same file parameterized for different environments (e.g. development, test, and production). You can read more about EnvGen in the EnvGen Release Documentation.

As I often do, I found myself reflecting at the end of these days as to whether I had properly prioritized my time. I could have been resolving tricky technical issues and advancing progress towards our desired future state in terms of architecture, technology, and development practices. But I am only one person. I realized that the focus on growing our group of solid Java developers and architects would pay off in the long run by increasing the overall capacity of problem-solving and progress-advancing.

Another consideration is that the implementation of the improvements that I wanted to see happen are limited by the capabilities and knowledge of the developers who actually put these things into practice. In the past, I have experienced problematic developers who inadvertently introduce issues that others are stuck resolving and use older technologies and/or less-effective practices through lack of awareness or poor habits. Hiring highly capable developers, in contrast, makes it much easier to make progress on these improvements. So for a limited short-term investment of my time, focusing on hiring has staggeringly immense long-term benefits.

The above logic regarding my personal limits apply just as equally to my direct involvement with hiring. As much as I may like to personally interview each applicant, I simply do not have the time, especially during hiring booms. So my objective in working on the hiring process is to put a system in place that can, if need be, operate without my involvement while obtaining the same results.

I received feedback recently from one director that the new hires we have brought on since my involvement in hiring have been almost entirely working out just fine. This is very rewarding to hear and is confirmation that my efforts in building an effective hiring system have indeed paid off.

My standard for blog articles is to provide meaningful, substantial content - usually a minimum of 500 words. There are exceptions, such as this post :) I sometimes have short thoughts and insights to share or come across interesting articles or videos that don't meet this standard. I'm planning to assemble these tidbits into an occasional email newsletter (probably no more than once or twice per month) as a special bonus for subscribers to my list.

Visit basilv.com/psd-list-signup to subscribe or use the signup form on the right sidebar of my website. Your email address will of course never be shared and you can unsubscribe at any time.