Industrial Logic - Better Software Sooner

Replace Hard-Coded Notifications with Observer

Tue, 27 Jan 2026 18:00:00 +0000

I was recently programming with Claude Code when we encountered a familiar smell: subclasses of domain objects that existed only to handle separate concerns, like database persistence.

Together with Claude, we fixed the design flaw with Replace Hard-Coded Notifications with Observer—a refactoring from Refactoring to Patterns.

While Refactoring to Patterns is now over 22 years old, it’s as relevant as ever, even in the era of agentic software development.

The Smell

Our code has a domain class named StationActivity that tracks player activity during a game.

Two subclasses of StationActivity had been created solely to persist game data to Firebase.

Here’s one of those subclasses:

class PlayerStationActivity extends StationActivity {
    constructor(
        unresponsiveThresholdMs: number,
        private repository: PlayerRepository,
        private playerId: string
    ) {
        super(unresponsiveThresholdMs);
    }

    flip(batch: Batch): void {
        super.flip(batch);
        const lastFlip = batch.lastFlip;
        if (lastFlip && lastFlip.playerId === this.playerId) {
            this.repository.saveFlip(
                game.eventId!,
                game.id,
                round.roundNumber,
                station.index,
                batch.id,
                { dieIndex, newValue, timestamp, playerId },
                batch.startTime
            );
        }
    }
}

There was also a HostStationActivity that did nearly the same thing for bot players.

So we had two subclasses, both hard-coded to persist data to a repository, both passing seven parameters manually.

The subclasses weren’t expressing behavioral variation—they were just injecting data and doing it from domain classes.

The Refactoring

The fix had three parts:

1. Make the domain emit events instead of calling repositories directly:

// In game.updateDie()
this.emit('flip', { batch, flip: batch.lastFlip! });

2. Let the domain object graph provide context:

Instead of passing seven parameters, add a path getter to the batch that derives location from the object graph:

get path(): BatchPath {
    return {
        eventId: this.station!.round!.game!.eventId!,
        gameId: this.station!.round!.game!.id,
        roundNumber: this.station!.round!.roundNumber,
        stationIndex: this.station!.index,
        batchId: this.id
    };
}

3. Subscribe at the application layer:

// Player app: persist own flips
game.on('flip', ({ batch, flip }) => {
    if (flip.playerId === this.session.playerId) {
        repository.saveFlip(batch.path, flip, batch.startTime);
    }
});

// Host app: persist bot flips
game.on('flip', ({ batch, flip }) => {
    const player = batch.station?.player;
    if (player?.type === 'bot') {
        repository.saveFlip(batch.path, flip, batch.startTime);
    }
});

Why This Is Better

Aspect	Subclass Approach	Observer Approach
Domain purity	Domain knows about repositories	Domain just emits events
Parameter passing	7 params manually threaded	Data derived from object graph
Visibility	Side effects hidden in subclasses	Side effects explicit at app layer
Duplication	Two subclasses doing same thing	One unified `saveFlip()` method
Testability	Must mock repository in domain tests	Domain tests need no mocks

The Observer pattern keeps the domain pure. The Game class says “a flip happened” and moves on. It doesn’t know or care who’s listening or what they do with that information.

The Pattern in Action

This is the same pattern you use with DOM event listeners:

// DOM
button.addEventListener('click', (event) => {
    saveToDatabase(event.target.value);
});

// Domain
game.on('flip', ({ batch, flip }) => {
    repository.saveFlip(batch.path, flip, batch.startTime);
});

The domain object is the event emitter. Application code subscribes with callbacks. When something happens, all subscribers get notified.

This event-driven approach to persistence fits naturally in the JavaScript ecosystem, where EventEmitter patterns are idiomatic—from Node.js core to frontend frameworks. This decades-old refactoring isn’t just still applicable, it’s idiomatic.

Sidebar: How Does Claude Refactor Code?

When I mentioned refactoring “with Claude,” you might wonder how an AI actually edits code. Does it use an abstract syntax tree like an IDE?

No. Claude Code uses text-based editing—reading files as plain text, searching with grep, and making edits via string replacement. There’s no AST, no symbol resolution, no type checker.

Instead, Claude relies on pattern recognition from training data. When it sees a subclass that only calls super and adds a side effect, it recognizes the smell—not by parsing node types, but because it’s seen millions of similar patterns.

This is both powerful (works with any language, can make judgment calls) and limited (might miss a reference, no compilation guarantees). It’s closer to how a human reads and edits code than how an IDE refactors it.

The decades-old patterns still guide the refactoring. The tools just read the code differently now.

When to Apply This Refactoring

Look for these signs:

Subclasses that only add side effects (persistence, logging, analytics)
Multiple subclasses doing nearly identical things to different targets
Domain classes with repository dependencies they shouldn’t have

When you spot these, consider Replace Hard-Coded Notifications with Observer. Your domain stays focused on business logic, and side effects move to the application layer where they belong.

Safe Releases

Wed, 01 Oct 2025 10:00:00 +0000

Series index

Beginning Well

The old saying from Franklin Covey is:

Begin with the end in mind

Indeed, it is odd to close the series with a blog about deployment and release when they are so fundamental to success. Folks from the XP and code craft world see no reason to write a line of production code if they have not already established version control, a test pipeline, and automatic deployment.

Setting up a conventional application and pipeline is trivial and a bit tedious, so lately we find an LLM can do it quite handily for us. We also find it useful to create a few helper scripts.

Why start with deployment?

As we work iteratively and in “walking skeleton” style, we let our sponsors and/or customers see our progress and assess our work in the “daily” builds.

The running, tested code displays progress, inspires feedback, and does no damage to the company or customers.

There is an advantage in that our work is accessible and visible, which reduces the urge of other people to micromanage us.

We can also do gap analysis between the new system we are developing and whatever processes we may be automating or replacing. Whether it leads to a shorter development contract or a longer one, it should result in developing the program that serves the customers’ needs best.

Why not wait a fortnight to deploy?

This is a common question.

The answer is we CAN, of course. We can do nearly anything.

It’s more work if we wait

If the change we’re testing and releasing is only five or six lines of code in two or three files, then it’s inexpensive and effortless to release. If we have sufficient tests, then we are reasonably sure the changes are correct and complete (enough).

If we stockpile many hundreds (or, God forfend, thousands) of lines of change before we try to make a release, then it’s a different story. There are too many things that can go wrong. There can be hidden bugs, big changes in security, big changes in performance, or perhaps we have divergent changes where one set of changes makes another set invalid.

The work of making a big release is a huge effort. Most people who release only once every year saw that they would need as many as 6 months of hardening and testing to release. When they went to once a quarter, it took several weeks. When they went to once every two weeks, it took only days.

Small releases require small efforts.

It withholds value

Are you holding value hostage? If you are doing the most important work first, then the work that has been finished is pent-up value.

If you don’t release it, but rather wait for lower-value work to be done, then you are effectively holding high-value, completed work hostage for low-value work completion.

Does it make sense to withhold value from the project sponsors and users?

This is why we tend to ask what would happen if we released it today?

We would like for that to be met with a business decision instead of not fear and panic.

If it is always ready, it’s not scary.

Waiting Defers Feedback

Let’s reiterate the process value of early feedback.

If all of our progress is visible at least daily, our sponsors and managers can see that we are working and plan for our actual rate of progress.
Silence inspires micromanagement, whereas honest visibility inspires trust. Which do you find more pleasant and useful?
If they can touch it and try it, they are likely to spot UI and UX issues our scripts cannot.
Every deployment attempt tests the delivery system and all the Infrastructure As Code (IaC) work we’ve done.

So why would we want to wait?

But why start with a test runner?

We fully intend to deliver many times a week, possibly many times a day.

If we are delivering to a visible and accessible place (sandbox or production) where customers will interact with our work, then we don’t want to ship any disasters to them.

We use extensive automated testing to show that we’ve not broken any functionality (yet).

We won’t have time for extensive manual testing.

If this is true on the staging build, it’s doubly true of production.

What do we test?

If code makes any decisions or transformations, we should have tests.
Changes to business behaviors and flows may require tests.
Every important E2E function needs automated checks.
Important exceptions need automated checks.
Content changes typically don’t require automated tests.
Cosmetic changes don’t generally need automated tests.
Pure plumbing code probably doesn’t need unit tests, as problems there will show up in a higher level of testing.

How do we test?

Microtests are unit tests that help us write and refactor code.
Unit Tests may be larger in scope than microtests, checking larger behaviors in code.
Component Test are those for entire subsystems or microservices.
End To End (e2e) tests ensure that the UI allows a user to perform a function.
Human Beings can tell if the system is easy enough to understand, to use, and to fit into the daily flow. They can tell if it’s too lax about inputs or does not respect configuration options. They spot cosmetic issues, too.

One of the things we cannot tolerate is flaky tests. We never want to deploy broken code. If tests are intermittent, then we may have code that doesn’t always work. Maybe the test is poorly written, and the code is fine, but should we take that chance? Why aren’t the tests proof of function?

Likewise, slow tests will delay our deployments. We want all our code to be deployed without error ASAP. To ensure this, we like to run all our tests while we’re working on our development computers before pushing it to CI.

I don’t see any of these as dispensable. They all seem deeply important if we are to make a new release in minutes-to-hours from now.

Don’t Ship Half-Done Work!

We work in E2E slices, so the system is working early and often. Early slices are limited, but functionality grows more competent and complete over time. All the parts that have been written work as an integrated (if thin) whole.

Thin slicing like this is helpful if you wish to allow your sponsors to say that’s enough of this feature, let’s move on to something more important.

Top secret pro tips:

Sometimes our sponsors and BAs will realize after we’ve done the most important 50% of a feature that the remainder isn’t all that important
It can be helpful to our managers if they can cut scope in order to make their promise dates. If we deliver all the work they specified in one fell swoop, we rob them of the opportunity.
Remember what I repeatedly said about micromanaging?
You are more safely interruptable if you’re not trying to keep a huge changeset in your head.

You may find that the visual verification of progress and quality is far more valuable than the convenient familiarity of working on huge changes before eventually merging and deploying.

Is This Asking Too Much?

Many companies deliver software to production many times per day. Even our small website group can deliver a dozen times a day to production. This isn’t unheard-of or strange, it’s the practice known as Continuous Deployment.

Every team has constraints and context that may make some of these practices challenging. We understand that. It doesn’t make the work impossible.

I am a huge fan of working in a Trunk-Based way and delivering to production many times a day, but I have worked in regulated environments and mixed hardware/software where this kind of change isn’t as straightforward. In one project, we worked on delivering to a hardware-in-loop test environment at least until they were able to make a virtual hardware environment and deliver to it.

Do We Have To Work This Way?

No. I can’t tell you what you have to do. I’m not the boss of you.

OTOH, you may be having trouble making releases safely. The old saying is:

If you keep planting the same seeds, you will keep harvesting the same crop.

You may be in a team where the DORA metrics are being tracked. If so, then the person who really is your boss is telling you to work in a similar way so you improve the measured capabilities.

If you follow all the articles in this series, it should help you avoid or minimize many of the problems organizations face in moving to continuous delivery.

If you aren’t using the DORA metrics, you are still doing the right thing if:

You can develop the deployment process with tight feedback loops
Deployment becomes a frequent non-event
You are able to give real, verifiable proof of progress as it happens
You experience high pass rates through the quality gates

I wish you well.

Easy Integration

Wed, 26 Mar 2025 15:00:00 +0000

Series index

What Makes Integration Hard?

Software systems often need changes.

Sometimes, the goal is to reduce runtime costs. In other cases, the focus is on improving performance. Changes are frequently made to add new features or capabilities. Changes are needed to correct defects or inadequacies within the system.

A developer (or team) starts with an approved, stable version of the system’s source code. They create a set of changes (a “changeset”) to introduce new functionality or improve software behavior.

Any codebase will have many interacting constraints and rules. Much of the art of software development involves making new changes while remaining compatible with existing functionality.

Team A is working on a change and started with version A of the main development line, which works fine as an isolated change.

Team B is working on a change also based on version A of the main code line, and their change also works fine.

The problem shows up much later. While each change works independently, the two don’t work together.

As long as changes are isolated, the developers are unaware of the defects they injected.

You can't ensure compatibility if you can't see what's coming.

When an integration issue (an incompatible set of changes) surfaces, the developers must identify the underlying issues, locate the affected regions of code, and make the necessary design changes.

This rework usually falls to the team whose code changes were merged last. The first team’s changes integrate fine. This often makes the earlier-merged team seem superior to the teams whose code was equally valid but whose changeset was added just before the error was spotted.

When blame is given, it’s the “luck of the draw” who receives the blame. “It was fine before we added your code” doesn’t mean that the team merged last did a poor job. This is counterintuitive but worth considering. If the merges had happened in the opposite order, the other team would seem to be at fault.

The problem is that neither team could tell that their changes would not mesh with the other team’s changes.

Large changesets are not only more likely to hide incompatibilities, but they are also harder to diagnose and repair.

The larger the changesets are, the greater the risk of divergence and the more complicated the remediation may be.

If incompatibilities go unnoticed until late in the process, you can expect:

delayed releases
missed release schedules
flawed releases that must be rolled back
error reports from customers

These are common problems in corporate software development arising from the same underlying cause.

These issues don’t occur in single-developer projects or when a small, tight team works on a greenfield project. You simply don’t encounter these problems when only one group of people works on one codebase with a single changeset at a time.

People from small, nimble startups who join their first large corporate development team are unprepared for problems of scale. The same is true for university graduates who have mostly worked solo on class assignments or in small groups.

Integration problems are problems of scale.

Significant factors for these integration problems are:

The number of parallel changes
The size of each changeset
Deferring discovery of issues until it’s too late to recover gracefully

How To Reduce Difficulty

Limit The Number of Parallel Activities

Sharing a codebase across many teams requires finesses and solid product knowledge, but several time-honored approaches are used with some success.

Only do one thing to a product at a time. This is hard to do at scale, because organization grow primarily to achieve more than one thing at a time. If you can do only one thing at a time in one codebase, you eliminate all the chances for collision.
Parallel single new feature work with fixes. Defects are (by nature) identified in existing code, whereas new features tend to involve creating new code. There may be some overlap, but it is minimized. Separating work this way may run counter to business priorities when the focus is on new features, and there is still a risk that some oversized core modules will receive conflicting changes.
Ensure that parallel features affect different areas of the codebase. Doing this requires intimate knowledge of the source code since things that seem unrelated in the eyes of users (and non-technical managers) may connect tightly in the codebase. This also may run counter to business priorities by delaying highly desired but risky, changes.
Limit radial changes while implementing a feature can minimize merge collisions but also leaves the code with accumulated neglect (often called technical debt). There must be a follow-on task to return to the code and rationalize the design. This technique trades one set of technical risks for another, so be sure that you are solving the more troubling issue.
More modularity of design and the use of interfaces can help to make parts of the system independently modifiable. This is one of the useful features of Hexagonal Architecture.
While I have no significant experience in Event Driven Architecture, I am often told that it solves a lot of problems via modularity and repeatability, so that one can replace entire services without upset, and can run old versions of an app in parallel with new ones to judge readiness for switch-over.

Avoiding incompatible changesets is an attractive idea, but hard to achieve for management. It requires intimate knowledge of the code and a commitment to rationalizing the design after adding features.

I wouldn’t rely on trying to avoid the issue since doing so lacks safety.

Once a collision or incompatibility sneaks into the changeset, if you spot it, you pay the full price to remedy the situation.

Let’s see what else we can do either in addition or instead of avoiding the problem.

Reduce Period of Isolated Code Development

If we do less work between integrations, we have smaller changesets with a lower chance of collision. Less code also means less effort to isolate and correct a fault.

We can have smaller changesets if we break our features into smaller sub-features, each one a fully working subset of the planned, final result.

Only one small change per changeset makes each changeset small, and ensures they are completed more often.

This will mean more integrations, but each will be simpler.

We use many story-splitting techniques such as Role-Action-Context exercises, Use Case Mapping, Story Mapping, and Example Mapping. The specific technique you choose is less important than the act of splitting a feature into a series of fully-working “value drops” or “slices.”

This is iterative and incremental development.

The increment must be fully working, even if the feature implementation is not comprehensive. It is best if it has a suite of tests to ensure that it is fully working and that some other feature doesn’t “break” its functionality.

The next increment may broaden the functionality of that feature. In this way, the feature is “grown” in a series of releases, rather than “constructed” in a single pass. This means that we may be integrating just a handful of lines of code at a time, all tightly focused on specific behaviors in the code.

The small changesets and tight focus on a narrow bit functionality reduces the opportunity for collisions. It also reduces the cost of collisions.

We often see teams with smaller changesets completing incremental changes many times a day or at least several times per week.

Find Problems Sooner

We don’t want to find out about large, incompatible changesets on the last day before a release.

The longer we hold changesets in isolation (for instance, in git branches or on someone else’s machine), the longer it takes to discover incompatibilities.

This is because when we merge two branches, we produce a new version of the product’s source code; one that has never existed or been tested. There was no opportunity to discover the problem early and have a solution already in hand.

The term “continuous integration” (CI) is described and discussed at the C2 wiki:

What if engineers didn’t hold on to modules for more than a moment? What if they made their (correct) change, and presto! everyone’s computer instantly had that version of the module? You wouldn’t ever have Integration hell because the system would always be integrated. […] there wouldn’t be any conflicts to worry about.

Since this work is aided by a service that automatically rebuilds and re-tests (well, runs the automated checks and scans at least) when changes are made, people often think of those build-and-test servers as “CI Servers.”

CI is not the act of running a build-and-test server. CI is a developer behavior, not a technical appliance.

A CI server itself will usually fail if the code cannot be compiled or if a scanner reports a seriously ill-formed bit of code, but the server doesn’t know if the compiled code actually works and can carry out useful tasks.

We need fast, automated tests that can run in the CI server and flag issues as they occur.

This suite of tests needs to match the current functionality of the program, growing as the codebase grows.

We need some kind of continuous testing and continuous development of automated tests.

By the way, if you have achieved TBD and CI, there is little reason to not go all the way to Continuous Deployment (to a machine) or even Continuous Delivery (to production).

Is This Asking Too Much?

When one is accustomed to working with a small team in a greenfield project, this all seems overblown and inconvenient.

Without doing something to mitigate these problems, your large projects will suffer from unpredictability, errors, delays, defects, and rolled-back releases.

Does this happen? All the time.

Do they still release software? Yes, but releasing is full of problems.

These problems are inherent in the system of work.

If the system remains unchanged, then other attempts to mitigate the problem (like increasing the reporting burden or replacing programmers or managers) will make little positive difference and may worsen the situation.

If you change the system, you don’t have the same problems.

Do We Have To Work This Way?

You are doing the right thing if you can address these three problems:

Likelihood of incompatible changes
Size of incompatible changesets
Late identification of problems

If you are not having any problems with incompatible changes, late failures, merge collisions, or surprising errors, then whatever you are doing is working.

If you keep having these problems and don’t want to fix them, then that’s your choice, but maybe looking at the picture from this point of view may help you understand what your people are going through.

I wish you well regardless.

Shouldn't It Be Faster?

Tue, 18 Feb 2025 15:00:00 +0000

Too Slow

Developing software takes longer than we want.

In hopes of getting work done sooner, teams (or their managers) divide jobs into tasks that can be performed asynchronously¹.

The assumption that dividing up is faster is seldom challenged. It’s really a matter of faith rather than measurement. It is a familiar habit and the ticket systems support doing it this way, so it’s comfortable and familair.

Teams have been consistently losing the parallel game for years but never realized it because they never measured or had any way to reason about time being saved or lost; it is a matter of faith and familiary.

Sometimes it works incredibly well, though.

Is this random?

Is it luck?

Does it only work when the people are hard-core and good enough?

Is there some rational, logical reason why it’s only sometimes successful?

Amdahl’s Law

Gene Amdahl studied parallelization extensively. He was a computer scientist who worked with mainframe computers at IBM and later his own companies, including Amdahl Corportation (who made supercomputers).

He gave us a useful observation, represented as this equation:

Amdahl's equation.

S is the speedup you can expect
(p) the part of the process that can be sped up by working in parallel
(1-p) is the part of the process that cannot be made parallel
(s) is the factor of the speedup from going parallel

This might seem a little elementary on the surface of it - you have the process that’s not working parallel plus the parallel part. Hardly brilliant and hardly shocking at first glance.

But hold on a minute. There is more to think about.

Provided that the part of the process we want to speed up (p) is a critical area for performance, and that we have a meaningful speedup factor (s), and that we don’t have significant complications to the rest of the process (1-p part), we can see a measurable speedup by working in parallel.

One would assume that splitting our team members up, each taking a part of the program, and all working in parallel would be faster and result in software that we can deliver sooner.

That is not always how it happens, though. Let’s explore…

Maybe it’s not parallel?

The parallel part of the process begins when we launch the parallel processors, or in our case when we assign the parts of the work to our developers.

It ends when the last of the parallel parts complete.

In most software development organizations, especially the ones struggling with slow delivery, the work is not parallel, only asynchronous. The two are not the same.

Asynchronous work isn’t necessarily parallel.

This is examined in more detail in the blog post Scatter-Gather also on this site.

If any parts of the divided job are delayed or interrupted, that delay is included and extends the “parallel part” of the process.

What if the individuals are on different teams (perhaps a front-end and a back-end team) and each of those teams has different priorities? You can imagine if team A starts the work right away and finishes their big, but team B has more pressing concerns and doesn’t start their portion for a few weeks. This asynchrony is not parallel, so the overall time to completion is worsened.

The individuals don’t have to be on different teams for parallelism to be broken. It often happens that individuals have personal work queues, making it unlikely that their asynchronous tasks will coincide.

The higher the individual utilization, the more likely at least one “parallel” task will queue and wait, swallowing up any speed gains.

Consider a typical software development process where there are quality gates that may cause work to flow back to developers. If the work returns, then the task’s time expands. Backflow of work is a primary source of unpredictability and in this case it can consume any intended time savings.

What is the pass-rate, or First-Time Through rate, for your quality pipeline?² What is the turn-around time from when the work is submitted the first time until it is ready for integration?

For any of the above reasons (or dozens of others) the time from assignment until the final completion of the final part may stretch out in time and consume any advantage we may have gained from allegedly-parallelizing the tasks.

The Non-parallel Part

The equation treats the non-parallel part (1-p) as a single value, so one might not see that it includes extra work that preceeds and follows the parallel work.

In order to divide work up among developers, we have to have a pretty solid plan in place. The plan must be complete enough to ensure that these independently-built parts will all fit together at the end of the process.

Prework involves getting a design that is complete enough to be portioned, breaking it into sufficiently complete tasks to be done in parallel, assigning the work to be done³, and then (later) inspecting, assembling, and testing the finished work.

It may also involve marshalling or scheduling all the people who are to do the work, who may otherwise not be available to do it in parallel.

If the time for prework plus the time for parallel work plus the time to integrate and finish the work takes at least as long as it would have taken the available staff to do the work synchronously, then there is no speed advantage.

This happens more often than one might think.

Managing Parallel Work Is Harder

When work is divided, someone has to track it to make sure the prework is done, the parallel tasks are all being completed well, and that the integration is progression well at the end.

These steps require additional effort from the managers of the work, partly because the status of the job is fragmented and distributed.

It’s also a larger management burden because certain conditions must be met before the work can be reliably assigned for parallel development (cf “Definition of Ready”) and integration cannot be completed until all the parallel parts have finished.

Hopefully all the individual pieces will integrate cleanly and easily⁴.

But also, be aware that sometimes there may be dependencies between the jobs that would otherwise be parallel. Watching for these, resolving them quickly, and preventing them through clever design requires qttention and skill.

How Can We Win This?

We need only a few conditions:

A low-cost, low-ceremony way to divide work
We need a low-cost way to integrate
We need to ensure the work is actually done in parallel

It is hard for someone outside the development team to ensure these conditions are met. It’s not impossible, but I’ve seen it fail and degrade to asynchrony and extra work more often than not.

When it works, it works a treat!

Within the team it’s easier to make it work.

It helps if team has limited work in progress (WIP), and that the work to be done is assigned to the team as a whole rather than parts of the work being assigned to individuals.

Success is likely with front-end, back-end, UX, data services, reliability, and cloud architecture all focused on the same feature at the same time. This can happen even without pair programming or teaming.

The trick is that they are communicating, adapting, and integrating all simultaneously⁵.

When integration is continuous, there is little non-paralellizable work at the end of the period. This is possible when teams work in a Trunk-based way, or at least share the same branch among all the people working on a new feature.

It is a low-ceremony, high-energy way of working that provides quick results with lower incident of integration error or escaped defects.

You can win this game. Amdahl’s equation gives you all the information you need if you can apply it.

Where it is hard to win, or rare to win, perhaps it is best to work in a simple synchronous way and complete one task at a time.

Is this really a problem?

If you aren’t accounting for the additional non-parallel time and asynchrony, it is likely that you have been losing the parallel game.

A lot of people don’t consider the physics of the situation (as described by Amdahl’s law) and blindly assume that asynchronous work with late integration is the most efficient and speedy way to work. They lose this game continuously, but don’t realize that they’re losing.

There are definite engineering tradeoffs to be considered.

Is your team too slow and unpredictable even though you are doing the maximum amount of work in parallel, or because you are doing the work asynchronously with extra overhead?

That explains it!

Please consider the information here:

You may want to contact us for further training or coaching for Ways of Work that are more likely to give you the outcomes you want and need.

No, this isn’t part of our problem…

If you are not suffering the consequences of ill-considered paralellization you may find there are ways to improve your process by focusing on creating quality with superior desk disciplines. Consider our Industrial Logic Ways of Work, including:

There are many ways to focus improvement on having higher overall throughput and satisfying the DORA metrics. We can discuss these with you, and help you make better software, sooner.

Footnotes

Breaking up work this way is often erroneously described as “divide and conquer”. ↩
The pipeline’s pass rate is the product of individual gates’ pass rate. ↩
Often the person assigning the work goes to the extra work of dividing the work so that each task fits a specific person’s skill set (for efficiency’s sake). ↩
It is unlikely, since fragmented work is done by different people at different times, in different branches. ↩
I’ve observed that software development is done most well and most quickly when we focus all of the relevant skills on the task at hand. ↩

Agentic Extract Method

Mon, 17 Feb 2025 15:00:00 +0000

Extract Method is one of the most commonly used automated refactorings. Many of us have been using it for well over two decades in tools like Eclipse and IntelliJ (and its family of IDEs). It’s a workhorse.

And, while I appreciate it for what it can do, I’ve been tired of what it can’t do for a long time now. I wrote about this problem in a 2014 blog called Improving Extract Method.

In this blog, I’m going to show you how I used ChatGPT 03-mini and Junie (join the EAP for this excellent tool) to perform a more intelligent Extract Method. The code we’ll look at is from the Parrot Kata, a code example from Martin Fowler’s Refactoring book, which was turned into a Kata (in many languages) by Emily Bache.

The code I want to refactor is the test code. As you can see below, for each test in ParrotTest, a Parrot instance is instantiated with specific values. As you can also see, there are three types of parrots (European, African and Norwegian):

@Test
public void getSpeedOfEuropeanParrot() {
    Parrot parrot = new Parrot(ParrotTypeEnum.EUROPEAN, 0, 0, false);
    assertEquals(12.0, parrot.getSpeed(), 0.0);
}
@Test
public void getSpeedOfAfricanParrot_With_One_Coconut() {
    Parrot parrot = new Parrot(ParrotTypeEnum.AFRICAN, 1, 0, false);
    assertEquals(3.0, parrot.getSpeed(), 0.0);
}

@Test
public void getSpeedOfAfricanParrot_With_Two_Coconuts() {
    Parrot parrot = new Parrot(ParrotTypeEnum.AFRICAN, 2, 0, false);
    assertEquals(0.0, parrot.getSpeed(), 0.0);
}

@Test
public void getSpeedOfAfricanParrot_With_No_Coconuts() {
    Parrot parrot = new Parrot(ParrotTypeEnum.AFRICAN, 0, 0, false);
    assertEquals(12.0, parrot.getSpeed(), 0.0);
}

@Test
public void getSpeedNorwegianBlueParrot_nailed() {
    Parrot parrot = new Parrot(ParrotTypeEnum.NORWEGIAN_BLUE, 0, 1.5, true);
    assertEquals(0.0, parrot.getSpeed(), 0.0);
}

@Test
public void getSpeedNorwegianBlueParrot_not_nailed() {
    Parrot parrot = new Parrot(ParrotTypeEnum.NORWEGIAN_BLUE, 0, 1.5, false);
    assertEquals(18.0, parrot.getSpeed(), 0.0);
}

The Parrot Kata is about introducing polymorphism into the code, and I have an approach to doing that which is driven outside-in, from the test code to the production code.

I begin by extracting Parrot instantiation into its own creation method via the Extract Method automated refactoring. The trouble is, IntelliJ cannot do this the way I want. Here’s what it gives me when applying Extract Method on the first parrot instantiation of an African parrot:

@Test
public void getSpeedOfAfricanParrot_With_One_Coconut() {
   Parrot parrot = createParrot();
   assertEquals(3.0, parrot.getSpeed(), 0.0);
}

private Parrot createParrot() {
   return new Parrot(ParrotTypeEnum.AFRICAN, 1, 0, false);
}

@Test
public void getSpeedOfAfricanParrot_With_Two_Coconuts() {
   Parrot parrot = new Parrot(ParrotTypeEnum.AFRICAN, 2, 0, false);
   assertEquals(0.0, parrot.getSpeed(), 0.0);
}

@Test
public void getSpeedOfAfricanParrot_With_No_Coconuts() {
   Parrot parrot = new Parrot(ParrotTypeEnum.AFRICAN, 0, 0, false);
   assertEquals(12.0, parrot.getSpeed(), 0.0);
}

Notice how it did not change all African parrot instantiations to use the new creation method? This happened because the other instantiations of African parrots happen to pass in different constructor values. What it generated wasn’t even close to what I wanted and therefore I did not even bother to rename it to createAfricanParrot(). The Extract Method refactoring has been incapable of figuring this out for way too long.

It’s time for a change.

Extract Method with ChatGPT

Let’s see how AI can help. Using ChatGPT 03-mini, I give it the ParrotTest source code and ask it to do the following:

“In the ParrotTest, please extract a single creation method (a method that creates a Parrot instance) from all of the code that instantiates a Parrot class.”

It does the work in seconds, giving me the following:

@Test
public void getSpeedOfEuropeanParrot() {
    Parrot parrot = createParrot(ParrotTypeEnum.EUROPEAN, 0, 0, false);
    assertEquals(12.0, parrot.getSpeed(), 0.0);
}

@Test
public void getSpeedOfAfricanParrot_With_One_Coconut() {
   Parrot parrot = createParrot(ParrotTypeEnum.AFRICAN, 1, 0, false);
   assertEquals(3.0, parrot.getSpeed(), 0.0);
}

private Parrot createParrot(ParrotTypeEnum parrotTypeEnum, int numberOfCoconuts, 
   int voltage, boolean isNailed) {
    return new Parrot(parrotTypeEnum, numberOfCoconuts, voltage, isNailed);
}

Now I have one place in this test code where I instantiate parrots. It’s good, and I’m glad AI can produce this for me, but I actually want it to do more. What I want is one creation method for each parrot type.

It can do that too. I begin with the original test code and ask ChatGPT the following:

“Instead of making one createParrot method, can you make three: one for each ParrotTypeEnum that is instantiated?”

It immediately does the work and gives me the following:

private Parrot createEuropeanParrot(int numberOfCoconuts, double voltage, boolean isNailed) {
    return new Parrot(ParrotTypeEnum.EUROPEAN, numberOfCoconuts, voltage, isNailed);
}

private Parrot createAfricanParrot(int numberOfCoconuts, double voltage, boolean isNailed) {
    return new Parrot(ParrotTypeEnum.AFRICAN, numberOfCoconuts, voltage, isNailed);
}

private Parrot createNorwegianBlueParrot(int numberOfCoconuts, double voltage, boolean isNailed) {
    return new Parrot(ParrotTypeEnum.NORWEGIAN_BLUE, numberOfCoconuts, voltage, isNailed);
}

@Test
public void getSpeedOfEuropeanParrot() {
    Parrot parrot = createEuropeanParrot(0, 0, false);
    assertEquals(12.0, parrot.getSpeed(), 0.0);
}

@Test
public void getSpeedOfAfricanParrot_With_One_Coconut() {
    Parrot parrot = createAfricanParrot(1, 0, false);
    assertEquals(3.0, parrot.getSpeed(), 0.0);
}

@Test
public void getSpeedNorwegianBlueParrot_nailed() {
    Parrot parrot = createNorwegianBlueParrot(0, 1.5, true);
    assertEquals(0.0, parrot.getSpeed(), 0.0);
}

This is exactly what I want at this stage in the Parrot Kata. Had this been client code or my own organization’s code, I may not have wanted to expose it to a public AI tool. Ideally, we could get similar AI help for Extract Method from directly within our IDE. That’s where Junie comes in.

Extract Methods with Junie

Junie is a new AI plugin by JetBrains, currently in an Early Access Program. I’ve been playing with it and am really enjoying how it works. For this blog, I’ll show you how I interact with it to get help with Extract Methods.

Inside of IntelliJ, I ask Junie to do the following:

“In ParrotTest, create one Creation Method for each Parrot Type, passing in number of coconuts, voltage and isNailed, even if they aren’t needed.”

Junie chat window inside of IntelliJ

After a few seconds, It gives me what I want:

private Parrot createEuropeanParrot(int numberOfCoconuts, double voltage, boolean isNailed) {
    return new Parrot(ParrotTypeEnum.EUROPEAN, numberOfCoconuts, voltage, isNailed);
}

private Parrot createAfricanParrot(int numberOfCoconuts, double voltage, boolean isNailed) {
    return new Parrot(ParrotTypeEnum.AFRICAN, numberOfCoconuts, voltage, isNailed);
}

private Parrot createNorwegianBlueParrot(int numberOfCoconuts, double voltage, boolean isNailed) {
    return new Parrot(ParrotTypeEnum.NORWEGIAN_BLUE, numberOfCoconuts, voltage, isNailed);
}

@Test
public void getSpeedOfEuropeanParrot() {
    Parrot parrot = createEuropeanParrot(0, 0, false);
    assertEquals(12.0, parrot.getSpeed(), 0.0);
}
@Test
public void getSpeedOfAfricanParrot_With_One_Coconut() {
    Parrot parrot = createAfricanParrot(1, 0, false);
    assertEquals(3.0, parrot.getSpeed(), 0.0);
}

@Test
public void getSpeedOfAfricanParrot_With_Two_Coconuts() {
    Parrot parrot = createAfricanParrot(2, 0, false);
    assertEquals(0.0, parrot.getSpeed(), 0.0);
}

You may wonder why I had to tell Junie what parameters to pass in to each Creation Method. Junie is clever. If I didn’t specify that, Junie would omit any parameters that it could simply hardcode into the constructor calls, if they happen to be the same values for all instantiations (e.g. a “false” value for isNailed).

BTW, if and when Junie does something you don’t want, you can simply tell it to undo the work and it will gracefully retreat to the prior state of your code.

So far, I’m loving how Junie performs. I’d encourage you to play with it by getting into the Early Access Program.

Conclusion

Refactoring has traditionally been an atomic operation: you implement or execute one refactoring at a time. To complete the design change you want, this may involve performing a sequence of refactorings. I’ve been doing that for well over 2 decades now and frankly, I’m tired of it. I’m lazy and I simply want my refactoring tools to do more. Thanks to AI, that’s now possible.

Today, I’m using a combination of automated + agentic refactoring when improving the design of code. While I can ask AI to perform very large steps, that tends to be more dangerous. So I’m asking it to perform slightly more powerful refactorings. And keep in mind that I could not and would not do this if I didn’t have thorough, automated checks to ensure that my changes preserve behavior. With or without AI, refactoring must be safe.

The use of AI in refactoring is a rich new area, changing rapidly. If we use it wisely, it can help us improve the design of code more efficiently and effectively. I for one am overjoyed that I am no longer stuck with the limited functionality in the available set of automated refactoring tools. AI has the potential to help us make more intelligent design transformations. We just need to make sure that these transformations are safe.

Fast Feedback

Thu, 13 Feb 2025 15:00:00 +0000

Series index

We’ve covered the Clean Start protocol and how to make for Easy Recovery already, and it’s time for us to discuss Fast Feedback.

When someone in engineering talks about quality, they usually mean that the item in question

performs a useful function
does so reliably
has low cost
has easy maintainability

We have many arguments in social media where people take the word “quality” in a different sense, to imply aesthetic purity, opulence, luxury, and outrageous lists of features.

When we talk about feedback, we take the engineering sense of quality. We want the product that we deliver to fulfill useful functions reliably and affordably, with easy maintenance.

In order the product we produce fulfills our intentions we need to measure our effectiveness with feedback.

We don’t want to operate from unchecked self-confidence alone (although apparently some do).

To this end, we need both in-session feedback and external feedback. Let’s take an inside-out tour of this important topic.

In-Session Feedback

Programmers work within ambitious schedules, even when they set their own. This often motivates them to work in big batches and huge steps when a smaller, more measured approach is preferable.

Working in small steps isn’t a virtue unto itself, but rather enables us to have better feedback loops to maintain and improve the product’s quality and our productivity.

The tight “inner loop” of software development is the interaction of the programmer(s) and the IDE when creating code.

Consider the life cycle of a defect:

Period a governs period b

A defect is injected when some programming decision is made in the codebase or its dependencies that does not pan out for all the uses of that code.

Some defects are “shallow”; they are easy to fix now, and will still be easy to fix a year from now.

Some defects are “deep”; they are hard to reproduce, and understand, and occasionally hard to fix or work around. It may take many hours or days to figure out what behavior in the code causes defective behavior, which is then remedied in a few lines or characters of code. Some of us have had the unpleasant job of explaining why our multi-day struggle resulted in changing such little code¹.

In general, period A governs period B.

If (after a clean start) I’ve written only one line of code before the alarm bells go off, I know I have only one line of code to examine. The error can not be anywhere else.

If I undo the change immediately, everything returns to its original function with no errors or failing tests.

The larger the set of changes before the defect is found, the more work it takes to discover what is wrong.

This is painfully true if changes from multiple authors are all merged and only then the newly composed application is found to have a logical error.

I want to know whether every change is valid, and I want to know within seconds.

This is the goal of what we call continuous testing. We want some element of our process to surface errors as soon as they occur.

To support continuous testing, some IDEs provide a feature to rerun the tests after every code change.

Sometimes auto-testers are included with a language’s unit testing framework, running the tests whenever a file has been saved. Often these run in a separate window. Most programmers have multiple screens or at least enough screen real estate to have multiple windows open and visible at once, so this is “good enough.”

Feedback loops in the editing session can be either one of two things:

Fast²
Ineffective.

There is no room for slow checks and tests in the tight inner loop of coding.

Developers will run slow tests less often and do more work between checks. This increases period A, and period B ( searching for causes of defects) naturally expands.

If the tests are too slow, hurried developers will disable them.

We can do better than that.

We have several tools at our disposal to help:

We work together to produce code, which not only helps us spot any mistyped code and logical errors, but we spot and correct thinking errors before we even type the code – preemptively. That’s faster than “immediate.”
Any good IDE has syntax highlighting to spot common spelling and punctuation errors. This is immediate. I use a lot of JetBrains editors, and I respond to warnings (shown as a yellow triangle in the upper-right corner). I try to have zero warnings and zero errors.
Most editors support an error checker (“linter”) that will spot common code issues (“code smells”), security vulnerabilities, and potential errors. I use SonarLint in most of my editors and recommend doing likewise.
Microtests ensure that the various functions and classes still work as we intended. This feedback allows us to refactor the code freely into any other form that passes the same tests³. The period of these tests is sub-one-minute, often less than 10 seconds, so we use these with an “auto tester” that runs the tests whenever the code changes or a file is saved.
We often have suites of integration, component, or large unit tests. These take longer to run, generally but sometimes fast enough is fast enough. If they are too slow for the continuous testing part of our workflow, we can still run them periodically, at least every 20 or 30 minutes.
Pre-commit hooks are often used to run slower tests before a change is allowed to be made permanent.

The faster the tests are, the more useful they are for continuous testing and refactoring.

We use TDD as a way to establish microtests so that when we write any line of code, we already have test coverage and can immediately refactor it safely.

Teaming, continuous testing, TDD, and a well-configured IDE give us the safety we need to apply CI & CD.

Of course, all of this is in service of writing and refactoring the code so we deliver a solution with low maintenance costs and high code virtues.

External Feedback

I listed the in-session feedback first not because it’s the most important for the success of a project, but because it’s the feedback that most of the readership will engage in the most often. It’s important to build the thing right and to move quickly with changes so that our external feedback can be fast, too.

Imagine that you’re rolling up a yoga mat. The inner diameter of your first roll has a huge effect on the outer diameter of the fully-rolled mat.

Outer diameter is constrained by inner diameter

In the same way, if development takes weeks or months, we can’t get user feedback very often. Quick deployments lead to faster feedback for companies willing to capitalize on the opportunity.

So there are a few different things to offer here.

Quality and speed won’t necessarily lead to success. Products tend to die either for reasons of internal politics or because they fail to achieve market fit.

User Experience disciplines have goal-seeking feedback mechanisms. Some can be applied prior to beginning development. Interviews, research, user persona, MVPs, – it’s an important skill set for ensuring that you are creating a solution that might have a market fit.

The idea of “plan it all up front” has a decidedly poor history due to its “all or nothing” bests on the first product design chosen. Incremental efforts have about a 40% higher chance of success.

It is not enough to be “right to begin with.” We have to adjust on-the-fly to the changes in needs and expectations if we wish to remain viable and competitive.

Not only are software solutions dealing with wicked problems (multiple stakeholders, different intentions and needs, different perspectives, etc) but also providing a solution can change the nature of the problem.

Given a good way to collect payments, the shop suddenly finds that they need financial analysis, costing based on materials and labor, and business forecasting based on the payment data. In this way, an available solution uncovers or creates new needs.

When touchscreen devices became commonplace people suddenly “needed” mobile apps to do things they used to do on desktop computers. This need was generated by changes in the larger consumer electronics market not from any direct forces.

Thankfully, UX doesn’t end at initial analysis and planning, but continues with the Build-Measure-Learn cycles and the whole toolbox of product evolution techniques.

We need the development teams to provide rapid build/release cycles to support the UX efforts to find and improve our products’ market fit.

This relates back to our defect lifecycle diagram (above): how can we shorten the period between making a product decision and finding out if it makes a positive difference?

Does it matter?

Not every company needs or wants to be nimble and experimental. You might have a job that doesn’t have any interest in the speed, quality, or market fit for your work. That might actually be quite nice.

If you work for a company that is not interested in requesting feedback or acting upon it, evidenced by making no move to enable it, then this may be wasted effort.

On the other hand, the defect life cycle is affecting you. If you have to spend time remediating bugs via debugging, or loading the application with print/log statements, or bisecting commit histories and various other “extreme sports” of the programming field then you may find immediate relief and value in using various technical feedback techniques like the ones described here.

It might make your work easier and more fun. Who doesn’t like that?

Notes

My colleague had cast a C “long” through a “short” integer in one case. It was easy to understand, hard to find. ↩
See Unit Tests Are First for more on this topic. ↩
I won’t say that TDD improves design. it lets me change design freely, so improvement is always a possible choice. ↩

The ROI of Saving 700+ Hours Per Month

Tue, 04 Feb 2025 01:31:16 +0000

How Teams Waste 700+ Hours/Month

By Wyatt Sutherland

Time is the most valuable resource for any software development team. Yet, despite the best intentions, teams lose over 700 hours per month—17+ weeks— to inefficiencies, unnecessary meetings, redundant processes, and outdated workflows.

If you’re feeling the strain of tight deadlines and overloaded sprints, you’re not alone. The good news? You can reclaim those lost hours—without hiring more staff or working overtime.

The Shocking Truth About Your Team’s Time

Many development teams unknowingly and habitually waste hundreds of hours each week on tasks that don’t contribute directly to delivering value. Here are some of the biggest culprits:

Unnecessary Meetings: Team members spend an average of 32 hours per month in meetings which could have been achieved with short sync-ups and improved ways of working (WoW).
Manual Processes: Repetitive tasks, such as code reviews, deployments, and regression testing, take hours to complete, while better working methods or automation could handle them in minutes.
Poor Workflows: Quality gates catch issues too late in the development cycle, resulting in significant rework, and overhead, costing your organization at least twice as much and often more.
Lack of Collaboration: Blockers, insufficient requirements, unknown acceptance criteria, handoffs, waiting on others, and unclear communication channels lead to confusion and costly delays.
Context Switching: Constantly jumping between tasks can reduce productivity by up to 40%, costing hours of lost focus daily.

How to Get Those 700 Hours Back

We at IL are a pragmatic group and know that reclaiming those lost hours doesn’t require drastic changes. Instead, we recommend a series of strategic adjustments that can yield significant time savings. Here are some simple recommendations:

Subdue the Meeting Monster

Evaluate your meeting schedule and eliminate or consolidate unnecessary gatherings. Use async communication tools like Slack, Teams, Loom, Discovery Trees, or project management platforms to replace long status meetings with concise updates. At IL we use Slack and Discovery Trees for our initiatives and apply The Law of Two Feet (leave meetings) if we’re not learning, or contributing.

Quick Win: Implement a “No-Meeting Wednesday” policy and watch productivity soar. After 3-4 weeks, add another No-Meeting Day. We find this approach highly effective and beneficial to our clients.
Automate Everything Possible

Leveraging automation tools for everything from CI/CD pipelines to automated testing and code formatting can drastically reduce manual workload and free up hours for meaningful work.

Quick Win: Use tools like Jenkins, CircleCI, or GitHub Actions to automate your build and deployment processes. Use an IDE with solid refactoring tools and use the tools. Too often we observe folks manually refactoring when their IDE is faster and not as error-prone as a human.

We’ve found in-IDE scanners like SonarLint help us find common issues and vulnerabilities during our code-editing sessions. These tools usually will not only mark the problem but also explain it so you can avoid it down the line.

Useful automation doesn’t have to be an expensive proposition. We get frequent small boosts in speed using simple tools like the gnext and gdone functions from Save Your Game or the prepare script described in The Clean Start Protocol.
Optimize Workflows

At IL, we believe building in quality early into your process is a critical step to achieve as soon as possible. You’ll want to emphasize shift-left testing during the coding phase for improved software quality, better tests and coverage, and faster time to market. This approach will reduce overlapping redundant tests usually found in teams that have a separate testing unit, and shift-left testing will help get the software to production with fewer escaped defects.

With many of our clients, we have seen test engineers writing code and in some cases, clients have added testers as full-time team members and eliminated their testing department.

Quick Win: Ask your testers to collaborate with the developers closely during design and development.

Gamify Testing: Challenge testers and developers to compete in fixing defects during the coding phase using leaderboards, rewards, or recognition to reduce defects found in the QA phase. HINT: Look into Stryker for automating your mutation tests to see insufficient test coverage.
Increase Collaboration

Another big source of wasted time is waiting on dependencies. Instead of simply managing these dependencies, focus on identifying your value stream partners. Rather than handing tasks off to another team or waiting for them—much like the principles of shift-left testing—aim to encapsulate your dependencies and work collaboratively through ensemble development.

If you’ve not experimented with ensemble development we recommend that you try. There’s a sweet spot ensemble size and it depends on the context. We’ve written many blogs on this modern practice. Visit IL’s blog posts to learn more.
Reduce Context Switching

Each switch costs time—often 15-30 minutes per switch, which adds up quickly.

Dedicate specific time blocks for deep mindful work and make sure folks know it’s non-interruptable time. This allows your brain to stay engaged in one type of task at a time.

Limit your Work in Progress (WIP) by focusing on fewer tasks at once. At IL we use a Kanban/Swarm board and WIP limits. Setting WIP limits will help avoid overloading simultaneous tasks and reduce interruptions; this approach requires discipline so try not to let the old habits creep back in. We also practice taking a short break about every 50-60 minutes so our thinking remains fresh and uncluttered.

Quick Win: Similar to “Subdue the Meeting Monster,” set aside a portion of each day where the entire team works on focused, uninterrupted tasks. Start with 2-3 consecutive hours in the morning and see if you can extend the length over time. Even a single day of deep focus can significantly boost productivity and reinforce the habit of reducing context switching. Communicate your intent with your manager so they support your efforts across the org.

Where Did 700 Hours Come From

Seven hundred hours might sound like an exaggeration, but when you break it down, it’s actually a conservative estimate for a seven-member team—a product manager, a designer, and a mix of developers and test engineers—working in a traditional software development model.

At IL, we refer to this traditional approach as Scatter-Gather—where engineers receive their assignments, work independently in silos, and then come together later to integrate their code into the larger system. While this approach might feel comfortable, it introduces inefficiencies at nearly every stage of development.

We’ve worked with hundreds of companies, and the vast majority still operate in a Scatter-Gather model without fully examining the hidden costs of their workflow. So, we did the math.

Our calculator accounts for time spent on Non-Value Tasks (NVTs)—things like coordinating meetings, knowledge sharing sessions, manual regression testing, integrating work, handling pull requests, hardening sprints, waiting on blockers, acceptance criteria validation, design clarifications, triage and bug fixing, etc.

For this blog’s example, we analyzed a team’s average weekly overhead, calculated as follows:

Each team member spends on average 28 hours per week on NVTs.
The team on average completes 10 features/stories/tasks per week.
Work goes through 5 quality gates, with a 90% pass rate at each gate.
Due to this compounding failure rate, 41% of tasks are returned for rework (calculated as 1-(90%^5)).
Each returned task requires 3 hours of rework on average.
That results in 4.1 returned tasks per week, leading to 13 additional hours of rework per week across the team.
Total weekly overhead in a traditional model:
- Non-value tasks: 28 × 7 = 196 hours
- Rework due to returned tasks: 13 hours
- Total: 209 hours per week

Now, scaling up to a month:

The average number of weeks per month is 4.33.
Total monthly overhead: 209 × 4.33 ≈ 900 hours.

The Impact of an Optimized Approach

In contrast, teams that shift to optimized ways of working drastically reduce this overhead. When teams implement a streamlined workflow that minimizes handoffs, reduces waiting time, and improves collaboration, the average weekly overhead drops to 43 hours per team and better.

Optimized WoW total monthly overhead:
- 43 × 4.33 = 186 hours per month
Total waste prevented per month:
- 900 (traditional WoW) - 186 (optimized WoW) = 714 hours saved per month.
- Rounding adjustments bring this to 719 hours of recovered productivity.

The Hidden Costs of Traditional Development

Using real-world data, we built the Way of Working – Waste Cost Value Calculator to quantify inefficiencies.

Why This Matters

Data doesn’t lie. Across teams we’ve analyzed, the cumulative impact of these inefficiencies exceeds 700 hours per month—the equivalent of nearly 18 full-time workweeks or hiring four additional full-time developers. But instead of throwing more people at the problem, the real solution lies in optimizing how work gets done.

By shifting to more streamlined ways of working, teams reclaim lost time, ship higher-quality software, and achieve more—without overtime, burnout, or budget increases.

The Bottom Line

Reclaiming 700 hours per month isn’t a fantasy—it’s achievable with the right approach. By eliminating waste, automating tasks, and optimizing workflows, your product delivery team can focus more on innovation and delivering value rather than getting bogged down by inefficiencies.

Want to put these strategies into action? Start small, measure progress, and watch your team’s productivity skyrocket.

Waste Cost Value Calculator (Traditional vs. IL WoW)

Cost Efficiency Gains with IL WoW: A Summary

This section showcases the substantial time and financial savings a seven-member team can achieve by shifting from a traditional WoW to an IL WoW. The calculations assume a team composed of a product manager, a designer, and a mix of developers and test engineers.

719 hours saved per month – Optimized workflows and reduced inefficiencies free up valuable development time.
$77,328 monthly overhead cost – The financial burden of inefficiencies in traditional WoW.
$1.16M annual overhead cost (traditional WoW) – The yearly cost of maintaining outdated processes.
$240K annual overhead cost (IL WoW) – A dramatic reduction in waste after IL WoW adoption.
79% reduction in annual operating cost – A significant efficiency gain per team.
$927,940 in annual savings: The projected cost reduction after one year of IL WoW implementation.

This breakdown highlights the massive potential for teams to improve efficiency, lower costs, and boost productivity by embracing modern, optimized ways of working.

From Waste to Efficiency: Cutting Overhead with IL WoW

This section highlights the time and financial costs associated with non-value-added tasks (NVT), delays, and inefficiencies in traditional workflows vs IL WoW.

209 hours of overhead per week → 43 hours per week: Teams operating under traditional workflows lose an average of 209 hours per week to inefficiencies. By adopting IL WoW, this is reduced to just 43 hours—a dramatic improvement.
75% of working hours lost → 15% lost: Under traditional WoW, 75% of available time is consumed by overhead, compared to just 15% when IL WoW is implemented.
$22,468 overhead cost per week → $4,623 per week: The financial burden of wasteful processes is significantly reduced, cutting weekly overhead costs by nearly 80%.

This data underscores how inefficient workflows eat away at productivity and budgets. By addressing waste, teams can reclaim valuable time and resources for more impactful work.

The Hidden Cost of Rework: Quality Control Inefficiencies

This section highlights how reducing unnecessary quality gates minimizes rework, streamlines processes, and accelerates delivery. By cutting down on redundant reviews and revisions, teams can improve efficiency, reduce wasted effort, and focus on delivering higher-value work.

🛠️ Tip: Build quality in up front rather than relying primarily on downstream verification.

Traditional Ways of Working:

Teams pass through five quality gates, each requiring approvals, reviews, and handoffs.
Despite a 90% pass rate, 41% of work still requires rework, adding 13 hours of returned work per week per team.
Multiple gates cause delays and waiting, leading to inefficiencies and bottlenecks.

IL Ways of Working:

The team works in an ensemble model, collaborating throughout development, eliminating the need for multiple gates.
Only one quality gate remains—customer or stakeholder approval when the product goes live.
With a 95% pass rate, only 5% of work requires rework, reducing overhead to just 1 hour per week per team.
This shift ensures that work is completed correctly in real time, avoiding unnecessary handoffs, approvals, and bottlenecks.

By adopting IL WoW, teams eliminate redundant approval steps, reduce delays, and significantly cut rework time, leading to faster delivery and higher-quality outcomes.

Before & After: Reducing Non-Value Tasks with IL WoW

This section highlights the significant reduction in non-value tasks (NVTs) achieved through IL WoW. By minimizing time spent in meetings, waiting, knowledge transfer, manual regression testing, and integrating work, teams can focus on high-impact tasks that drive innovation and efficiency.

Coordinating Meetings: Teams in IL WoW spend an average of just 3 hours per week, typically in 10–15 minute daily sync-ups. The end-to-end value stream is embedded within the team, minimizing external dependencies and distractions.
Time Spent Waiting & Knowledge Transfer: Whole-team ensemble development ensures continuous collaboration with zero to minimal interruptions, reducing workflow delays and unnecessary knowledge handoffs.
Integrating Work: Trunk-based, small-batch development with frequent integrations (multiple times per day) eliminates the overhead of complex, large-scale merges.
Manual Regression Testing: IL WoW embraces XP development principles, with engineers practicing Test-Driven Development (TDD) to build comprehensive automated test suites. Discovery testing is limited to a handful of manual tests, while mutation testing is automated using tools like Stryker.

Want To Calculate WoW ROI with Your Team or Group?

We’d be happy to spend a half-hour with you on a call to explore your context and calculate potential savings from better Ways of Working.

Gilded Rose with AI and PyCharm

Thu, 30 Jan 2025 15:00:00 +0000

This story comes from a software technical craft person of some standing who has never been an AI pundit or promoter.

It seems to be helpful when incorporated with other good habits and managed respectfully.

I used the Gilded Rose kata, which may have colored my results. It is possible that the AI assistant has been trained on various solutions to Gilded Rose, and merely memorized the results. I can’t tell if this is true or not, so I can only report what I know.

Start Well

For my example, I cloned the Python version of the Gilded Rose kata, created a virtual environment and git repository ( via UV), and installed the requirements.

I fired up my editor, in this case PyCharm.

Following the usual Clean Start protocol, I began by running all the tests in the project.

I clicked the “rerun” option so that the full test suite will run automatically when any file changes. I find that this makes continuous testing a trivial undertaking, and I’m a big believer in both making my work effortless and also continuously testing as I code.

Famously, it comes with only one test, which is made to fail. The test ran and failed as expected. Good.

I used the gnext script, described in Save Your Game to set the intention of my next action.

I have since put an improved version on github so you can feel free to clone it and use these commands on your daily work (without warranty and at your own risk).

I made the test pass, and used gdone to commit my changes.

Add Tests

The whole point of the Gilded Rose is that it is a bit of ugly legacy code without tests.

The scenario of “I have to work on this code, but there are no tests” is quite familiar to most of us.

gnext "test: add unit tests"

In the AI window, I set the scope to be the gilded_rose.py source file, and asked the AI to write unit tests for the update_quality() method.

I have used test generators before, and have been uniformly disappointed. They tend to be run-on, ill-structured tests with meaningless names, but they do provide coverage for code I want to refactor. This is my first time doing this with an AI.

In a radical departure from expectations, it did a pretty good job. Not great, but pretty good.

The tests were of the form:

    def test_normal_item_quality_does_not_go_below_zero(self):
       items = [Item("Elixir of the Mongoose", 5, 0)]
       gilded_rose = GildedRose(items)
       gilded_rose.update_quality()
       self.assertEqual(items[0].sell_in, 4)
       self.assertEqual(items[0].quality, 0)

The test naming isn’t too bad. It’s roughly as good as i sometimes see from students in my TDD training.

The tests follow the Arrange, Act, Assert pattern. They have exactly one action per test, as I prefer. It also is willing to have multiple assertions when there is more than one evidence of the action, which is also to my taste ( YMMV).

On the other hand, I don’t like how it subscripts the list to reference the item.

Some people will argue that the micro-efficiency is poor since it subscripts the array multiple times, and array indexing isn’t free. I will recognize that, but it is not my objection.

I worry that the array indexing distracts the reader and complicates the test just a little.

Since the tests work, I’ve satisfied one level of the Code Virtues and now duplication and simplicity become more prominent concerns.

I asked the AI to try again but give the item in each test a name instead of referencing it through the items array. I decided to push my luck and not give a very concise and clear set of instructions.

My prompt is imprecise and curt: “rewrite these tests but reference the item directly instead of using items[0]”

It now created a named object in each test and passed the object in an “immediate” array to the GildedRose constructor.

    def test_aged_brie_increases_quality(self):
        aged_brie = Item("Aged Brie", 2, 0)
        gilded_rose = GildedRose([aged_brie])
        gilded_rose.update_quality()
        self.assertEqual(aged_brie.sell_in, 1)
        self.assertEqual(aged_brie.quality, 1)

I used the handy “copy” button from the AI’s code window and pasted it all, replacing the content of the test_gilded_rose.py file entirely in one easily revertable step.

The tests ran, and they all passed. So far so good.

gdone

I suddenly realized that i hadn’t run the tests with coverage, which I think is reasonable if one is going to rely upon tests for refactoring.

So, okay, we run with coverage. I have line coverage of the entire source. That’s nice.

But what about mutation testing? In python, I installed mutmut and configured it quickly. It also found no problems with the tests. There were no “surviving mutations” from the code.

The “Production” Source

At this point, I decided not to follow the standard Gilded Rose script, for fear that the AI assistant was parroting all the prior answers it had memorized.

I decided to attack negation in the source.

gnext "refactor: avoid negation in conditions"

My prompt: “refactor update_quality to avoid using negation in if statements.”

This is a thorny thing to ask, given the nature of Gilded Rose.

The AI provided some refactored code, and I copied it all and pasted it over the top of the entire gilded_rose.py contents.

Immediately the tests failed. The refactoring was not valid.

That’s okay, I use gnext/gdone to save my state, and I’m making changes one at a time so that the undo is easy.

I used undo, and got my original code back: all tests are green.

My prompt is even less specific: “try again, but be sure all the tests pass”

I paste, and the tests pass. Surprising! I guess I should have stated that the tests need to pass to begin with. Lesson learned.

The code is structured very differently now. Methods have been extracted, and a guard clause added for “Sulfuras”:

    def update_quality(self):`
        for item in self.items:
        if item.name == "Sulfuras, Hand of Ragnaros":
        # "Sulfuras" never changes in quality or sell_in
        continue

       # Update `sell_in` for all items except "Sulfuras"
       item.sell_in -= 1

       if item.name == "Aged Brie":
           self._update_aged_brie(item)
       elif item.name == "Backstage passes to a TAFKAL80ETC concert":
           self._update_backstage_passes(item)
       else:
           self._update_normal_item(item)

This is more improvement than I’d asked or hoped for. I’m torn whether I like that big change or not, but overall I think I do.

All tests pass. At this point, I’ve achieved the intention of this commit, so one more time with gdone.

Cross-Class Refactor

For fun, I decided to even further break with tradition and modify the Item class.

gnext "refactor: items know their type"

This might not be a thing that I would do normally, but I’m kicking tires. My prompt: “refactor update_quality so that all the name comparisons are part of the item class.”

I’m intentionally vague, but it seems to understand what I’m asking just the same.

An excerpt:

    # Helper properties for special item checks
    
    @property
    def is_aged_brie(self):
    return self.name == "Aged Brie"
    
    @property
    def is_backstage_pass(self):
    return self.name == "Backstage passes to a TAFKAL80ETC concert"

Not only did it add name checks to the Item class, it implemented them as properties rather than normal member functions. I didn’t ask for that, but it’s not a strange or troubling choice.

The update_quality isn’t odd-looking now, either, so I paste the new code and watch as the tests pass.

    class GildedRose:
        def __init__(self, items):
            self.items = items
    
    def update_quality(self):
        for item in self.items:
            if item.is_sulfuras:
            # Sulfuras has no quality or sell_in changes
            continue

           # Decrease sell_in for all non-Sulfuras items
           item.sell_in -= 1

           if item.is_aged_brie:
               self._update_aged_brie(item)
           elif item.is_backstage_pass:
               self._update_backstage_pass(item)
           elif item.is_normal_item:
               self._update_normal_item(item)

I can do without the superfluous comments, but that’s a separate intention. With the code valid (passing tests) and the current intention met, I use gdone again.

It modified both classes, the caller and the callee, in cooperation. This is some good semantic editing, even if I know later I will be renaming the functions to be less string-specific and more type-specific. It’s a good step on the path.

Reducing Conditional Complexity

The methods for updating various items are riddled with silliness like this:

    if item.quality < 50:
        item.quality += 1

So:

gnext "refactor: eliminate needless comparisons to 50"

It seems to me that taking that much space to deal with “increment up to 50 only” is pointless. Can the assistant work out what I mean from a simple prompt? Let’s try “in _update_backstage_pass, refactor the code to use max instead of checking the value is less than 50 and then incrementing.”

Will it understand?

    def _update_backstage_pass(self, item):
        if item.sell_in < 0:
        item.quality = 0 # After the concert, quality drops to 0
    elif item.sell_in < 5:
        # Increase by 3 if sell_in is 5 or less using `min` to keep it at most 50
        item.quality = min(50, item.quality + 3)
    elif item.sell_in < 10:
        # Increase by 2 if sell_in is 10 or less using `min` to keep it at most 50
        item.quality = min(50, item.quality + 2)
    else:
        # Increase by 1 otherwise using `min`
        item.quality = min(50, item.quality + 1)

Not only did it understand, it recognized that I should specify min, rather than max as I suggested. This is a signature error - I have done it several times, and then have to go back to it. This time, my AI “programming pair” didn’t let me do something that backward and silly.

I paste the code, the tests still all run green. All code has line coverage.

gdone

Ideating

My new partner and I pursued a logical improvement. We hit a few minor snags along the way, but I thought it was illuminating.

Here is a bit of ugly conditional code:

    def _update_backstage_pass(self, item):`
        if item.sell_in < 0:
            item.quality = 0 # After the concert, quality drops to 0
        else:
            change_amount = 1 # Default increase
        
        if item.sell_in < 5:
            change_amount = 3 # Increase by 3 when 5 days or fewer are left
        elif item.sell_in < 10:
            change_amount = 2 # Increase by 2 when 6-10 days are left
    
       # Update quality and ensure it doesn't exceed 50
       item.quality = min(50, item.quality + change_amount)

In the way of improvement, it suggested this:

    change_amount = (
        3 if item.sell_in < 5 else
        2 if item.sell_in < 10 else
        1
    )

It is relatively terse, relatively clear, has pretty formatting, and works.

On the other hand, I’m not sure that cascading ternaries will be anyone’s favorite choice for expressing this idea.

I’m willing to be convinced otherwise.

It is, however, an implementation I’d not considered before since I have a bit of a prejudice against nested ternary expressions. The AI tool doesn’t have my prejudices, and may suggest things I wouldn’t have tried. That’s not a bad thing

I asked about ways to represent a lookup table instead. It suggested this form:

    lookup_table = {
        range(0, 51): "a",
        range(51, 101): "b",
        range(101, float("inf")): "c"
    }

    for key_range, result in lookup_table.items():
       if value in key_range:
           return result
    return "unknown"  # In case value doesn't fit any range

This isn’t bad. It is quirky though.

Using a range object as a dictionary key is unusual. In earlier versions of Python, it wasn’t even legal.

The weirdness of this even confused my AI Assistant. In some versions of python, the range object isn’t hashable and can’t be a key. In later versions, it is valid. The AI seemed to be confused about what version we are using, and went back and forth between recommending this structure and declaring it illegal.

Since we are not using the dictionary as a dictionary anyway, why use a dictionary? It seems to be a lot of mechanism for a simple thing. I would think a list of tuples would be more natural, and we could even have made this a tuple of tuples to keep it immutable, for what little difference that would make.

I switched to use the more obvious array of tuples, and didn’t have any issues with either the tests, the compiler, or the assistant.

In the next move, it deleted my version and replaced it with the nested if statements again. I can see that I will need to watch this AI thing very closely if I want to maintain my own updates.

Impression

Using an AI can help with proper code craft including coding, testing, and refactoring.

The fact that it is trained to know programming languages and their libraries means that it may suggest solutions I’d not considered, though I did find it lacking on knowledge of some python libraries like itertools and operators.

I’m surprised how my intentionally poor, loose prompts still got good behaviors most of the time.

I was also surprised at how often I needed to re-center it to use the existing version of the code and not something it had imagined or suggested earlier in the session.

I was pleased with how it generated tests for existing code.

When I’m dealing with legacy code, I will probably extract methods and have an AI provide tests.

In code other than this kata, I’ve had it generate invalid tests, even some that didn’t compile. When it works, it works.

These tools will be helpful to people who work with discipline and proper craft. It may also help people who don’t, but risks are greater that they will generate troublesome code that they don’t know how to fix.

With microcommits and tests, I think I’ve limited the damage it can do.

I will remind you at this juncture that my AI assistant may have been trained on Gilded Rose and may have been merely regurgitating memorized code samples

It seems to be doing my bidding with semantic editing, but it could be a convenient parlor trick in this case.

I will have to try the same techniques and behaviors in code bases the AI has never seen before.

Don’t turn off your own intelligence, intuition, creativity, and code sense, but consider using AI as a partner. Don’t trust it too much, but don’t ignore it.

AI gets better fast; it currently is the worst it will ever be. Where it falls short today it may be soon be more competent.

See also:

Consider using Industrial Logic’s competence to upskill your teams.

Save Your Game

Thu, 09 Jan 2025 15:00:00 +0000

Series index

Welcome to the second stage in the process.

The Basic Steps

There is a rather lightweight discipline here. It’s unfamiliar to many people, but it isn’t difficult to learn. You may find these steps described more succintly in Intentional Commits’ treatment of scope control

First, choose what you will do. This means the next VERY SMALL step, possibly as small as writing a function or even adding a log message, that moves you a little closer to the goal you have in mind. We’ll discuss this more, below.

As you work, run all of your checks. That includes the reformatter (either habitually while editing, or automatically on save), the lint checkers, the static security scans, and all the tests fast enough to support your minute-by-minute development work.

You can commit any time all the checks are fully successful (they “run green”).

You can pull from the main branch after any commit.

You can push as long the tests are green and the application works.

If tests fail, you can either add a fix in the next step or else make a graceful retreat.

Choose What You Will Do

The mindset of taking smaller steps is described as “iterative and incremental development,” “walking skeletons,” “ tracer bullets,” or “evolutionary design.”

Here, we assume that you have read up on these concepts and are ready to implement your big feature as a series of small end-to-end changes. It’s too much to try to introduce incremental development as a sidebar to this Easy Recovery topic.

What step can you take now (ideally within the next hour) that would help solve some part of the problem for some set of people?

As a side note: sometimes the thing you most need to do is explore the application and understand the flows you are fitting your change into.

You may wish to modify code experimentally to learn how the system will react. Since you began with a Clean Start, there’s nothing to fear; you can always return to that starting position.

Once you have made your choice, that is the ONE thing you are going to do.

Commit to Doing Just One Thing (at a time)

I’ve become a fan of Intentional Commits.

It’s a simple idea: choose what you will do next, control your scope, and do the thing.

If you think of some good side quest, don’t pursue it. Instead, write it down and consider doing it next, after you finish the current task.

Intentional commits are a lot like Test-Driven Development.

In case you need a refresher on TDD:

Choose what behavior you want to create next
Write a test (to a hypothetical API) that will expose that behavior
Write the code that will pass the test
Ensure that the new test and all prior tests are passing
Refactor the code toward the design you would like it to ultimately have.
Integrate: collect the changes your teammates have pushed and share your changes with the team. When this seems like overkill, at least “save your game” by committing locally.

With intentional commits, you:

Choose what you will want to do next, perhaps composing the commit message in advance.
Make the change you have planned, and only that change.
Anything you realize you also want to do? Write it down. Don’t do it yet.
When the change is made and all is “green”, commit with the message you composed earlier.

Both cases rely on your ability to slice work small and do just one thing at a time.

The advantages are convenience, safety, and time:

All the time previously spent composing meaningful commits by selecting passages of code from various files? Gone. Since you only did one thing, there are no stray changes to separate. Most of the time you can just add everything (git add .) and commit.
Your commit will be appropriate for a code review (if your company is still doing asynchronous code reviews).
Once you push, you are completely free to do something else. That might be exploration, testing, refactoring, UI improvements, expanding the functionality you worked on in the last commit, or picking up a side quest you’ve recorded for yourself.
Your work is more visible more often. This means that managers, sponsors, bosses, supervisors and that lot will not have the same pressure to micromanage you.

There is safety, convenience, and freedom in working this way.

The problem, of course, is that 95% of preference is familiarity, and most programmers are only familiar with “load it all in your head and do all the programming, then look for the problems” from prior practice. Since that’s what they know, and it’s gotten them this far, that’s what they prefer to do. This is why people tend to order familiar items off of a restaurant menu, and go places they’ve already been.

This isn’t a moral imperative, just an advantageous way of working. There is no moral “should” involved. If you don’t already do this, that doesn’t make you a bad person. If you like to stick with the old way of working in big steps with fear of loss, that doesn’t make you a bad person either.

If you choose to work in small steps, you will get certain advantages. It is an adjustment. You may fall back on habit sometimes. Maybe you can experiment with this on the side, or try it only on small jobs, or when pairing or working in an ensemble.

If you get used to it, you may not want to return to the old way.

Graceful Retreat

Once I learned to do TDD (probably around 2004?) I got used to the idea that all the tests passed 5-10 minutes ago. Since I was working in small steps, I could safely and easily delete my most recent change.

In the Industrial Logic eLearning, Joshua Kerievsky described it as a “graceful retreat”: you can revert your local changes using your version control tool.

In git, that’s git reset –hard

I know people who would rather amputate a finger than type that sequence of characters. They imagine, perhaps rightly, that they have too much invested in their current unit of work. The solution sounds overly-simplistic, but it is actually a simple idea: Don’t invest that much between commits.

For safety’s sake, we strive to never have a large commitment at risk.

Instead, we do one thing at a time and commit when the tests run green.

We call these microcommits.

Gamers talk about “saving their game”. If you save your game (at least before a boss fight) you can gracefully return to your healthy pre-battle state.

It’s the same with software. Thanks to Distributed Version Control Systems (like git), you can save your game at will.

You certainly should do so before attempting anything tricky or error-prone.

If you commit locally while the code is in a good state, you can return to that good state with all tests running successfully and your application still working.

I was talking with Bryan Beecham, Mike Hill, Ron Jeffries, and Chet Hendrickson and I don’t remember who said what or in what order, but it was mentioned that we don’t use the reset command often enough. Everyone agreed that it is better to reset to the last good state and start over rather than fight through a badly started change.

This is only a sensible thing to do if you are saving your game regularly.

Calling Your Shot

I’ve gone so far as to create a set of zsh functions to help with my intentional, tiny steps.

You can find (and try) these via GitHub. The commands are gnext to declare your next change, and then gdone to commit it or gnopeto delete the message and revert all changes since the last commit.

I don’t know if you find this useful or not, but for me it is handy to pre-write my git message, just as it is to write the test before the code in TDD.

Even if you don’t use TDD, you could use gnext and gdoneto see if it helps you to think in intentional, small steps. This is provided as an example, no guarantees of any kind included. It works, and it’s the script I use, but take responsibility for your use of it.

But WHY?

Fear of loss drives poor practice.

When you aren’t afraid to revert, you aren’t afraid to move forward.

Test-Driven Development and microcommits are not common (enough) practice, so the whole “easy recovery” thing is a bit of an open secret.

Developers often do a large amount of work at once, and their changes are spread out among many lines of code in many source code files.

It’s not unusual in some projects to have many hundreds of lines of code in several dozen files, all as part of making a single change. Sometimes it’s thousands of lines in hundreds of files.

These are undifferentiated changes in the file system. They’re not segregated and indexed in a way that describes why the change was made or when. If the developer doesn’t practice the Clean Start protocol, these changes may be mixed with unrelated changes made previously.

Eventually, developers will run the code, and maybe even the tests¹.

They will find the errors they made hours (or days) ago and will fix the ones they find. They will likely test for side effects and fix those as well if time and system knowledge are sufficient.

Then they will commit the current mega-change-set so that they can push it to a shared (or sharable) code branch. This may be the only time they save their changes other than locally in source files.

But let’s say it doesn’t go quite that well.

If they find that they have made a change with puzzling and unwanted consequences, they may have to undo their changes and redo parts of the work. There is no easy and clear way to do this, so they may spend hours combing over files, undoing changes, looking for potential side effects or mistyped algorithms, seeking the error, separating the good changes from the bad ones.

In the worst-case scenario, it may be so baffling and complex that they are forced to revert the whole change set and lose all the work they’ve done so far.

The fear of losing that much work is so significant, they may press on desperately, making change after change.

We’ve seen people spend days debugging and experimentally “hacking” at the code to avoid losing a prior afternoon’s work, always certain that they are only one or two more little problems from completion.

The problem here is that the change is “all or nothing.”

This kind of practice involves individuals making large changes without any safety mechanisms in place. They don’t think much about that, though, because they’re used to having a lot of work at risk and usually it turns out okay. It’s “ normal.”

It shouldn’t be normal. It turns out badly often enough that it causes fear. It encourages people to leave code in a poor state. It makes the goal of actually implementing a feature such a big accomplishment that people seldom will stay in the moment and tidy up after themselves. The fact that they managed to get it to work at all is success, and eliminating the collateral damage feels like a pointless chore to them.

Mind you, this isn’t the single worst problem in software development, but it is enough of a drag on developers that people avoid improving the design of the software or trying new ways to achieve new results. They work in a pathologically conservative way and the code base quality degrades.

We find it safer to work in small (incremental) steps so we’re not at the ragged edge of human cognition at all times. We also make it safer by working together, so that we can cover each others’ blind spots.

Tiny Steps Make It All Work

You may find it’s faster and easier to start over than to “gut it out” when things are confusing.

With a lower cost of failure, you may find yourself being more innovative!

I am confident that you will feel less stress and make more progress when you always know exactly what you are doing and that you can cheaply and safely step backward.

It works even better in pairs or ensembles.

Notes

Not a joke. We found that a lot of developers never run the test suite at all, but send their changes blindly to build-and-test server, hoping that the tests will pass there.

The Clean Start Protocol

Fri, 13 Dec 2024 15:00:00 +0000

Series index

A clean start is the beginning of a great programming session.

There have been times that I was working on some new code and found an unexpected problem either with the automated tests or by manually running the code and seeing a flaw that wasn’t covered by a test.

I have spent hours trying to understand how my code caused such a failure, only to learn the fault wasn’t mine. It was usually that someone else had pushed a bad commit to the shared branch, and I pulled without noticing it.

Though it pains me to admit it, I’ve also been the one who pushed code that couldn’t work in CI or on my coworkers’ machines. It was usually that I had something on my machine that wasn’t on everyone else’s machine; a library, module, data file, or compiled library that was not a part of the configuration.

I’ve had the same experience when pairing or mobbing when my coworker did not start clean.

Controlling the dependencies and quality of the code is crucial for CI and CD and is a good all-around discipline.

It is surprising how many people have no “clean start” discipline despite having been through degree programs and years of work experience. I certainly didn’t for decades of my early career (no shame in it, just acknowledgment).

The Modern Agile wheel includes an element in the bottom, shown in blue, that states “Make Safety the Prerequisite.”

This means that we should begin in such a way that we will not be unnecessarily at risk of loss or unnecessary struggle.

One part of the safety prerequisite is to have tests, particularly fast-running automated checks.

We need a way of quickly validating that our system is working. If we have to manually test all the features of complex systems, we might have no time left over for programming!

If you don’t have fast automated tests, it’s a good time to start building some, but keep reading so you understand what advantages they will give you at the start of each new unit of work.

We will take automated checks as a given.

What is a Clean Start?

I begin new work ensuring the workspace is free from pollution, leftovers, failing tests, and half-finished work.

We will put our workspace into a “clean” state before beginning each job, even if the job is working in the same code base and the same branch (we only use ‘main’).

We don’t switch jobs without ensuring a clean start.

Step 1: Git Status

Make sure that you have no untracked files in your workspace. None.

If you have any, use the appropriate step:

If they are created programmatically, have them deleted programmaticaly.
Add their directory to .gitignore if (and only if) the file is harmless and generated by tools during linting, editing, or checking.
Remove them by hand (scratch dev files, temp data files, etc).

When you are done, you should be greeted with this happy message:

You want to make sure you have no uncommitted changes to any of the files in your codebase. You can commit them or revert them, but you cannot have any uncommitted changes.

You need to be free of the work you’ve been doing before you start a new job.

Achieve this first.

Step 2: `git pull –r`

Pull the current working version to your machine.

Since you’re entirely clean locally, there is no reason to rebase, but I find -r to be a good habit, and I don’t want to have to decide whether to do it or not.

You want the most current and most integrated version available to you. You shouldn’t have to decide when to do this; it should be automatic.

You might be surprised how many people forget this, and base their new work on some old version of the main branch. Don’t make that mistake!

Why the -r? Well, I use pull -r whenever I pull and find it a useful practice, so I want the habit to be to use the rebase flag all the time. I trust habits more than decisions in this case. It’s harmless enough in this case, and I can always just recall the last pull command rather than retype it.

So now you have nothing unique in your space, and you are current with the main branch as far as the file content is concerned. You’re still not ready to start work.

Step 3: Update the Local Environment

This step depends on your application stack.

If you are in Python, then you will use your local build/dep tool to update. This is often pip, poetry, or pants.

If you’re in Javascript or typescript it might be npm.

In Java, it is likely Gradle, Maven, Ivy, or Ant.

Whatever the tooling, make the environment current. It is too easy to overlook this step and end up with tests failing or even failing to run.

Now you have a fresh environment, and there are only a few steps left.

Step 4: (Clean) Build

Compiled languages often require a build. A clean build deletes any artifacts left over from prior builds, and that’s the preferable situation.

Since you’ve done nothing yet, are current, and have a clean environment, there is no good reason to have trouble building the current application.

If the build fails, there is a problem with your local environment to solve before you touch a line of code.

Clear up any problems that emerge at this point, while you know it’s not your fault.

Step 5: Run The Tests

Run your test suite(s). There should be no failing tests at all.

Are the tests running in the CI pipeline? You should double-check that. Maybe you need to correct your machine’s global environment!

If tests are failing on your machine and the CI environment, you can be sure you’ve inherited that problem from the main branch. It’s not your fault, and you may not have to be the one to solve it.

If you investigate the failure and solve it yourself (after informing the person who pushed the change), you do so knowing that you don’t have ANY local changes that might have contributed to the failure.

I like this “clean room” approach.

DEVELOP!

Now if anything fails to build or test successfully, it’s because of the code you have just written, and you can solve that at your workstation. You won’t spend hours or days chasing other people’s problems.

As you develop, you will probably want to run the tests frequently (maybe automatically, in the background?) so you find out about problems while the “search space” for the failure is quite small.

You will likely want to do microcommits as you work, or at least right before starting anything tricky or interesting. We will discuss this more in forthcoming articles.

Don’t leave it up to your microtests, though.

Consider having all the human augmentation available and affordable in your editor. I like using SonarLint (because it’s free), and any other checkers I can find. I’m trying some AI tools, though the results so far are mixed.

In Python, I use type hints and protocols to help my IDE and various code-checkers spot problems as I’m editing the code. It saves me time to keep warnings out of my code.

I use automatic reformatting rather than making a mess for now, only to tidy it up with an automated reformatter later..

BTW: any time you have committed, you can also do the above 5 clean start steps again to maintain currency with the development version and avoid a big merge conflict later.

Committing

Since you have no changes except those related to the one thing you are doing, you don’t have to spend time composing a commit out of selected passages from selected files.

You will still have to check (git status again) to be sure you’ve added all the files to version control, or at least do a global add (git add . at the root) to get all the files included. Just be sure you don’t commit temporary files that need to be deleted or moved instead.

You may be interested in the discipline of Intentional Commits. I strongly recommend an intentional approach to work, so you are doing one thing at a time.

When you are focused on doing one thing at a time, you won’t get confused or befuddled, nor will you lose your thread if you are interrupted (as often happens when one is in a side-quest of a side-quest of a side-quest).

You might find it helpful to add precommit hooks to do a new build and run tests again.

I’ve gone so far as to write zsh functions so that I can compose my commit message before doing the work and then commit (with ‘-a’) afterward, using the message I’ve composed.

If that sounds like a lot of discipline to you, I will comfort you with the knowledge that I never have to carefully compose all my commits. It’s always all-or-nothing, always easy, and I am not burdened with excess decision-making. I have found it easier and less stressful this way.

Make Clean Start Easy

There are 5 steps in the “clean start” protocol described above. That sounds like a lot, and even I don’t always remember to work through the whole list.

I find it’s easier to automate such a simple process than to memorize it.

I like to create a script (I usually call it prepare, though maybe I should call it cleanstart?) that will do a git status, a pull, update the environment, build, and run tests.

It’s trivial for me to type ./prepare before beginning any new work, and trivial to remember to so do.

It is easy to write a bash/zsh script that exits if any of the steps fail. I can’t imagine any reason not to do so. I usually have a ./prepare script and a ./run script in the root directory of a project just to keep it simple and easy.

Here is one actual example from a python program I work on regularly:

    #! /bin/bash -e
    source venv/bin/activate
    git pull -r
    pip install --upgrade pip
    pip install -q -r requirements.txt
    pytest
    echo "Completed successfully."

The -e tells bash to exit on the first failed command, so the rest of the script isn’t performed after a failure.

If I run it from the wrong directory, the source command fails, and if I’m in the correct directory then it’s harmless.

If I have uncommitted changes, it fails with this message:

    error: cannot pull with rebase: You have unstaged changes.
    error: Please commit or stash them.

I upgrade pip before using it to install new requirements. A lot of my newer projects use poetry, and I’m looking at perhaps moving forward from there soon, but this project has been using pip all along and I’ve not felt the urge to modernize it yet.

Pytest runs all the tests, and stops the script execution if any active (un-skipped) tests does not pass.

The script takes on the order of 5 seconds when pip is up to date, there is nothing new on the main codeline, and there are no new requirements. That’s a very small amount of overhead to ensure that I’m truly ready to start my work.

If you begin with the Clean Start Protocol, I’m sure you will find your work becomes more tidy and effective. It also provides a base situation for the next 4 steps of this process.

What did I leave out? Maybe you can add steps I’ve forgotten to mention?

Industrial Logic - Better Software Sooner

Replace Hard-Coded Notifications with Observer

The Smell

The Refactoring

Why This Is Better

The Pattern in Action

When to Apply This Refactoring

Further Reading

Safe Releases

Series index

Beginning Well

Why start with deployment?

Why not wait a fortnight to deploy?

It’s more work if we wait

It withholds value

Waiting Defers Feedback

But why start with a test runner?

What do we test?

How do we test?

Don’t Ship Half-Done Work!

Is This Asking Too Much?

Do We Have To Work This Way?

Easy Integration

Series index

What Makes Integration Hard?

How To Reduce Difficulty

Limit The Number of Parallel Activities

Reduce Period of Isolated Code Development

Find Problems Sooner

Is This Asking Too Much?

Do We Have To Work This Way?

Shouldn't It Be Faster?

Too Slow

Amdahl’s Law

Maybe it’s not parallel?

The Non-parallel Part

Managing Parallel Work Is Harder

How Can We Win This?

Is this really a problem?

That explains it!

No, this isn’t part of our problem…

Footnotes

Agentic Extract Method

Fast Feedback

Series index

In-Session Feedback

External Feedback

Does it matter?

Notes

The ROI of Saving 700+ Hours Per Month

How Teams Waste 700+ Hours/Month

The Shocking Truth About Your Team’s Time

How to Get Those 700 Hours Back

Subdue the Meeting Monster

Automate Everything Possible

Optimize Workflows

Increase Collaboration

Reduce Context Switching

Where Did 700 Hours Come From

The Impact of an Optimized Approach

The Hidden Costs of Traditional Development

Why This Matters

The Bottom Line

Waste Cost Value Calculator (Traditional vs. IL WoW)

Cost Efficiency Gains with IL WoW: A Summary

From Waste to Efficiency: Cutting Overhead with IL WoW

The Hidden Cost of Rework: Quality Control Inefficiencies

Traditional Ways of Working:

IL Ways of Working:

Before & After: Reducing Non-Value Tasks with IL WoW

Want To Calculate WoW ROI with Your Team or Group?

Gilded Rose with AI and PyCharm

Start Well

Add Tests

The “Production” Source

Cross-Class Refactor

Reducing Conditional Complexity

Ideating

Impression

Save Your Game

Step 2: `git pull –r`