To respond, please visit
http://www.cutter.com/content-and-analysis/journals-and-reports/cutter-it-journal/callforpapers01.html

But, too often, product and process are not given the architectural priority they deserve. While physical products such as cars or planes are highly engineered, enterprise architects tend to overlook the architecture of information-based products and view them instead as the domain of business managers. (Note that physical products, such as the engineering of cars or computers, are more likely to be architected than information or digital products, such as financial or insurance components.) Process, on the other hand, is more likely than product to be mentioned in the business architecture yet is often regarded as too intricate to warrant an architectural approach, leaving its detailed examination to business analysts. As a result, architects often bypass product and process in favor of strategy and capability.

In addition, the architectural interrelationships and dependencies between product and process are not fully recognized. So not only do product and process receive less attention than they warrant, but we fail to architect products and processes together. Consequently, organizations frequently reengineer processes only to find that introducing a new product or changing an existing product requires further significant process alteration. Or, we see new product features being introduced with widespread disruption to services and processes.

Whenever an organization requires change, certain people must be involved. These people are on the “critical path” for change. Yet, as more and more people collaborate on that critical path, change becomes more complicated. When those people are in high demand and are difficult to pin down, it takes longer and adds administrative overhead. If those people are in short supply or are highly skilled, it results in resource issues and spiraling costs. So it makes sense to reduce the number of people on the critical path, reduce their degree of involvement, and minimize the need for specialized skills, knowledge, or experience.

Products and processes are probably the most demanding components in enterprise architecture when it comes to involvement of large numbers of decision makers, designers, analysts, and architects who are highly experienced, knowledgeable, and skilled. EA can, and should, reduce pressure on the critical path for change. By architecting product and process at the same time, we can reduce the number of people blocking the critical path — and significantly improve agility. Think of an important architectural challenge you face today: how many people are on the critical path?

For more on the main architectural constructs that support product and process, see my Cutter Consortium Executive Report “Architectural Constructs for Agile Products and Processes“.

Agile, as both a theory of management and of software engineering, is tuned very well for one to several teams of individuals working closely together. Beyond that scale, agile alone won’t address all organizational challenges — no agile transformation initiative alone could do so. Solving larger-scale problems will require other management and engineering methods, some of which I discuss below.

When companies grow beyond a handful of individuals or teams, many changes will affect the organization of work and people. One of the first results of large scale is decreased information sharing, both across teams and for individuals joining existing teams. At small scale, and for a limited period of time, a team can organically “remember” its own history. Beyond that small scale, though, organizations need to promote mechanisms that encourage learning and sharing across the organization.

Mentoring

Today’s organizations are heavily focused on knowledge work — the creation and application of knowledge. It is truly the people that are the most valuable resources. Learning can’t be rushed or accomplished in a quick orientation. The 10,000-Hour Rule says that it takes approximately 10,000 hours of deliberate practice to master a skill. That is about five years of working full time at something in an environment where feedback and progress can occur.

Mentoring is a mechanism for long-term leadership and growth. It encourages learning and transfer of knowledge within and between parts of an organization. Mentoring is used extensively in Japanese management, and it is the foundation of craft and artisan learning. In such organizations, everyone has a mentor who assists and actively guides them.

Value Stream

Beyond the challenges of sharing information effectively, organizations at scale frequently structure into functional silos. This is often the result of focusing on efficiencies of scale in the attempt to keep individuals fully utilized and working on the most appropriate tasks. This functional efficiency comes at the cost of cycle time and responsiveness, though, and it doesn’t account for the cost-of-delay economic tradeoff.

Value stream-based organizations will optimize for the flow, and reduced cycle time, of valuable product features. Instead of trying to maximize efficiency of any one function, this approach measures the whole system while trying to reduce total time to delivery. In this method of managing work, deep queues of work don’t exist to cause significant delay.

Aligning toward the customer and stream of value promotes another benefit for individuals and teams: it is easier to define a vision and more motivating for people to work toward that vision when they can connect themselves with delivering actual value to real customers.

Risk Management

Planning work at scale involves committing resources with energy and momentum behind them, often with a fairly detailed budget for a fiscal year or more. This commitment creates an inertia that is a particular challenge for agile teams, which by definition assume the future is going to change, yet are pressured by a static plan.

Risk management can be applied during yearly planning cycles to projects and technical investments alike. Assuming “economics is an exchange of risk and opportunity,” we can conclude that budgeting and financial planning activities should account for the temporal nature of both risk and opportunity.

At smaller scales, risk management can be as simple as creating what-if scenarios, but as investment and commitment scale, so too should risk management. Such topics as technical debt, new or old technologies, competitive landscape, and so on, should all be represented in the planning cycles to mitigate the risks posed by the unknown. It may be very appropriate to take a risk, but it is never appropriate to forget about the facts — your economic outlook depends upon it.

Real Options Theory

In addition to avoiding or mitigating risks, deferring the planning or execution of work until the appropriate time is a planning option that can be used effectively at any scale. Real options theory holds that options have value, and options expire. Never commit early unless you know why.

Options thinking encourages us to delay commitment instead of trying to lock things down as soon as possible. This delay improves our position by giving us time to acquire more information and, when it’s time, make a better decision. This style of planning blends well with agile teams: both the decision making process and execution evolve together over time.

Evidence-Based Management

Finally, in a large-scale environment where it’s all too easy to be less connected with customers, colleagues, and other teams, it is even more important to pursue empirical data of what is truly needed and actually happening.

Evidence-based management is the “conscientious, explicit, and judicious use of current best evidence in making decisions.” There is a long pedigree of methods based on this idea:

• W. Edwards Deming taught using data to make decisions in TQM.
• The Toyota/Lean process has both “go to the source” (the Japanese word gemba means “source,” or where the work is actually being done) and data-driven continuous improvement toward target conditions.
• OODA/maneuver warfare teaches both gathering information and making decisions with imperfect information. (John Boyd defined the OODA [Observe, Orient, Decide, Act] loop for fighter pilots while a member of the US Air Force. It is now a theoretical basis used in maneuver warfare.)
• The Lean Startup method is strongly focused on growing knowledge through experiments.

This is the scientific method applied to business and management: have an idea, experiment to prove it out, then either alter the idea or invest in it. Agile teams are very good at doing exactly this sort of experimentation and changing direction based on the results. Overly lengthy and detailed planning horizons or “hope-based” management (which skips right from idea to investment, skipping over any real validation) often leads to death march projects.

We would argue that most C-suite executives are selected on their management skills, not their leadership skills, which is why there is a dearth of leadership across both corporations and government. This has occurred in large part due to the fundamental reengineering of organizational structures, operations, and finance that began in the early 1990s and continues today. Organizations are now increasingly organized around short-term projects. Instead of being hired to work for possibly a lifetime in a company, people are hired to work on a project that has a definite lifespan. Once the project is over, individuals are likely to be let go unless they can find work on a new project. According to US government statistics, up until the recent economic troubles, the years a person worked for the same employer has, for more than a decade, been steadily dropping for all age groups except those over 65.

The growth of the project-oriented organization has meant that the acquisition of management skills is seen as the key to career advancement. The other career path is to become a specialist in marketing or sales, or in the financial/human resource management aspects of a company. In earlier organizational structures, where managers could learn leadership skills as they advanced through their careers, there was more opportunity to acquire the experience needed to develop robust spoke behaviors and generalized rim skills. In today’s working environment, specialization is rewarded, whereas being a generalist is undervalued.

The opportunities to learn to become a leader have dwindled precipitously, which means that the deliberate nurturing and selection of individuals for leadership positions is more important than ever. A recent Chief Executive article stated that the best companies for leaders, which generate dramatically greater market value over time than the companies weakest in leadership development, make leadership development a very high-priority commitment despite the current economic situation. Leadership development pays.

Developing the IT Leaders of Tomorrow

A recent article in the Wall Street Journal asked, “Do Techies Make Good Leaders?” The answer was yes, but there are a number of unique challenges the IT industry faces in comparison to others. For instance, the IT industry is full of young people with strong backgrounds in science and technology. They often like working on technical problems more than they like working with people, yet leadership is, by definition, working with people. Surely this makes the development of both leaders and managers more difficult. A 2005 Cutter survey on IT leadership supported this view: the lack of empathy (a spoke behavior), lack of emotional ability (a hub trait), and lack of interpersonal communication (a spoke behavior) were rated as the top three failings in IT managers.

In addition, the IT industry tends to hire (and promote) the person with the best technical skills for a given project. Little thought is given to what is required to develop future corporate leaders. As in other industries and in the HP example, those promoted to the IT C-suite are more likely to be from marketing or finance than engineering, since these individuals are perceived to be more capable of leading (and growing) an organization than someone from engineering, computing, or another technical discipline.

Indeed, few IT organizations have internal programs that can objectively evaluate and enhance leadership — as opposed to project management — skills. Although a previous Cutter study has shown that virtually all attributes significant to the profile of a leader are observable and quantifiable, in the absence of a formal evaluation process, simply knowing what to look for may in itself yield large benefits in terms of organizational performance.

User stories that are not based on real business stories will struggle to be innovative. The user story describes what happens at the interface and is mostly what the product owner thinks the user wants to do with the software. But without some innovative thinking, it is all too easy to provide just some incremental improvement and let it go at that. Let’s look at an example of a user story that was written without any real concern for innovation or the real need that the story is meant to satisfy: “As a bank account holder, I want to check my balance online.”

At first, this might seem a reasonable and obvious story. However, it can be made a lot better. Some authors suggest that the “so that” part can be omitted. We suggest very strongly that you always include the reason in your stories. While the Agile Manifesto favors working code over documentation — and there is a lot to be said for that — there is nevertheless a need for the development team to leave behind a guideline of its thinking. Without justifying the requirement — that means including the “so that” — future maintenance teams are deprived of a valuable clue as to why a particular requirement was included in the software product and, hence, what would be the effect of changing it.

Our first question is, “Why does the account owner want to check the balance?” Let’s revisit the story and this time look at the reason given for the requirement: “As a bank account holder, I want to check my balance online so that I can access my daily balance 24 hours a day.”

This is not exactly a good reason for checking the balance. The “24 hours a day” is slightly more enlightening but doesn’t really do more than tell us that the account owner might have nocturnal habits. Why does the account owner wish to check the balance? It is not something we do for fun, so there probably is some business reason behind it. We just don’t yet know what that is. Let’s make some conjectures.

Suppose the reason for the frequent checking is that the account owner is on a tight budget and is concerned about becoming overdrawn. If so, the owner of the product, presumably a bank, has the opportunity here to be more effective and at the same time provide a better service. Instead of the account owner having to repeatedly check his or her balance to see that it’s not going into the red, it would be better to build a feature that notifies the account owner if the normal monthly payments such as rent, electricity, school fees, and so on, will reduce the account balance to zero or beyond: “As a bank account holder I want to be informed if my monthly balance is projected to go to zero or below so that I can arrange for an overdraft.”

Furthermore, we suggest that it is far more useful for the account owner to be periodically informed of the amount of discretionary money in the account: “As a bank account holder I want to be informed of my projected balance after all regular monthly payments have been deducted so that I know how much I can safely spend.”

Having a feature that lets the account owner check the balance online is the simplest feature to have. However, we suspect that without any kind of business thinking, or innovative thinking, it would probably turn out to be a feature that did not solve the real problem. If other banks solve the real problem (i.e., they understand the real needs of their customers and offer this service to attract more customers), then the original story could hardly be said to providing real business value.

When a business analyst investigates a business story, he or she is consciously trying to encourage innovation by first making abstractions that uncover the real business problem. Figure 7 illustrates the skill of looking at the same business story from several different points of view and thereby discovering undreamed of innovations that could make a significant difference to the business.

Figure 1 — The Brown Cow model illustrates four points of view that help to uncover the real business problem and identify useful innovations.

In the bottom-left quadrant of the Brown Cow model 6 the viewpoint focuses on how things are done now. This is commonly what business people ask for — a little bit more of what we already have. In the bottom-right quadrant the viewpoint focuses on how things could work in the future. Once again, it is likely that someone who has a requirement will express it in these terms. In other words, like our nurses in the previous example, rather than asking for what they really need, they ask for a solution.

As part of his or her analytical skills, the business analyst also learns to explore the problem in a solution-neutral fashion. We refer to this as the ability to “think above the line.” The line in this case is the horizontal line that separates the top and bottom quadrants of the model — the abstract thinking about the real problem from the technological view of the solutions.

The top-left quadrant focuses on what we do now, independent from how it is done now or how it might be done in the future. This view uncovers the essence of the problem: the business rules and the business data that has to be there independent of any solution. Above the line, the business analyst exposes the real business requirements by stripping away all solution-oriented aspects and thereby coming up with a policy-only statement of what the business is really doing. The top-right quadrant of the model is where the business innovation happens. It is here that the business analyst can make suggestions about improving business rules or using existing business data to be able to make better business decisions.

Innovative business value discovered by this sort of innovative thinking are then included in the business stories. These are innovations that could never be discovered by focusing on a software interface.

• Protecting the business from change and mitigating or benefitting from any change that might occur
• Mitigating risk that arises in any area of business and technology
• Identifying and decreasing reliance on nonresilient programs, processes, and procedures
• Enabling the business to adaptively respond to internal and external pressures
• Strengthening and enhancing business processes by improving governance and flexibility
• Using disruptions to improve efficiency and develop more effective responses, rather than merely focusing upon survival
• Reducing insurance costs and exposure to uninsured losses by providing enhanced oversight and a better understanding of response

The goal of resilience is not just recovery or continuity, but transformation from reactive to proactive and then to adaptive. Recovery efforts have demonstrated the value of several principles that constitute the core of business resilience, including:

• Mission focus. All resilience measures need to focus on the core business processes and values of the company at all times, ensuring the confidence of customers, partners, and suppliers. This is often critical to recovery, particularly from situations that might impact the reputation of the firm.
• Risk management. A thorough risk management program is essential, including a business impact analysis and development of mitigation strategies for key threats. Risk needs to be understood and evaluated in a unified way across the enterprise.
• Agile local empowerment. Empowerment of managers and workers who are close to the crisis, either geographically or by specific area of the threat (IT security, for example), ensure that lines of authority are known and response can begin immediately.
• Agile structural flexibility. The capability to realign organizational structure to meet new needs of the changed business or technological situation is vital. This realignment may require immediate and radical changes to existing hierarchies.
• Agile collaborative capability. The capability to cooperate with other businesses, outside organizations, and others, as needed, to meet recovery demands is also important. This is often critical to success, particularly in catastrophes that involve an entire region or business sector.
• Practice and rehearsal. Practice, rehearsal, and simulation of recovery locates weaknesses in resilience and strengthens the ability to recover. Rehearsals and simulations demonstrate the current recovery capability and prepare employees to respond to the real thing.
• Absorption of new learning. This involves the capability of learning from adversity and developing stronger and more focused business processes, using the disaster as a platform for beneficial transformation.

Resilience has become a concern of businesses operating in every sector and risen to the fore with events such as earthquakes in New Zealand and Japan, flooding in Australia and Thailand, hurricanes and flooding along the US East Coast, and fires in California. Other events have included the global economic crisis and major recent technological changes involving mobility and cloud IT.

Concepts of agility, adaptability, and resilience have been in development for at least the past four decades, with both progress and need accelerating recently in the wake of economic and technological change. These concepts, which began as lean manufacturing, have gained further validation through work on complex adaptive mechanisms and through practical application at the process level in software development, where methodologies such as Scrum and XP have provided significant benefits to companies in meeting the challenges of change.

As a result of growing concern, resilience has come under investigation over the past several years, particularly in the context of integration with existing IT solutions. Because it involves other aspects of risk management, auditing, and compliance and must incorporate the whole enterprise, we have seen the emergence of several frameworks and approaches aligned with existing management frameworks, such as ITIL, COBIT, and CMM. These frameworks, including the IBM Resilience Framework and its Resilience Maturity Model (RMM) and the CERT Resilience Management Model (also RMM), provide guidance for incorporating various elements into a resilience program.

The June 2012 Cutter IT Journal with Guest Editor Bill Keyworth, seeks to identify how ITIL can be used effectively to satisfy the customer goals of IT service management and how IT operations can balance the conflicting demands of IT process and business needs.

Please send us your ideas – proposals of interest are due 6 April 2012.

To respond, please visit

http://www.cutter.com/content-and-analysis/journals-and-reports/cutter-it-journal/callforpapers02.html

I know you’ll enjoy getting to know Joe. You’ll able to interact with him more, here on the blog, and of course, if you’re a subscriber, you’ll benefit from his insights in the pages of his quarterly CBR issues.

Part of the complexity with governance is that it varies widely and is a tradeoff of constraints and flexibilities. The telecom industry in the early 1990s illustrates one view of governance. This was a time of great expansion of ideas/capabilities (e.g., cell phones, network features such as voicemail, data transport in addition to voice) but also a time of great constraint (50% of customers did not even have touch-tone phones — remember those circular dial faces!, and the connections were twisted-pair copper wires, not higher performance wires, cables, or fibers). At that time, the enterprise architecture — the way calls were switched, the way signals were sampled and processed by software, the way endpoint devices (phones) worked, and so on — was a set of relatively immovable constraints. This is not to say that the role of architecture (and the architects) was to limit innovation, but rather help “engineer” tradeoffs based on the current reality to encourage workable innovation and delivery. There was no doubt, though, that governance was there to enforce architectural constraints, and that the flexibilities were limited to understanding what those constraints did and didn’t mean. Outcomes included ways to deliver network services, such as caller ID, call waiting, network voicemail, and so on, to subscribers that had the most basic handsets and far-reaching features like “number portability” (the precursor to today’s ability to migrate phone numbers from landlines to cell phones or between providers). There were disruptive transitions, such as adding area codes or splitting them, that needed to be managed.

Another view of governance is that it manages the roadmap from the “as is” to “to be” architectures. The nature of the concerns can range from explicit transitions (e.g., moving from one document management solution to another) to abstract goals (e.g., adoption of XML, SOA, or REST) to operational concerns (e.g., the JDK version or App Server version used for implementation). In this view of governance, the architectural transition is driving the change, and the enterprise is shifting from a state of compliance with one set of standards to compliance with another set of standards. An audit function is performed to evaluate if each project/delivery is to either be part of the status quo or part of the managed change. This audit often occurs so late in the process that there is not much that can be done if the answer is other than what is expected. The audit and the mandate for compliance may occur at an idea level, not based on an integrated end-to-end capability that involves the details of how the transition was applied.

Enterprise governance ultimately provides answers to the following questions, and the considerations for how to answer them in an agile enterprise are significant:

• What third-party technologies are we using across the enterprise, and is that consistent with what we want to be using (vendor, version, licensing, migrations, policies on open source usage)? Agile methods use the term “situational awareness” as necessary to understand how a project is doing so that next steps can be evaluated. Governance provides an enterprise-level situational awareness.
• What are the third-party technologies being used on a given project, and are those from the approved list of technologies (including transitions)? The system-level projects may or may not be able to avail of the full set of options (i.e., variants) within the enterprise, so there is a fit-for-purpose analysis that is made. In the agile development process, that purpose and fit may emerge rather than be planned and this will need to be addressed with governance.
• When have technology choices changed on a project/system, what are those changes, and are those changes expected/approved? This is similar to the above, but emphasizes that much of the desire/need for technology transition is discovered at the systems project level. There are big ideas that might be able to flow from top down (e.g., adopt messaging over remote procedure call (RPC), or adopt document management over file storage), but within those large domains, the discovery of what works and doesn’t comes from the projects. Governance needs to be aware of these transitions, allowing for a managed, bottom-up innovation.
• What are the internal supplier-consumer (integration) relationships between projects/systems? Governance is not just about technology usage within a system, but also integration between systems. Integration has a technology underpinning, but the more significant concerns are at the level of interface models, data exchange models, and the ability to test system-of-systems integrations for functional, nonfunctional, and error-handling issues. Agile development emphasizes a continuous integration approach that necessities an integration environment that is always available (to each system/project), always up to date (and able to be moved to different dated revisions easily), and supports automated testing. But these responsibilities to the system-of-systems integration are, by their nature, bigger than any one system, and hence effective governance can make this workable and efficient.
• Are the third-party technology and integration choices across projects/systems compatible (note: this does not need to mean identical)? The key is compatibility, not conformity. This is a transition in the governance approach for standardization but appeals to the sense that “one size does not fit all,” and what organizations really need is a way to realize that objective in a way that supports agility without leading to chaos.

Truth is, neither of these architectural paradigms can be agile because they both result in inflexible juggernauts that defy economical impact analysis when new features are needed. Both approaches scale to installations that are so inscrutable and fragile that it becomes more economical simply to solve new requirements with new applications rather than updating the existing assets. Both the Inmon and Kimball approaches leave companies struggling with “legacy” warehouses.

One of the most promising solutions to this juggernaut problem revolves around data architectures that go beyond our traditional 3NF. Warehousing teams have achieved more robust designs when they push their target schemas past fourth and fifth normal forms into a variety of “hypernormal forms” (HNF). These forms include “data vault” forms, associative strategies, and even CJ Date’s new sixth normal form, where all attributes of a source schema should be shredded into something akin to key-value pairs when stored in a target system. My recent year of fieldwork revealed that, indeed, hypernormalized approaches yield warehouse designs that are far more robust in the face of changing requirements.

There are several tool suites and practice communities that have demonstrated the commercial viability of HNF for large projects with tight budgets and even tighter deadlines. What I particularly appreciate is how these new solutions truly enable agile data warehousing by enabling and accelerating incremental warehouse delivery patterns.

In terms of engineering advantages, HNF yields target schemas that are what I call “three-way robust”:

1. Teams can add small increments of functionality without undertaking massive reengineering.
2. Modifications to existing database tables involve only local impacts.
3. As the model grows, the cost of scaling the warehouse increases less than linearly.

My fieldwork’s most remarkable discovery, however, was how the epitome of hypernormalized tools truly enable the fourth tenet of the agile data warehousing manifesto, where we focus on “evolving data models over incrementing application code.” There are tools that can take business models and automatically generate 90% of the integration, presentation, and semantic layers for a team. By leveraging such technology, warehouse developers can focus on the remaining 10% where the business rules are complex and hand coding adds the greatest value. Such tools make the warehouse project and the entire organization agile because IT and business can now collaboratively model a small sliver of the warehouse, generate a user-tangible result, and evaluate it together. With the distraction and time lost to coding now reduced to a minimum, IT is able to rapidly address additional requirements step-by-step by evolving the information users can touch, keeping the business constantly involved in the partnership, and rapidly zeroing in on the exact operational insights the current business challenge demands.

So, the days of endless Inmon/Kimball debates are over. New tools and practices have moved the pivotal question far beyond “star schema versus third normal form.” DW/BI project kickoffs should now be immersing themselves in the question: “Which hypernormalized modeling technique and supporting toolset will work best for our organization?” It is the astounding improvements in programmer productivity and customer satisfaction this new generation of solutions allow that demands we make this change in mindset.