The story could be also about …

• On-Prem → Cloud → Hybrid-cloud / Cloud-Repatriation
• Server-side rendering → Client-side rendering → Hybrid rendering (client-side hydration)

Does this sound familiar?

The pattern behind this story: In software engineering - maybe not just there - we alternate between different solution approaches. But it’s not just that.

Why is that? Speaking from my experience I see two possible reasons.

Reality vs. Theory

We focus so much on the cause (in the example above “close coupling”) and how to fix it that we become blind to everything else. We forget that there is no free lunch … every new solution has certain characteristics. These can lead to other challenges. In the example above, it’s the complexity that arises when handling many microservices (e.g. runtime dependencies, interface compatibility). However, these downsides are on a theoretical level. So, we don’t take them into account. The real problem gets overrated, the arising but so far only theoretical problems are getting underrated.

The perfect the solution

Another reason could be striving for the perfect solution. But as said above, there is no free lunch, meaning: There is no perfect solution, only an optimal one for the current situation. And since every situation changes continuously, the optimal solution changes as well.

Whatever it is, alternating between extremes is an expensive and lengthy undertaking. It creates high opportunity costs, requires a long-running commitment, and has an increased risk to get stopped.

I will shine a light on how to do it better in the next article.

In the last article I shared a fundamental principle of Trusted Shops’ software engineering organization.[1] It makes sense to read it first to understand one or the other aspect better. This time it’s about the role of an Engineering Manager at Trusted Shops.

Responsibilities

Given the separation of “Working IN a system” and “Working ON a system”, the Engineering Manager is purely focussing on the latter. An Engineering Manager is responsible for multiple Engineering teams. (There are no Team Leads in the software engineering organization. More on the reasoning behind this in a separate article.) The responsibilities boil down to two buckets.

• Disciplinary responsibility: This is meant for all technical roles (Software Engineer, QA Engineer, etc) and mainly focuses on talent development and hiring.
• System responsibility, where “System” relates to the environment teams are working in: This is mainly about the continuous optimization of the setup, solving systemic issues and giving guidance where needed. At Trusted Shops, we are not heading for a dominant disciplinary structure. Quite the opposite, we strive for servant leadership, meaning “Working ON a system” serves “Working IN a system”. The focus is on the teams and the operational work. This strengthens the customer focus and it inevitably ensures that teams become stronger and mature.

Engineering Manager is technical role but with on a different flight level

Serving the engineering organization, that doesn’t mean that engineering manager is a weak role. That’s not the case. And even if coding and other hands-on technical work is explicitly out of scope of this role, it is a technical role.

Engineering Managers are also leading certain technical decisions, especially one-way-door decisions. This kind of decision is hard to change later and therefore requires additional attention. An example could be the introduction of a new programming language.

This example also gives a glimpse about the technical focus. It’s more the tactical and strategic level rather than the operational level.

Link(s)

[1] Working ON a system & Working IN a system

As context always matters, let’s start with it:

• There are roughly 130 people across all functions working on the products and services of Trusted Shops.
• Trusted Shops is not a startup anymore. But we are still in an evolving game, which requires us to adapt to learnings, market changes, etc.
• The organizational setup is guided by two pillars of the Tech Strategy: Autonomy on organizational level & Empowerment and ownership on team level

System-thinking and beyond

Now that the context is clear(er), let’s look at the fundamental principle:

• We understand our software engineering organization - actually product engineering organization - as a system. It’s composed of smaller interrelated subsystems (domains and teams) who work on different products to achieve an overall goal. (If you want to learn more about organization as system, this is a good source. [1])
• We separate between “Working on a system” and “Working in a system”. This separation is important because both aspects have different goals and require different needs to achieve them.

Many companies use their disciplinary structure to handle operational work. To illustrate this, one example are Team Leads or Head of Engineers who take technical decisions for their teams. Operational work dominated by the disciplinary structure works fine for repetitive work. But it hardly works for knowledge work. Knowledge work is about solving complex problem(s) which requires expertise, critical thinking and interpersonal skills.

Working IN a system

To address this demand we highlight the operational structure. It’s described as “Working in a system”. “Working in a system”, there are no disciplinary hierarchies. There are only individual contributors with different roles and technical responsibilities. The decisive factors are skills and experience. Cross-boundary collaboration (e.g. to manage dependencies) is handled via network approach. This means, there are direct connections between teams where needed and when needed.

Working ON the system

Working on the system is represented by the hierarchical structure. In our case these are Engineering Managers, Director Software Engineering and CTO. The goal is to provide an effective environment on all levels. This starts with the organizational setup and the hiring and the development of talents. But it’s not “people business” exclusively. It’s also about the involvement in one-way-door decisions, defining standards, applying best practice and solving systemic issues - to name a few -, always based on the mindset to measure and improve the environment. This work happens more on a tactic and strategic level. It still requires a technical background even though it contributes only indirectly to the overall success.

At the end the disciplinary structure is serving the operational structure. But this doesn’t mean it’s less important. The system can only unveil its potential if both complement each other.

Link(s)

[1] The Common Thread: All Social Organizations as Systems

37signals successfully showed that cloud repatriation is feasible and has the potential for huge cost saving. Is this the way to go? No, it’s definitely not for every company but for some it can be a valid option. It “just” requires the right goal and a supportive context.

In 2023 37signals left the cloud. They successfully repatriated their cloud services to on-premise servers, saving a significant amount of money without compromising on performance, reliability, or security. AND they achieved this while retaining their existing team, disproving the notion that cloud services significantly reduce operational staff requirements. Absolutely astonishing! The entire journey is quite well summarized here (including the most frequently asked questions). [1]

So what does this mean for other companies?

Shall every company repatriate cloud services back to on-premise?

The specific answer to this boils down to two questions.

#1: What’s your goal?

This question is of key importance. It defines your focus and the single piece you want to achieve. In regard to 37signal, their only goal was cost savings. But it also could have been new products, new features or technical aspects like performance or resilience to name a few.

It’s important to have a clear answer because capacity is only available once. That’s why mixed goals (this and that) or vague goals (“becoming better”) do not help. Once your goal is set, you have to deal with a second question.

#2: What’s the context you are operating in?

Every company is different, so every context is. The context is defined by all aspects impacting the decision. It’s crucial to understand all limitations and restrictions. In regard to cloud repatriation the following aspects (among others) are of interest.

• Costs: What’s your cloud cost structure? Can you afford the high upfront investments to buy servers and such?
• Performance: Do you have specific requirements, e.g. super low latency which require a special setup, e.g. CDN?
• Control, security and compliance: Are you bound to certain standards and customer agreements?
• Scalability: What are your realistic growth expectations? What’s the impact on infrastructure utilization?
• Technical expertise: Do you have the skills to take care of lower layer topics? Is your technology stack capable of running outside the cloud?
• <add your individual aspects here>

If you have the answers to both questions you have all the ingredients to make a proper decision.

Even though cost saving is your only goal, this is not sufficient to justify a cloud repatriation. It also requires a supportive context, like in this example:

• Cost analysis favors on-premise: There is enough cash to do upfront investments and participate from long-term cost benefits of owning hardware versus ongoing cloud service fees.
• Technical expertise available: The organization has or can acquire the necessary technical skills for managing on-premises infrastructure.
• Control and customization needs: There is a strong requirement for direct control over the hardware and software environment.
• Security and data sovereignty: Enhanced control over data storage and security is crucial, especially for compliance.
• Stable demand: The company experiences predictable, stable demand that aligns well with the capabilities of on-premises infrastructure.

37signal had very likely a comparable context back then. They had the money, the skills, the stable demand, etc. So they decided for a repatriation of their cloud usage to an on-premise environment!

Finally

Given the different aspects and their characteristics, for the majority the context will - most likely - not speak for a cloud repatriation (regardless of the goal). Still, 37signal might have paved the way for more companies to leave the cloud.

Link(s)

[1] The Big Cloud Exit FAQ

This metric isn’t new. It belongs to McKinsey’s standard toolbox to measure organizational efficiency and productivity. It differentiates between core development activities (inner loop) and supporting tasks (outer loop). The goal is to optimize the inner-outer-loop ratio. But is it a good metric for software engineering teams? Let’s dive in.

Based on McKinsey, the inner loop encompasses “activities directly related to creating the product: coding, building, and unit testing”. The outer loop includes “other tasks developers must do to push their code to production: integration, integration testing, releasing, and deployment.” From their standpoint “maximizing the amount of time developers spend in the inner loop is desirable”.

While I like the overall separation, I disagree with the definition/ assignment of activities to the inner/ outer loop. From my point of view all activities from building to deploying software in production belong to the inner circle. Why? Simply because otherwise there is no value! And secondly, you split apart the ownership of a team.

You could argue that you want to streamline efforts for testing and deployment and therefore these activities should belong to the outer loop. But reducing efforts should be done and measured by degree of automation not reduction of scope. Otherwise you deprioritize long-term effects.

So what’s left for the outer loop? I definitely see all surrounding activities (e.g. alignment meetings). These activities are important as well. And while automation is hard, usually there is lots of room for optimization … talking from my own experience.

PS: This post also got posted at LinkedIn

Yes, QA changed quite a lot during the last decade - btw, same as software engineering!

Often, there is this saying that QA should be done by Software Engineers. What’s actually meant by this is a high degree of test automation. And I absolutely agree to this. However, there are two issues …

1. Environments with high degree of test automation in all aspects rarely exists. This means, manual QA is still required.
2. Even if you are working in such an environment - lucky you -, you will realise that the mindset of QA Engineer and Software Engineer are very different. Coming up with good test cases is a dedicated skill.

So what?

In my humble opinion QA will stay. It’s just the approach which changes based on your context.

PS: This post is a reaction on a discussion at LinkedIn

So, how to fix that? If you have veto rights, it’s a clear indicator that delegation of decision making/ power is broken somewhere in your system. Delegation poker is a simple but powerful - and still my favourite - tool to create transparency and alignment between different parties who’s calling which shot. It’s part of Jurgen Appelo’s Management 3.0.

Delegation Poker, Management 3.0 by Juren Appelo

Btw, there is no problem - as a manager - to take more decision power (and therefore delegate less) at the beginning. Delegation poker an explicit and transparency approach, which allows everyone align with.