Having observed this approach for 6 years, I’ve only grown more confident in its success. Part of that confidence comes from seeing it play out. Our team vastly simplified maintenance, reduced bugs, and boosted performance by making the jump. (The jump was made possible by Long Term Refactors btw.) Another reason for my confidence is the many comments I’ve received over the years confirming that it works for other teams too. Finally, I’ve received a number of challenges and condemnations, although they were either entirely theoretical, or based on a misunderstanding of the article. Since I consider it somewhat my fault for not being clear enough, I want to address these misunderstandings. So here they are, organized into topics, each one addressed in turn.

1. You reinvented HTML!

Multiple readers have informed me that I can apparently serve HTML directly from the server, instead of doing all that JSON payload nonsense. Having started building websites in 2003, I get it. However, the reality is that today we work with front-end teams, they work with React, and React comes with certain practices. 15 years of them in fact, in addition to the baggage of pre-built components. That’s worth some respect. If a JSON payload is preferred, and I have no trouble delivering it, why would I mind? Personally, I much prefer the good old Ruby on Rails-esque stack with server-side HTML rendering, but we work in teams, and should probably play to each other’s strengths.

That said, the bigger issue with the “you rediscovered HTML” crowd is that they’ve misunderstood the advice. HTML involves content, structure, and style (esp. in the case of Tailwind) — which is basically everything you can serve, aside from assets. In our JSON world we only serve content, a faint hint of structure, and no style at all. That’s way higher level than HTML. It’s important to understand that I’m not asking the back-end to serve JSON.dump(html) to the front-end, only the gaps the page requires to be filled. The rest of the page’s static content should just be hardcoded on the front-end.

I do, however, need to correct one mistake. My old advice went: “content and structure come from the back-end”, but I didn’t realize that folks would interpret it as literally as “serialize the entire HTML into JSON”. That’s not at all what I meant. You only need the bare minimum JSON structure to help the front-end engineer understand which values should go into which parts of the page. For example, if your HTML is something like this:

<div>
  <article>
    <h1>Title</h1>
    <p>Body</p>
  </article>
</div>

You don’t need this: { "div": { "article": { "h1": "Title", "p": "Body" } } }. You simply supply the keyword arguments to this otherwise hardcoded ArticlePage’s constructor:

{
  "title": "Title",
  "body": "Body"
}

If there are multiple articles, you make an array of these. As simple as that. Cater to the needs of the page.

2. Pages will load slower without async!

There exists a concern that with my approach you can’t load parts of pages asynchronously, and that this would result in bad performance. If you are a front-end engineer who has only ever worked with generic API endpoints, I get why you have this impression. You needed 10 endpoints to render a single page, you parallelized the requests, you saw that they can be flaky and slow at random, you prioritized some over others. Right? I’m sorry, but that pain was self-inflicted. If you asked a back-end engineer to give you all of that data in one bundle, the server would most likely produce it in under 30 milliseconds in a single streamlined 200ms roundtrip rather than juggling 10x200ms roundtrips competing with one another.

The whole idea of asynchronously loading pages is a truism of the front-end world that only feels correct in theory. In practice, it’s like hiring 10 trucks to deliver 10 USB drives in parallel, realizing how slow it is to manage 10 trucks, and concluding “maybe I need some more trucks”. Async loading only makes sense when you are actually dealing with slow, heavy, or streaming data sources. If it’s your own back-end, attached to your own database, spitting out kilobytes of content, you are making things hundreds if not thousands of times slower by running parallel requests. On top of that, you’re making it harder for the back-end engineering team to bundle, optimize, and cache data, because it must be sent piecemeal.

Then there’s the reverse concern as exemplified by this whole thread, and this comment. It argues that you should be able to submit individual form fields to the server, rather than entire forms. Is that something you should do? The real answer here is: sure, if you want to. Nothing about my approach prevents you from submitting forms in any way you like. But just like in the situation above, performance isn’t really a good reason to split data (in most cases).

3. This makes no sense in a Single-Page Application!

Some folks have struggled to see how serving pages can work in the context of a single-page application. The answer is: just rename “page” to “screen” in your mind and you’re good to go. When transitioning between them, ship the next screen’s worth of data to the front-end in one bundle. If you need to fetch a specific screen’s sections individually, feel free to provide special endpoints for those. Although, in most real-world cases, reloading the whole screen’s worth of data only to swap out a single section would still be insanely fast and simple. Only watch out that you don’t provide raw data from the database. For example, if the front-end is showing a table of items, give it a table_items endpoint for this specific screen, where all the data is already pre-arranged for display in this specific table. Don’t make the front-end do the extra legwork of accessing multiple endpoints to wire this table together.

4. Why not use GraphQL? Or an aggregation layer?

A couple of readers have wondered: why not use an aggregation layer over a generic API, or switch to GraphQL.

Not gonna lie, an aggregation layer sounds absurd to me. It might be a symptom of front-end-centric thinking. We first build a generic API, thereby creating the problem. Then we build another layer that hides the problem without solving it. This leaves us with twice as much back-end code, double/triple the back-end complexity, and all the same performance issues, while requiring more time and effort. Why?

As far as GraphQL goes, the reason you might want to avoid it is that it comes with an enormous complexity and development style trade-off. GraphQL is a web of infinite possibilities that back-end developers must be able to manage and secure. Supporting such an infinite maze starts to make a lot of sense when you consider how many types of clients Facebook has. Their clients span from computers and phones to TVs, fridges, and washing machines. For them, producing an insanely flexible layer that allows thousands of unique clients to fetch unique sets of data is probably worth it. How many unique types of clients do you have? Let me guess: one website and one mobile app (for most of you). Maybe not even. Should you be supporting an infinitely flexible query language for this?

Remember the problem you’re trying to solve: providing the data your front-end needs to display your software. Just go ahead and provide what’s needed. Problem solved.

5. You took away flexibility from the front-end!

Perhaps I didn’t communicate this clearly enough in the original article, but people are still telling me that having a generic API enables front-end flexibility. They talk about losing the ability of the front-end to build new features and do redesigns without communicating with the back-end.

There is no world where the front-end can build new features or redesigns without involving the back-end. Yes, they can use old endpoints in new, unexpected ways. But then, the back-end will have to catch up to the mess of unpredictable patterns of server bombardment that the front-end introduced, lose all track of what endpoints got used for what purpose, and try to post-hoc optimize inefficiencies, creating a half-baked version of the solution I proposed in the original article.

The cost of this is that the back-end will indefinitely have to support all unexpected usage patterns, and be afraid to change or remove anything, because it’s very hard to trace exactly how and where the front-end relies on specific endpoints. You will need to deal with API versioning, and a forever growing (i.e. never-shrinking) codebase. To make matters worse, there isn’t even that much flexibility gained, because for redesigns and new features, the front-end is still at the mercy of what the back-end provides, and will still probably require new endpoints built for it. Meanwhile, old endpoints will never be removed “just in case”.

On the other hand, you could vastly simplify your redesigns when the back-end serves the exact data needed for each page. You can redesign each page separately, and never wonder if something is being used in unexpected ways.

Bottom line is, there is no real flexibility to be gained from a naive CRUD API based on database records. The cost of making the back-end actually flexible for diverse use cases is much heavier than many teams realize. It only sounds nice in theory.

6. What do I put into the payload?

One of the most important questions I keep getting is exactly what data should be provided to the front-end, and how it should be structured and named. As mentioned before, some people erroneously think that I was suggesting to send essentially the entire HTML auto-translated into JSON. Others thought I was suggesting to create some sort of JSON-based UI-construction protocol. And yet others were just not sure what to do with state and static content that exists entirely on the front-end. Should it come from the back-end too?

The answer to all three is: you’re overthinking it. The front-end developer should hardcode a page as much as possible, and only leave gaps for what makes sense to come from the back-end. The data to fill those gaps should be arranged in a neat little JSON payload. End of story. Hardcode all static content on the front-end, keep all front-end state on the front-end, ship things that the back-end controls from the back-end.

Furthermore, do not try to make your JSON formalized and consistent across different pages. Each page should have its own custom code that takes JSON values and puts them in the right places in the right way for this one page. You may have certain very similar components on multiple pages that you might want to normalize arguments for, but don’t try to normalize overall page structure. It will only make things difficult. Just do what one specific page requires. Your JSON is nothing more than arguments into the constructor for this one page. A different page should have a different JSON structure. Any similarities between overall page structures should be accidental.

7. CRUD makes back-end easier to maintain!

Some comments insist that CRUD is easier to build, document and test than custom pages. This is a misunderstanding of the term. CRUD isn’t supposed to let the front-end CREATE/READ/UPDATE/DELETE your database records, it’s meant for doing that with resources. Figuring out what those resources are is the key to architecting your web app correctly. They may occasionally map directly to database records, but more often they won’t.

A page is a resource with a READ endpoint. The CREATE/UPDATE endpoints are rarely useful for pages, but they are useful for more granular resources, like various items and relationships. Sometimes they map to DB records, but more often you need a resource at a higher level of abstraction. “Form objects” can play that role. They can represent an end-user entity that “changes together”, served as a single form on the front-end. Your controller would then take the data from this object, and transactionally commit it to however many records in the DB are backing it.

All that is to say, the idea that “CRUD is easier to build, document, and test” is really saying “exposing my database records directly to the front-end is easier to build, document, and test”. By doing this, you skip the whole app, and expose low-level storage directly to your front-end. No wonder it’s easier. This shifts your entire app to the front-end, where composing data comes at a huge cost of network failure modes. In a way, I guess you did make development “easier”, because the front-end often gets a pass for omitting tests and docs.

8. What is “General Purpose API”?

Someone asked me for a definition.

When I say “don’t build a general purpose API”, I mean an API for a wide variety of public use cases, available for use by your customers. I’m instead advocating for a BFF (back-end for front-end) API, where you only cater to your front-end team’s requirements, and don’t make this API generally available. If you actually do need a general API, build it separately from your BFF API to avoid clashing requirements, and unnecessary release management + documentation overhead.

9. How is this applicable in the AI era?

This is a question I’m asking myself about all of my programming-related writing. Perhaps you can feed my articles into an AI, so it can follow my advice for you? Who am I kidding, they’ve already been ingested by LLMs and diluted in the ocean of other blog content. I don’t know. Be positive and have fun, I guess, everything is going to be okay.

Right now, self-driving cars still require human monitoring and intervention (outside of specially-designated areas). Isn’t this also true of a sufficiently complex system where you might need to intervene quickly in case AI fails to resolve an issue? Worth considering, right?

You might say — so what? AI-written code is arguably better (or will eventually be better), often with more comments and docs, humans would understand it faster anyway. And that may be true, but with human-written code you can usually find a human who wrote it and ask them questions. If AI gets mixed up in too much context, and can neither fix nor successfully explain what’s going on, there might be nobody else familiar with the codebase. Are we saying that we should try to maintain some level of familiarity just in case?

You might say — well, we already have AIs capable of storing large permanent context, so it will know the codebase better than any human would. At some point AI will just become strictly better in every way. And that may be true, but I will keep asking the same question:

Can we forego human intervention? What if AI servers are down? Can we ever completely rely on a technology caring for itself?

If the answer is a “no”, even a tiny “no”, then doesn’t it kind of negate the entire “full AI takeover” narrative? As we start to unpack this chain of thought back down from “AI perfection” to “but human intervention might still be needed in rare cases”, aren’t we inevitably back at the question: What should we do to help humans intervene?

Once you start answering this question, you might find yourself back at square one:

• Humans will need to be reading and reviewing code at the very least.
• Humans will need to maintain good understanding of the codebase.
• The best way to learn is by doing. (I.e. writing the code.)
• If humans are expected to jump in to fix something, they should probably have the final say in its implementation.

It’s like one of those little “snags” that you hit, which seems insignificant at first, but as you drill down on it, it may have way bigger implications than you realized.

You might say — well, most projects out there are not that critical. That’s true, but what if it grows into a bigger, more critical one later?

Even the tiniest possibility of human intervention leads me to think that it’s always going to be better for software developers to work together with AI, and never simply be replaced by it. Otherwise, failover is going to fail in a situation where it’s needed most.

Where am I wrong?

Well, I dislike reading giant PDF docs, love writing Ruby, and there’s an awesome RubyLLM gem, and Gemini supports PDF parsing, so maybe I can just throw together a quick CLI tool that can answer questions for me? Alas, Gemini is limited to 1000 pages. Either way it would probably be too wasteful to send the entire doc every time. RubyLLM supports tools, so I decided to try that out.

Reading PDF Text Locally

My doc is mostly text, there isn’t any pics in there I care about, so this part is easy. A quick search later, there’s a gem called pdf-reader. Perfect for a tool.

bin/ask_api_doc

#!/usr/bin/env ruby

require 'ruby_llm'
require 'pdf-reader'

class PdfPageReader < RubyLLM::Tool
  DOC = PDF::Reader.new('docs/big-doc.pdf')

  description 'Read the text of any set of pages from the doc.'
  param :page_numbers,
    desc: 'Comma-separated page numbers (first page: 1). (e.g. "12, 14, 15")'

  def execute(page_numbers:)
    puts "\n-- Reading pages: #{page_numbers}\n\n"
    page_numbers = page_numbers.split(',').map { _1.strip.to_i }
    pages = page_numbers.map { [_1, DOC.pages[_1.to_i - 1]] }
    {
      pages: pages.map { |num, p|
        # There are lines drawn with dots in my doc.
        # So I squeeze them to save tokens.
        { page: num, text: p&.text&.squeeze('.') }
      }
    }
  rescue => e
    { error: e.message }
  end
end

Now my LLM can use the tool to extract text from any page.

And We’re Basically Done

Unlike “draw the rest of the owl”, the rest of the code is actually pretty straightforward (goes after the above):

# Grab key from my 1Password.
GEMINI_API_KEY=`op read "op://Private/Google Gemini API Personal/credential"`

RubyLLM.configure do |config|
  config.gemini_api_key = GEMINI_API_KEY
end

chat =
  RubyLLM
    .chat(model: 'gemini-2.5-pro-preview-03-25') # Pick a model.
    .with_tool(PdfPageReader.new) # Add the tool.
    .with_instructions(<<~TEXT) # Add general instructions.
      Use provided tool to find requested info in the multi-page doc. Ask for
      multiple pages at a time to avoid roundtrips.

      Respond only with results of your findings. Don't do ascii tables, I prefer
      text and bullet points.

      To find info, use table of contents. Make sure you scan the full table of
      contents before you give up. Don't go to irrelevant parts of the doc unless
      absolutely needed.

      Total number of pages: 1249
      Table of contents is on pages: 31-49
    TEXT

response = chat.ask(ARGV.join(' ')) { |chunk|
  print chunk.content
}

# Some stats at the end
puts "\n\n-----------\n"
puts "Input tokens: #{response.input_tokens}"
puts "Output tokens: #{response.output_tokens}"
puts "Total tokens: #{response.input_tokens.to_i + response.output_tokens.to_i}"

That’s it.

Now I can ask a question and sit back, watching the llm scan table of contents, read relevant pages, and spit out a catered response. Pretty nice!

(Below is just sample output, not what’s really in my doc.)

❯ bin/ask_api_doc "what are all available statuses?"

-- Reading pages: 31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49

-- Reading pages: 1123

The available statuses are:
- `ACTIVE`: The default status for a new object.
- `INACTIVE`: The object is inactive and cannot be used.
- `PENDING`: The object is pending approval or activation.
- `ARCHIVED`: The object has been archived and is no longer active.
- `DELETED`: The object has been deleted and cannot be recovered.
- `SUSPENDED`: The object has been suspended and cannot be used.
- `EXPIRED`: The object has expired and is no longer valid.

-----------
Input tokens: 95288
Output tokens: 643
Total tokens: 95931

I bet there are more involved “talk to your docs” solutions out there, but this was quick and easy, and I can tweak it as needed. Speaking of which, let me know if you have any ideas for improving this.

Update (2025-05-26): Since I wrote this, I slightly extended it with a search tool based on pdfgrep:

class PdfPageSearch < RubyLLM::Tool
  DOC_PATH = 'docs/big-doc.pdf'
  description 'Get page numbers by a PCRE regular expression.'
  param :regex, desc: 'PCRE Regular expression to search by, case insensitive.'

  def execute(regex:)
    command = "pdfgrep --color never -inP #{regex.shellescape} #{DOC_PATH}"
    puts "\n-- Running: #{command}\n\n"
    output = `#{command}`
    pages = output.split("\n").map { _1.split(':').first.to_i }.uniq
    puts "\n-- Found results on: #{pages.size} page(s)\n\n"
    { pages: pages }
  rescue => e
    { error: e.message }
  end
end

and added it to RubyLLM like this:

chat =
  RubyLLM
    .chat(model: 'gemini-2.5-pro-preview-03-25')
    .with_tool(PdfPageReader.new)
    .with_tool(PdfPageSearch.new) # <------- HERE
    .with_instructions(<<~TEXT)
      ...
    TEXT

Also switched from Google Gemini to OpenAI o3, and together these changes considerably improved the search performance.

• Convince business that it’s worth the delay.
• Decide what features will have to wait.
• Produce regular status updates and ETAs.
• Justify the refactor as we go. Is it the right approach?
• Keep ourselves from burning out.
• Allow time for the team to digest and review the huge diff.
• Fix a bombardment of QA issues.

And we better do this all quickly, because god forbid original and refactored code coexist!

Is this really the only way? Feature freeze, a rush, a buggy rollout, and likely burnout?

The Other Way

I have a theory that long refactors get a bad rap because most of them take far longer than we expect. The length leads to stress, an awkward codebase, a confused team, and often no end in sight. Instead, what if we prepared an intentional long term refactor? A few years ago, I began trying this method, and it has led to some surprisingly successful results:

• We didn’t need to negotiate business timelines.
• We didn’t need to compete against business priorities.
• The team quickly understood and even took ownership of the refactor over time.
• There was no increase in stress and risk of burnout.
• PRs were easy to review, no huge diffs.
• The refactor was consistently and collaboratively re-evaluated by the entire team.
• We never wasted time refactoring code that didn’t need it.
• Our feature development remained unblocked.
• The team expanded their architectural knowledge.
• The new engineers had a great source of first tasks.
• We rolled out the refactor gradually, making it easier to QA, and reducing bugs.

Long-term refactors involve the whole team from the beginning, which is one of their most powerful aspects. So far, I’ve participated in ~10 big refactors using this method across 2 companies with at least 3 different teams, and I’ve yet to see it go wrong. Here was our approach.

Prerequisites

To start, you should have the following:

1. An experienced software engineer with a vision for the refactor.
2. A team of software engineers at various levels of expertise.
3. An internal knowledge base. (Any of Github Wiki, Notion, Confluence, Markdown files, etc)
4. Less than ~5-10 long term refactors already in progress, depending on their scope.

Process

Almost every big refactor I’ve encountered follows a semi-consistent pattern. What makes a refactor big is the sheer number of times you must apply the pattern. In an ideal world, this labor is divided. Unfortunately, the refactor often requires case-by-case decision making. My proposed process is centered around explaining the refactoring idea to your colleagues, so that they can also make decisions.

NOTE: The process is for the “experienced engineer” from prerequisite #1.

1. Identify code that should be refactored.
2. Identify the refactoring pattern.
   Explore the codebase to identify a common pattern of required changes. A rough idea is fine for now. It’s okay to ignore special cases and focus on the commonalities.
3. Implement an example of the refactor.
   Find the smallest representative sample that you can apply your rough pattern to, and refactor it. This is where you want to be extremely diligent. Experiment and thoroughly refine your pattern. Don’t skimp on best practices. Follow 4 reasons to leave a code comment. Convey how, what, and why. Make it your best work, because it’s going to become the primary reference for the rest of your team. Submit a merge request.
4. Prepare the codebase for the refactor.
   Now that you’ve tried the refactor yourself, made decisions, and fought friction, you have an idea of what your colleagues are going to be dealing with. Use your experience to pave a smoother path for them. Go through the codebase and do minor preparations: reshuffle code, fill gaps, rename things for clarity, resolve ambiguities, create new dir structures. Keep your changes minor. Don’t refactor everything yourself. It’s critical that the bulk of the work is shared. Submit a merge request.
5. Name your refactor.
   Give your refactor a convenient name for use in discussions and docs. Make sure the name is concise, clear, and descriptive. For example, “Remove dependency on [package X]”.
6. Write up refactoring instructions.
   Create a document in your internal knowledge base and title it with the name from step 5. Some tips:
   • State exactly what to do, and how to do it. Be brief and specific.
     • Do NOT tell stories: “Over the years we’ve realized that the method we’ve been using …”
     • Do list specific steps: “Find a class that has function X. Create new class named Y. Move function X into class Y.” The steps can’t all be plain of course, but challenge yourself to see how brief and specific you can make them.
   • Link your example merge request from step 3. People should see the code before and after.
   • Finally, feel free to add some context at the end. Here, you’re welcome to provide the background, tell stories, link to relevant resources and discussions. That said, do hide this part under an expandable element, like <details>. We want the doc focused on the pattern itself, with an option to expand the context as needed.
7. Add this refactor to the list of long term refactors.
   Make sure that there is a page in your knowledge base that lists all long term refactors. The document from step 6 should be added to that list.
8. Introduce this refactor to your team.
   Use either a written announcement or a meeting. Explain how to pick a chunk that needs refactoring, walk them through your example. Don’t forget to link the instructions you wrote in step 6. It’s very important that your engineering team is aware of every long term refactor currently in progress. That’s why you want to stick to just a few at a time, and properly introduce each one. Adding a new long term refactor should be a big deal.
9. Assign refactoring tasks.
   Your refactor is now ready to be done gradually over time, but I advise against creating all the tasks up front in the task tracker. That would destroy one of the main benefits — not wasting time on unimportant or soon-to-be-deleted code. Instead, create tasks as they naturally come up in planning. “Hey, since you’re going to be changing that, maybe remove dependency on package X while you’re in there?”. Moreover, I advise keeping the whole umbrella-refactor away from the task tracker, or at least from the areas where business can see it. A successful long term refactor should be tracked by engineers, not the company management. As long as it’s written in the knowledge base, and is always present on engineers’ minds, you should be good. It shouldn’t matter to the business whether the refactor is completed, and how long it takes.
10. Stay aware of long term refactors.
    Get every new engineer joining the team to read the list from step 7. Make sure you have this step in your onboarding process. This is also a great source of first tasks for them, to help them understand both the existing code, and the new direction. It’s easy to refer back to the list anytime (remember, the list must remain short), but engineers also tend to remind each other of these refactors when planning.
11. Complete the refactor.
    A long term refactor doesn’t need to be 100% completed. Instead, one day you will find that your doc is redundant, because the codebase already speaks for itself. If all the major parts are refactored, and there is no more confusion about your direction, feel free to mark it done. This creates space for the next one.

Having followed this process carefully, I’ve seen something awesome happen. The team got into the habit of self-assigning refactors as needed. When they had questions, they’d initiate discussions and meetings. This got everyone on the same page around decisions that might’ve been controversial if made alone. With each completed refactor task, we’d all gain new examples to draw from in upcoming tasks.

Compare that to working on your own for weeks or months, and blindsiding your team with a huge diff.

Drawbacks

Here are some that I can think of.

• Albeit rare, some big refactors don’t have a common pattern. It’s possible that you’re actually dealing with multiple refactors that shouldn’t be under the same umbrella. Try to split them instead.
• You need patience to get through these refactors. They can span a year, two years, who knows. During that time, the old and the new code will coexist, and might cause some confusion if the list from step 7 is not on everyone’s mind. I personally haven’t encountered this drawback in practice, because the process constantly keeps everyone on the same page. Due to organization and communication, nobody is confused about where we’re coming from, and where we are headed.
• Certain parts of the code may never get refactored. There’s probably a good reason why. It could be that this part is easy to maintain as is, and doesn’t need to change. Or perhaps this code is on its way out. Think of it as a win — you saved time and didn’t introduce bugs unnecessarily.
• If you like doing everything alone, this ain’t it. This approach is designed to get everyone on the same page. You will have to agree on solutions and articulate your reasoning. If you don’t like doing that, you won’t like long term refactors.

Try it, let me know how it goes!

Reasons

1. An odd business requirement (share the origin story)
2. It took research (summarize with links)
3. Multiple options were considered (justify decision)
4. Question in a code review (answer in a comment)

Important caveat for number 4: if your code can be restructured in a way that answers the question without a comment, do that instead.

Before I start, there are already libraries out there that let you declare types to be checked at runtime. They offer a bunch of fancy-named classes and methods that let you construct your own types. I disagree with their approach, because it introduces a lot of cognitive overhead. They expect me to learn an extensive vocabulary only to describe simple boolean expressions. Why not just let me write those boolean expressions in the first place? This is the whole premise of my experiment: it seems easier to write a plain Ruby value check than to figure out how to build it with fancy type libraries.

A while ago, I wrote a little library called portrayal, which is a simple Struct-like object builder. It lets you declare keywords, which are just attr_accessors and a default initialize, plus some extra convenience. Using this lib as the basis, I wrote a proof of concept extension called Portrayal::Guards. In this article I show you how it works.

Leaning into boolean expressions

Let’s say we have a class Person, who has age and favorite_beer.

class Person
  extend Portrayal

  keyword :age
  keyword :favorite_beer, default: nil
  public :age=, :favorite_beer=
end

Note: Normally setters are protected, but I’m making them public above to illustrate how guards work.

Imagine that our data type requirements are as follows:

• Age must be an integer between 0 and 130
• Favorite beer must be nil or any string
• If favorite beer is not nil, then age must be >=21

Here is one simple way to do this with Portrayal::Guards.

class Person
  extend Portrayal

  keyword :age
  keyword :favorite_beer, default: nil
  public :age=, :favorite_beer=

  guard('age must be human and beer is only for >=21yo') {
    age.is_a?(Integer) && (0..130).cover?(age) &&
      (favorite_beer.nil? || (favorite_beer.is_a?(String) && age >= 21))
  }
end

This guard can be declared anywhere in the class body. It has a single boolean expression in it. If it returns anything truthy, the guard passes. If it returns false or nil, the guard fails. The string argument serves as the error message in case it fails. With this single guard we actually solved the whole problem.

Check out how this guard protects our object:

# Trying to init a person with invalid age
> Person.new(age: 200)
ArgumentError: age must be human and beer is only for >=21yo

# Making a valid person
> person = Person.new(age: 5)
=> #<Person @age=5, @favorite_beer=nil>

# Trying to assign a beer to <21yo with a setter method
> person.favorite_beer = 'corona'
ArgumentError: age must be human and beer is only for >=21yo

# Method `update` lets you apply multiple changes at once, in this case invalid
> person.update(age: 200, favorite_beer: 9)
=> {:base=>["age must be human and beer is only for >=21yo"]}

# Valid `update`
> person.update(age: 30, favorite_beer: 'corona')
=> nil

> person
=> #<Person @age=30, @favorite_beer="corona">

Three things to notice here:

1. This guard is guarding both initialize (.new), and writer methods.
2. We have a special method update, which lets you update multiple values at the same time. This helps resolve situations when you can’t assign attributes one at a time, because guards cross-check them.
3. Notice that the error we got from update is under a key :base. Keep it in mind for now, I will explain this later.

This was easy, it’s just a plain boolean expression that now completely guards our attributes. However, the expression is a little bit unwieldy, and the error message is not super useful for telling us what exactly is wrong. That’s okay. We can rewrite the guard into 3 separate guards.

guard('age must be an integer in human range') {
  age.is_a?(Integer) && (0..130).cover?(age)
}

guard('favorite_beer must be string or nil') {
  favorite_beer.nil? || favorite_beer.is_a?(String)
}

guard('favorite_beer is only allowed for age >=21') {
  favorite_beer.nil? || age >= 21
}

Much neater. Let’s try running the same code:

> Person.new(age: 200)
ArgumentError: age must be an integer in human range

> person = Person.new(age: 5)
=> #<Person @age=5, @favorite_beer=nil>

> person.favorite_beer = 'corona'
ArgumentError: favorite_beer is only allowed for age >=21

> person.update(age: 200, favorite_beer: 9)
=> {:base=>["age must be an integer in human range", "favorite_beer must be string or nil"]}

> person.update(age: 30, favorite_beer: 'corona')
=> nil

> person
=> #<Person @age=30, @favorite_beer="corona">

Nice, error messages are now more specific.

Just to recap, with guard and plain Ruby we can accomplish… everything.

But what about reuse?

Ah. Reuse is already here by default. We can have a module like this.

module ReusableTypes
  def int(name)
    guard("#{name} must be an integer") { send(name).is_a?(Integer) }
  end

  def age(name)
    int(name)
    guard("#{name} must be within 0-130") { (0..130).cover?(send(name)) }
  end

  def nullable_string(name)
    guard("#{name} must be nil or a string") {
      value = send(name)
      value.nil? || value.is_a?(String)
    }
  end
end

class Person
  extend Portrayal
  extend ReusableTypes

  keyword :age
  keyword :favorite_beer, default: nil
  public :age=, :favorite_beer=

  # Calling the guards!
  age :age
  nullable_string :favorite_beer

  guard('favorite_beer is only allowed for age >=21') {
    favorite_beer.nil? || age >= 21
  }
end

We put guards in module methods and call them. Nothing really changed, but we suddenly have reusable types.

In Ruby it’s a common tradition to return the name of what’s being declared. Portrayal’s keyword follows this tradition, returning the name of the keyword. If you’d like, you can put our type methods in front of keyword, and it works the same.

class Person
  extend Portrayal
  extend ReusableTypes

  # Calling the guards inline with keywords!
  age keyword :age
  nullable_string keyword :favorite_beer, default: nil
  public :age=, :favorite_beer=

  guard('favorite_beer is only allowed for age >=21') {
    favorite_beer.nil? || age >= 21
  }
end

If you don’t like the above style, you could do something else. For example, you could return name from methods in our module, and wrap the keyword names in them. Let’s also capitalize method names while at it:

module ReusableTypes
  def Int(name)
    guard("#{name} must be an integer") { send(name).is_a?(Integer) }
    name
  end

  def Age(name)
    Int(name)
    guard("#{name} must be within 0-130") { (0..130).cover?(send(name)) }
    name
  end

  def NullableString(name)
    guard("#{name} must be nil or a string") {
      value = send(name)
      value.nil? || value.is_a?(String)
    }
    name
  end
end

Which makes this possible:

class Person
  extend Portrayal
  extend ReusableTypes

  keyword Age(:age)
  keyword NullableString(:favorite_beer), default: nil
  public :age=, :favorite_beer=

  guard('favorite_beer is only allowed for age >=21') {
    favorite_beer.nil? || age >= 21
  }
end

When I said earlier that guards can be declared anywhere in the class body, I really meant it. This still works. I’m sure there are more ways you can come up with for using these guards. These are just a couple off the top of my head.

Looking at the above, you can probably already imagine how you’d be able to easily implement a type of any complexity, and Portrayal::Guards will make sure to guard your initializers and writers for you.

But what about composition?

Right, we actually might need some extra features to make composition nice. After toying around with some ideas, I decided to include the following additional features into the proof of concept.

Guard chaining

One way to compose guards could be to make sure that our reusable methods return the passed-in name, like we already did above. If every declaration returns the name that it received, then we could chain guards like this:

# Type methods
def Odd(name)
  guard("#{name} must be odd") { value = send(name); value.respond_to?(:odd?) && value.odd? }
  name
end

def Int(name)
  guard("#{name} must be an integer") { send(name).is_a?(Integer) }
  name
end

# Chaining example:
Odd Int keyword :odd_number

This could be especially nice for something like Nullable, where we don’t want to create NullableString, NullableInt, etc for every possible type. So maybe if we had

def Nullable(name)
  guard("#{name} can be nil") { send(name).nil? }
  name
end

Then maybe we could write Nullable Int keyword :number?

Unfortunately, we cannot. It won’t work, because Nullable will fail anything that isn’t a nil, and Int will fail anything that isn’t an integer. They don’t mesh, because we don’t have full &&/|| capabilities across guards. The good news is that perhaps we don’t actually need them.

I’ve thought about a few ways to enable this sort of composition, and came up with what I find to be a simple solution: a pass! guard.

Special pass! guard

A pass! is just like a regular guard, you can have as many as you want (but you probably never need more than one), and they always run first. If a pass! returns anything truthy, then we’re done, the object is valid, no further guards are called. With this new capability we can make Nullable like this:

def Nullable(name)
  pass!("#{name} can be nil") { send(name).nil? }
  name
end

And this kind of composition works now:

Nullable Int keyword :number, default: nil
Nullable String keyword :text, default: nil

Yay!

Because a pass! always runs first, the order doesn’t matter. If a pass! sees nil, other guards won’t run. If it sees non-nil, then we proceed into int/string guards.

Unfortunately, there’s still a problem here. All the guards are mixed together, so the Nullable check for number will stop all guards from executing, even the String guard for text. That’s because we add guards into the class, but we aren’t grouping them with each other.

To solve this, I added guard grouping. But don’t worry, it’s basically nothing.

Guard grouping

Remember that :base key in the error hash you saw earlier? Here’s a reminder:

{:base=>["age must be human and beer is only for >=21yo"]}

The :base is actually a default topic for guards. And it’s super simple to group guards into other topics. Just add one more argument to the guard:

guard(:topic_name, 'error message') { boolean expression }

The new first argument :topic_name (it could be anything really) is the topic. So all guards are actually per topic. A fail or pass! in one topic won’t stop guards in another topic. This is just a more generic way to let you make guards “per attribute”. And of course it’s just what the doctor ordered for ReusableTypes module. We can now do this:

module ReusableTypes
  def int(name)
    guard(name, "#{name} must be an integer") { send(name).is_a?(Integer) }
  end

  def age(name)
    int(name)
    guard(name, "#{name} must be between 0 and 130") { (0..130).cover?(send(name)) }
  end

  def string(name)
    guard(name, "#{name} must be string") { send(name).is_a?(String) }
  end

  def nullable(name)
    pass!(name, "#{name} can be nil") { send(name).nil? }
  end
end

By the way, notice how we’re no longer returning name from each method. That’s because each guard already returns its topic, so we don’t have to do that anymore. Another small win.

With these in place we can now declare our Person this way:

class Person
  extend Portrayal
  extend ReusableTypes

  age keyword :age
  nullable string keyword :favorite_beer, default: nil

  guard('favorite_beer is only allowed for age >=21') {
    favorite_beer.nil? || age >= 21
  }  
end

Or this way if you made methods capitalized:

Age keyword :age
Nullable String keyword :favorite_beer, default: nil

Or this way, if you like to keep keyword on the left:

keyword Age(:age)
keyword Nullable(String :favorite_beer), default: nil

Or this way, if you don’t want to interfere with keywords:

Age :age
Nullable String :favorite_beer

keyword :age
keyword :favorite_beer, default: nil

Or go back to plain guard declarations. Whatever you fancy.

Keep in mind, we only learned 2 methods so far: guard and pass! (well, maybe also update if you’re pedantic). The rest is just plain Ruby.

Listing guards

Just for fun, I wanted to be able to list guards declared on a class. It’s possible with Person.portrayal.list_guards, which returns the following:

> Person.portrayal.list_guards
=> {:age=>["age must be an integer", "age must be between 0 and 130"],
 :favorite_beer=>["favorite_beer can be nil", "favorite_beer must be string"],
 :base=>["favorite_beer is only allowed for age >=21"]}

Where is this lib?

At the time of this writing the implementation is just a gist. I’m curious what people think about this before I make it into a proper gem. Let me know your thoughts. Too crazy? Or not crazy enough? :)

I soon understood our fundamental difference in thinking. DHH approaches application code as an interconnected web of rich models. Each model is chock-full of concerns (modules with generalized functionality). Each model’s methods produce ripple effects far and wide across associated models, via numerous callbacks spanning many modules. This approach is somewhat graph-like. You have a graph of rich nodes (by “rich” I mean that they provide the highest level of business functionality), and as you activate one, it activates other nodes at various distances in all directions. These ripple “activations” could be anything, from additional database interactions, to 3rd party API calls, emails and text messages sent out, logging, and lots of other stuff.

This finally made me realize that with such a dynamic way of viewing application behavior, where all business needs are embedded at the very core, it makes sense why one might want to suppress entire classes and categories of callbacks. Those ripple effects are nearly untraceable, and require blanket suppressors from the top. Instead of directing logic, we’re constraining logic that would otherwise spread in all kinds of surprising ways.

This also explains why it’s so difficult to avoid globals in this world. You don’t want to impede your vast ripple effects with such minutia as passing the same data across associations over and over. It’s a waste of time.

I want to say that this is a highly unusual approach, but to be frank, any consistent thought-out approach is unusual in our industry. Thoughtful codebases are unusual. So yes, it’s unusual to have a well-established approach consistently applied to the entire codebase.

I know many people disagree with DHH, but I have not heard them provide a good alternative way of architecting the entire application. I’ve seen rebuttals to individual features of Rails, as well as OOP-obsessed approaches that (to be frank) were even more difficult to follow, but still no holistic explanation on how Rails apps should be written for maximum maintainability.

My biggest problem with DHH’s approach is that in his videos it was really hard for me to follow the story that his code is telling. Since all ripple effects are unapologetically triggered via chains of callbacks, it was difficult to follow so many paths into so many directions (or shall we say, indirections), ending up with so many outcomes. I find that this doesn’t work well with how I think.

My alternative way of building apps is narrative centric. Instead of creating a web of nodes with ripple effects, I want to write short stories, each living in an entry point (whether that’s a controller action, bg job, rake task, or test). Each story has a beginning (the initiating request/call) and an end (the response/output), with side effects in between. I love seeing a complete story where every major plot point is clearly visible at the controller level. If plot points are often repeated in the same order, we can bundle them under a laconically-named function. I love that I can look at every entry point, see exactly what it does, and decide how best to optimize it. I can decide to put various calls into a transaction, group multiple calls into one for a more efficient SQL interaction. I can kick something off into background processing, or introduce an exponential backoff retry for any of the steps involved. I find that with the entry point narrative-centric approach I have the most clarity and flexibility to achieve these optimizations. On the contrary, codebases where “stories” fan out widely through callback chains, don’t lend themselves well to these tweaks. You are never sure what might trigger a particular code path, and whether a particular optimization can be applied for all possible triggers.

To write beautiful short stories, I prefer to focus my attention on building myself the “domain language”. I don’t mean the literal DSL. Rather, a bunch of libraries needed for my entry points to call into, such that entry routines appear clear and concise, yet still tell the complete story. Both Ruby and Rails allow for very expressive code of this kind, as long as your business logic is neatly packed away beneath proper interfaces. This is where I prefer to spend the most time. Write clean domain-driven interfaces, then write beautiful short stories with them.

I wrote this post to explain my decision not to use certain features of Rails.

If you’re curious about this approach, check out the narrative app template for Ruby on Rails.

It’s not premature optimization when:

• They solved a problem elegantly, in a way that you didn’t think of
• They solved a problem elegantly by deviating from the beaten path
• They designed a clear and fitting code pattern that you didn’t come up with
• They used a fitting data structure that you weren’t aware of
• They used a fitting algorithm that you weren’t aware of
• They achieved extra performance without sacrificing clarity, with an approach that you didn’t think of
• They configured a piece of infrastructure for the required use case
• They let an appropriate existing system handle the work that it’s good at handling

…and they did that within the allotted time.

Respectively, it’s not premature optimization when:

• You propose a clean and elegant solution that they didn’t think of
• You challenge an existing practice with a simpler/cleaner alternative
• You introduce a new code pattern going forward, which improves codebase clarity and consistency
• You recommend a minor code change to use a more fitting data structure that they weren’t aware of
• You note that there’s a more fitting algorithm that they may not have seen
• You explain how a small change can make the code more performant without sacrificing its clarity
• You suggest an infrastructure config appropriate for the required use case
• You push for letting an appropriate existing system to handle the work that it’s good at

…and it takes an acceptable amount of time to accomplish.

Lumping good engineering with premature optimization is a sure way to discourage engineers, and push the codebase quality towards the lowest common denominator. Don’t do that.

irb> enum = Enumerator.new { |yielder|
  (1..100).each do |i|
    puts "counting item: #{i}"
    yielder << i
  end
}

irb> enum.count
counting item: 1
counting item: 2
…
counting item: 100
=> 100

However, Enumerable also has size. Except, by default it’s just nil.

irb> enum.size
=> nil

A little-known feature in ruby is that you can pass a parameter to Enumerator.new to give it a shortcut “answer” to the size question.

irb> enum = Enumerator.new(100) { |yielder|
  (1..100).each do |i|
    puts "counting item: #{i}"
    yielder << i
  end
}

irb> enum.size
=> 100

No more iterating to get the count. However, there’s an even more little-known feature. You can pass a lambda to determine the size lazily, and still faster than iterating. Let’s say that you’re enumerating over products in some kind of ecommerce API.

irb> api = EcommerceApi.new('connection config')
irb> enum = Enumerator.new { |yielder|
  api.products.each.with_index do |product, index|
    puts "fetching product: #{index}"
    yielder << product
  end
}
irb> enum.count
fetching product 0
fetching product 1
…
fetching product 235
=> 236

Let’s say our API has a more efficent way of obtaining the count: total_count endpoint.

irb> api = EcommerceApi.new('connection config')
irb> enum = Enumerator.new(api.products.total_count) { |yielder|
  api.products.each.with_index do |product, index|
    puts "fetching product: #{index}"
    yielder << product
  end
}
irb> enum.size
=> 236

We no longer have to iterate over products to get the total count, but notice a new problem: we now always run total_count, even if the user of our enum never calls size. Seems like a waste. Moreover, if the products are added to the API, our size will not change. The lambda would allow us to run the API call only when requested, and always get fresh count.

irb> api = EcommerceApi.new('connection config')
irb> enum = Enumerator.new(-> { api.products.total_count }) { |yielder|
  api.products.each.with_index do |product, index|
    puts "fetching product: #{index}"
    yielder << product
  end
}
irb> enum.size # Calls -> { api.products.total_count } lambda.
=> 236

This feature also exists when using enum_for/to_enum to create the enumerator. You have to return it from the block passed into enum_for. The block arguments are any additional arguments passed to enum_for.

irb> def each_number(max = 100)
  return enum_for(__method__, max) { |max| max } unless block_given?
  (1..max).each { |n| yield n }
end
irb> each_number(200).size
=> 200

P.S. I often forget how this Ruby feature works, and searching never brings up quick examples, so hopefully this article will help when in need of a quick reminder.

“For every complex problem there is an answer that is clear, simple, and wrong.”
— H. L. Mencken

It’s not style and shape that makes code hard to maintain. It’s the lack of clarity on how the code works, what it represents, and/or why it was written (this way). I’ll refer to these questions as “how?”, “what?”, and “why?” for short. The questions are straightforward, but there’s nothing straightforward about answering them. You may feel that short method bodies help with understanding the “how?” , but sometimes they make the program hard to follow. You may think that long, descriptive names always answer the “what?”, but often they add too much noise. You may feel that “wall-of-text” comments address any “why?” concerns, but now your readers TL;DR them. Every situation is different. It’s up to you, the programmer, to find an eloquent and considerate way to address the how?, what?, and why? in each particular case.

Maintainable code is code that eloquently and considerately communicates to its reader how, what, and why it implements.

How?

“I have only made this letter longer because I have not had the time to make it shorter.”
— Blaise Pascal

“How?” refers to the degree of expressiveness with which a routine or an algorithm is written.

The good news is that it’s hard to fail at answering “how?”. You’d have to write utter gibberish. The bad news is that it’s equally hard to succeed. You must break up complex algorithms into clear steps. You must seek out good metaphors that help people make sense of your abstract code. In other words, you must write code that continuously guides fellow engineers. That level of clear communication is rare, but so are great codebases. How often have you seen an algorithm expressed with such grace that it appears boringly obvious?

Another component in a successful answer of “how?” is the programming language itself. A flexible language allows you to write incredibly expressive codebases. However, your level of writing skill is all that stands between a magnum opus and a major oops. Make a few wrong moves, and your codebase is a total mess. This is why some engineering teams opt to commit to a strict programming language with guardrails. A codebase written in such a language won’t win you any poetry awards, but neither will it leave you with a magical ball of mud. Well, you might still end up with a ball of mud (engineers do have a boundless capacity to shoot themselves in the foot), but at least it won’t be magical.

As you can probably tell, there are practical business trade offs with both types of language. An expressive language better serves a small, experienced team, or a team with strong senior guidance. A strict language can support a larger and a less senior team. In the short term, both teams could accomplish the same amount of work. However, in the long term, a larger team will likely produce more code. That’s more code to support and maintain, which is certainly not ideal.

Failing at “how?”

When code fails at answering “how?”, it is often verbose, convoluted, or is again, simply utter gibberish. Much like the example below:

def r(s1, s2, s3)
  [s3.bytes, [32], s1.bytes, [10]*2, s2.bytes].map { |ba|
    ba.flat_map(&:chr).inject { |v, a| "#{a}#{v}" }.reverse
  }.inject(&:+)
end

In the above, we didn’t take the time to find a more considerate representation of the desired behavior.

Succeeding at “how?”

In the example below, we’ve put in the effort to make the code easy to follow. You can see that strings are being concatenated.

def r(s1, s2, s3)
  s3 + " " + s1 + "\n\n" + s2
end

Now, while the above code clearly answers “how?”, we still don’t know what business function it accomplishes.

What?

“Isolating complexity in a place where it will never be seen is almost as good as eliminating the complexity entirely.”
— John Ousterhout

If you have succeeded at what? it means that a new maintainer understands the goal of every piece of your code. In order to ensure that those goals are clear, you must figure out 1) what to abstract and encapsulate and 2) what to name it.

On one hand, the “what?” can be used to cover up the problems of “how?”. You can write awful code as long as your function is well isolated, well tested, and well named. Once the goal of your function is clear, nobody will ever need to look inside of it. Congrats, you saved some time now, and someone could simply swap out the whole thing later. Sounds like a win-win, but there is a catch. You better make damn sure you get the abstraction right, because the stakes are high. If you get it wrong, then someone will have to dig into your messy function and tease it apart. They will not enjoy that. The moral of the story is, if you have any doubts about your choice of abstraction, then definitely put some extra time towards a clean implementation.

On the other hand, it’s possible to take “what?” too far. For example, you might feel the need to blindly fixate on consistency in naming, or include the greater context in every name, length be damned. Maintainable code is not about communicating consistently or exhaustively. It’s about communicating the right amount of information at the right time (i.e. eloquently). Long names work well in high-level interfaces that mimic business terminology. However, they can be distracting in low level code, where it’s easy to lose sight of data transformations in a forest of names.

Much like a novelist’s prose, it takes years to develop good taste for eloquent and considerate naming. My advice is to get comfortable reading other people’s code. Put yourself in the shoes of your audience.

Failing at “what?”

Short names typically send a signal that they are contextually self-explanatory. Long names signal that we’re breaking away from current context, and we should pay special attention. Moreover, long names are harder to tell apart. Use them sparingly.

In the below example, the names are too long and redundant given their context.

def render_email_with_a_greeting(email_recipient_name_string_for_rendering_email, email_body_string_for_rendering_email, email_greeting_string_for_rendering_email)
  email_greeting_string_for_rendering_email + " " + email_recipient_name_string_for_rendering_email + "\n\n" + email_body_string_for_rendernig_email
end

Succeeding at “what?”

def render_email(recipient_name, body, greeting: 'Hello,')
  greeting + " " + recipient_name + "\n\n" + body
end

Here we understand what’s being done, and begin to form some valid “why?” questions.

Why?

“Give light and people will find the way.”
— Ella Baker

Some schools of thought consider all code comments to be failures of code expression. I tend to agree with this for “how?” and “what?”, but not “why?”. Trying to cram all business context into names of variables and functions is bound to make the code more confusing. The code already has more than enough to deal with answering the “how?” and “what?”. Let’s give it a break by answering “why?” in the comments.

That said, code is the most dependable source of truth, and unfortunately comments are a distant second. They tend to lie. This means that we should not overuse them. To excel at “why?”, it’s important to learn to:

1. Pinpoint which decisions actually need context.
   Usually, decisions that need to be explained derive from one of four circumstances: 1) there is a non-obvious business reason for your decision 2) you did a significant amount of research to arrive at a decision 3) you were on the fence about your chosen solution or 4) you were asked a question in a code review. In each of these situations, it is probably a good idea to leave a clarifying comment.

2. Identify the level of detail needed for your audience.
   The people who will read your code comments are likely to be experienced programmers who are familiar with your company’s internal terminology and processes. Lean on your shared knowledge to communicate efficiently.

Failing at “why?”

# Keyword argument `greeting` has a default value.
def render_email_to_send(recipient_name, body, greeting: 'Hello,')
  # Emails can be plain and html, and while most email
  # clients support html, it's a good practice to add plain
  # text versions as a fallback.
  greeting + " " + recipient_name + "\n\n" + body
end

Here we see multiple pitfalls: addressing an unlikely audience, going into arbitrary levels of detail, adding redundant information, and failing to answer likely questions. The outer comment is redundant. The inner comment neither seems relevant to the code, nor does it consider the audience it’s most likely addressing.

Succeeding at “why?”

def render_email(recipient_name, body, greeting: 'Hello,')
  greeting + " " + recipient_name + "\n\n" + body
end

When looking at the above code we can assume familiarity with the basics and identify a couple of potential questions:

• Why do we need to support a custom greeting?
• Since we use \n, is this function only used for plain text emails?
• (For rubyists out there) why are we concatenating with + instead of "#{interpolation}"?

Here’s one way to address them.

# We allow custom greetings because marketing wants to be able to
# personalize them by time of day, e.g. "Good Afternoon, Person".
def render_plain_text_email(recipient_name, body, greeting: 'Hello,')
  # We avoid interpolation because we want nil values to error out.
  # Helps prevent missing content in sent emails.
  greeting + " " + recipient_name + "\n\n" + body
end

There are now comments explaining why we allow custom greetings and avoid interpolation. We also clarified our use of \n by adding _plain_text_ into the method name.

Alternatively, we could consider eliminating the top comment by renaming greeting to personalized_greeting as follows:

def render_plain_text_email(recipient_name, body, personalized_greeting: 'Hello,')
  # We avoid interpolation because we want nil values to error out.
  # Helps prevent missing content in sent emails.
  personalized_greeting + " " + recipient_name + "\n\n" + body
end

Useful Framing

“If I had an hour to solve a problem and my life depended on the solution, I would spend the first 55 minutes determining the proper question to ask for once I know the proper question, I could solve the problem in less than five minutes.”
— Albert Einstein

When we work with fellow engineers and stakeholders, we engage in three of the most difficult kinds of communication: 1) giving feedback (in code reviews) 2) negotiating (in estimations) and 3) conveying abstract concepts (in code). These conversations can be anxiety inducing and we have them multiple times a day! The “how?”, “what?” and “why?” framework can help us organize our thoughts.

1. When conducting code reviews, you could be more specific in pointing out a problem:
   • “I see what you’re doing but have trouble understanding how it works under the hood.”
   • “I see how this works and why we need this, but extracting a method would make it easier to understand what this piece is doing.”
   • “I see what is being accomplished, and how it’s done, but I am unclear why we made this particular choice.”
2. When negotiating refactoring deadlines, you now have language that can help stakeholders understand exactly what you’re trying to achieve:
   • “It’s hard to understand how this code works under the hood. We need to do a refactor before we can confidently change it.”
   • “This code needs to be broken up so we can more easily follow what it’s doing.”
   • “A lot of our reasons why were never written down, so we’d like to try and add some context to the codebase before we forget.”
3. And finally, it provides a checklist to reflect on your own work before you share it with the team:
   • “Will a reader easily understand how my code works?”
   • “Do my names clearly convey what my code accomplishes?”
   • “Have I given the proper amount of context to convey why I wrote the code this way?”

It would be interesting to adopt a code quality standard along the lines of: “all new code must successfully convey how, what, and why, to at least 2 of your colleagues.” If you were to conduct such an experiment, I would love to know how it goes.

Code reviews are an integral part of our daily work as engineers. They help us reduce bugs, share knowledge, collaborate asynchronously, build rapport, feel recognized, and most importantly, keep software maintainable. Diligent code reviews can save the team from insidious architectural mistakes that may hinder all future development. So why do we often treat them as second-rate citizens, a distraction in the way of shipping? Why is it wrong for a good day of work to consist entirely of leaving PR feedback? Why do we try to sneak reviews past stakeholders, and outright skip them in the face of movable deadlines? There are definitely reasons for it, but whatever they are, it might help to take a deeper look at your engineering culture. I believe that in a growing (and especially geographically distributed) company, engineering success is predicated on embracing code reviews as first-class citizens, with full stakeholder buy-in.

Writing Under Pressure 📣👂

To members of the computer generation it’s no surprise that it is easy to accidentally come off dry, dismissive, judgmental, or worse in text. This is exacerbated by constantly putting out fires and missing deadlines while trying to leave feedback. On the receiving end, people most likely take pride in their work, and are attuned to listen carefully. Reading PR comments then becomes sort of like putting your ear directly onto an active megaphone. This is why I believe that rushed code reviews are harmful to engineering culture. All this pressure makes being kind and considerate a more difficult challenge than it has to be. If your company is a burning tornado, fixing that could be a crucial step towards mindful… everything, let alone code reviews.

Practices

Over the years in the industry I’ve compiled a list of my favorite code review practices. Here they are in no particular order.

1. Advocate for the Reviewee

Approach each comment from the position of respect for author’s work and decisions

Even when some of the author’s decisions appear to be “clearly suboptimal”, or straight up mistakes, assume the best intentions on their part. Spend some time advocating in your mind for the code you’re reading, challenging your own assumptions. If you understand where the author is coming from, acknowledge it before providing counterarguments.

2. Objectivity > Subjectivity

Seek out objectivity in all arguments

A comment asking for a change should make an objective case for it. When making a case, dig past personal preferences all the way down to objective underpinnings of your argument. A tiny nugget of strict objectivity is miles more effective than a 500-word opinion piece.

There’s one good kind of subjective comment: the code confuses you. Since the most important measure of maintainability is that code is clear for people on your team, “confusing” comment gets a special pass.

3. Conversation > Silence

Subjectivity is welcome as long as it’s a discussion

Sometimes we can’t help but voice a subjective opinion. In doing so, we must acknowledge that we have been unsuccessful in finding an objective argument, and are asking the author to indulge us for a moment. This is ok, as long as we present these opinions as topics for discussion, and not as something we insist on being implemented. Use the discussion as a tool to figure out the objective underpinnings behind your opinion. The team should support this exploration, and try to learn from it. Of course, be willing to accept that the discussion will not always result in your opinion making its way into the code.

4. Assume Competence

Use question form when suggesting something seemingly obvious

When you are suggesting something that appears obvious to you, it’s possible that you’re missing a problem that the author may have already discovered. If you don’t invite an explanation, the author may feel compelled to make the requested change, and work around the problem in some other way.

Instead, switch your statement into a genuine question. Let’s say you think code should be moved to another function.

• The original thought: “This code should be moved to function X due to [reasons].”
• The fake question form (don’t do this): “Could you move this code to function X due to [reasons]?”
• The genuine question form: “Have you considered moving this code to function X to avoid [reasons]?”

Notice how we are still being concise, and are still providing our solution. Except, now the author gets to choose. They can either explain why they did what they did (and maybe you’ll end up agreeing), or they can follow the request without wasting another round.

P.S. In my experience, this is the most powerful “hack” in this whole list. It’s incredibly easy to switch to a question form, not obscure any valuable info, and yet completely remove any sense of bitter judgement from your comment.

5. Care About Details

It is not a waste of time to discuss a detail in depth

Details can matter because technical debt tends to be a “death by a thousand cuts”. Besides, a discussion over a small detail can often be useful for other things, like establishing a rapport with someone. A self-conscious fear of wasting time could end up wasting more time than actually staying on topic.

6. Specific Examples > Generalizations

Try to propose a concrete solution

If possible, use pseudo-code or real code (untested is ok) to illustrate your points. If writing the code is not feasible, take time to make your comment easy to follow. This is especially important when collaborating across time zones.

7. Working Code > No Code

Always respect working code

If a fellow engineer submits a PR with a working and tested implementation, but you find that it could use a better architectural approach, this is a great problem to have. Now we can focus on refactoring this PR without worrying about implementation details, since they are already working and tested. This actually frees us to collaborate on reshaping the code’s architecture while maintaining the same logic.

8. Advocate for the Reviewer

A code review itself is an original work

When you are on the receiving end of a code review, treat the review itself as its author’s work. Even though they’re reviewing your code, their review is their original work. Instead of only focusing on the changes you’ve been asked to make, express some appreciation for comments that you found useful, or their effort to understand your code.

9. Use Complete Thoughts

Fight the instinct to leave a quick one-liner

It’s okay to use one-liners in a considerate way (i.e. as per point 4). However, if your one-liner is a short and dry change instruction, you are sending some bad signals, like:

• I don’t care whether you agree or disagree
• I don’t see you as my peer
• I don’t take code reviews (or reviewing your code) seriously
• My time (writing) is worth more than your time (unpacking what I mean)
• Your mistake was obvious to me

Practices in this article will help you avoid sending these signals.

10. Practice

These aren’t rules to be followed perfectly from day one

These practices aren’t meant to be a checklist. As long as you follow these practices in spirit, it’s ok to make your own judgment calls based on specific situations. The more you practice, the easier it gets.

11. Have fun!

Enjoy geeking out on technical discussions with your colleagues

Code reviews are places where we get to unapologetically talk deep programming, so let’s take advantage of it, and have fun!

Special thanks to the awesome Philip Szalwinski for suggestions and contributions.

TL;DR YAGNI, unless you’re working in a big company with federated front-ends or GraphQL.

It’s popular in web dev nowadays to build a backend that serves JSON, and a frontend that renders the app. This is fine. I’m not the biggest fan, but it’s really okay. Except it’s not okay if you think that your backend needs to be designed like a generic public API. This will not save you time.

Why not?

When you design a general purpose API, you have to figure out a bunch of annoying stuff.

1. How to predict and enable all possible workflows
2. How to avoid N+1 requests for awkward workflows
3. How to test functionality, performance, and security of every possible request
4. How to change the API without breaking the existing workflows
5. How to prioritize API changes between internal and community requirements
6. How to document everything so that all parties can get stuff done

And on the front-end side, there’s a bunch more:

1. How to collect all the data needed to render a page
2. How to optimize requests to multiple endpoints
3. How to avoid using API data fields in unintended ways
4. How to weigh the benefit of new features against the cost of new API requests

Do these really have to be your problems if you’re just making a backend for your frontend? Do you have to imagine every possible workflow, avoid N+1 request issues, test every request configuration, or deny yourself features when you know exactly what each page needs to look like? You can probably see where I’m going with this.

So what do you suggest?

I suggest you stop treating your frontend as some generic API client, and start treating it as a half of your app.

Imagine if you could just send it the whole “page” worth of JSON. Make an endpoint for /page/a and render the whole JSON for /page/a there. Do this for every page. Don’t force your front-end developers to send a bunch of individual requests to render a complex page. Stop annoying them with contrived limitations. Align yourselves. 🧘‍♂️

And in that JSON, actually render the page. Don’t render abstract models and collections. Render concrete boxes, sections, paragraphs, lists. Render the visual page structure.

{
  "section1": {
    "topBoxTitle": "Foo",
    "leftBoxTitle": "Bar",
    "linkToClose": "https://…"
  },
  "section2": {
    …
  }
}

This is similar but not quite the same as Server Driven UI¹. Perhaps we could call it Server Informed UI.

How is that better exactly?

Have you seen that list of annoying decisions up there? For one, they are gone now. 💨

For two, you are now free to decide “I want page a” and then implement “page a” in the backend, and in the frontend. Super straightforward. ✅

No more “what API workflows do we need to introduce to sort of make this page possible almost? 🤔”. You can keep “page a” dumb to only do what it needs to do. You test the crap out of “page a” for bugs, security, performance. You can even fetch everything for “page a” in a single big SQL query. You can cache the entire JSON payload of “page a”.

Frontend knows exactly what each field in “page a” payload is for. There are no discrepancies in field meanings. They represent exactly what frontend needs.

When a stakeholder tells you to change “page a” you will be able to literally go ahead and change “page a”, instead of spending meetings figuring out how your backend API could accommodate the change in “page a”. It’s not a choreographed conglomeration of API requests. It’s just “page a”. You have freed yourself from self-imposed limitations of your API.

Your business logic has now moved from being haphazardly split between frontend and backend into just backend. Your frontend can finally focus on presentation and UI. Your backend can finally focus on implementing exactly what’s needed. Kinda the goal, no?

Have you actually tried this?

Yes, I have tried this on a couple of production projects so far. One of them was personal, the other was a consistent multi-year refactoring effort in an existing company. The whole team was bought in, and it worked out well. The only problem we’ve encountered was that the front-end team has gotten increasingly bored. Nearly all business logic was taken away from them. At the same time, no “excitement” was added to the back-end team. It’s just gotten kinda boring all around. Somehow we all ended up talking more about the business than the code.

Feel free to stop reading here if you’re convinced. Next part is just responding to various rebuttals I keep hearing.

But I want my front-end team to have freedom! (Or, I want my front-end to be decoupled!)

Let’s be honest, your frontend doesn’t really have freedom. When they send you 7 requests to render a single page, that’s not freedom. It’s jumping hoops to meet basic requirements. As soon as requirements change, you probably going to need to change the backend anyway to accommodate it. The freedom is all accidental and mostly in the wrong places.

If you really want to give your front end team freedom, install them a GraphQL wrapper directly on top of Postgres and quit. 😛

But we actually want a general purpose API anyway, so this is 2 birds with 1 stone, no?

No, you would not actually want to make this API public. You think you would, but when time comes, you’d be like “crap, maybe I shouldn’t”. These 2 APIs have very different reasons to change. Public API needs to enable the workflows of your clients. Private backend needs to enable the next whim of your product manager. Stop jamming sticks into your own bicycle wheels.

But how will I reuse the logic when building JSON for pages? I reused so much logic in my CRUD controllers!

If your programming language lets you reuse logic (it does), then you can reuse logic. Use mixins, composition, inheritance, whatever you got to work with. If you make yourself some good abstractions, then you will have an amazing time putting together pages from your LEGO blocks.

But we can reuse this API for the mobile app too!

Your mobile app has a different set of pages with different info, structures, and reasons to change. You’ll save more time and sanity making another backend specifically for it. But hey, you can reuse a lot of your logic (see the previous paragraph).

But what if a page needs a partial XHR update? Am I supposed to always return an entire page?

No, it’s okay to make an endpoint that returns just something specific. You have my permission. Make endpoints for snippets of data for specific page sections or whatever. It’s okay. Render your React components from initial payload, then update them from XHR calls to these endpoints. But only introduce these endpoints when you need them on certain pages. These are exceptions, not the default.

But my frontend is a SPA, so it almost always needs data snippets, not entire pages

Those data snippets could still be provided as partial page structures, not generic resources. As long as your backend only serves the exact needs of your frontend, you’re good. 😇

But I’m building a site builder, so my frontend is dogfooding the site builder API

🗡 I dub thee a legitimate use case haver, congratulations!

Do you have data to support your claims?

I wish. It’s pretty hard to measure these kinds of things in our industry. Who’s gonna maintain 2 architectures for the same software for 3 years, and compare productivity between them? All I got is a mixed bag of personal experiences. Feels inductively justifiable. 🤷‍♂️

Update 2025-12-11: There is now a follow up article (4 years later).

Problem

Let’s take a look at this example. We query the database to find the user by id, then use their email to make an API call to download a profile and grab the name. While this example is indeed contrived, it’s fairly common to see variations on this theme in the wild.

def name
  @name ||= @api.fetch_profile(User.find(@id).email).name
end

Now, just to get it out of the way, there are various problems with this code. However, in this post, let’s just view it from the angle of memoization. So, what are the 3 reasons not to memoize like this?

Reason 1: Caller is misled about the real impact of making this call.

Typically, doing this sort of memoization goes hand-in-hand with naming your method with a noun. Since the method is named so inconspicuously (name), we’re signalling that a caller doesn’t have to worry what happens under the hood. We perpetuate the practice of calling this method mindlessly, with no regard for the fragile sequence of interdependent network operations that it takes to fulfill the request. I get it, we want to encapsulate the plumbing, but couldn’t we do it without misleading the caller?

Reason 2: Caller has no say in cache invalidation.

This memoization style assumes that caller will never want another fresh value. For web apps, it probably comes out of another assumption that we’re always living within a web request, and we never want to fetch any data twice. Unfortunately, each such memoization slowly eats away at our understanding of how data flows through our application, making it much harder to debug problems, or implement anything else on top of the same codebase.

Reason 3: Caller has no way of stopping redundant work.

In our example, if a caller already has a user available, the method will fetch it again anyway. In a well architected system we should be able to inject that dependency, especially if it took something as error-prone as network or database roundtrips to obtain it.

Solution

How would we avoid all 3 of the above problems? It’s not that difficult, but with a caveat that you didn’t already overcommit to bigger architectural mistakes. Still, it’s never too late to stop making things worse. So without further ado, here’s the code free of all of the above problems.

def retrieve_name email: User.find(@id).email, api: @api
  api.fetch_profile(email).name
end

You might’ve just done a double-take: wait, how is this the solution? We just removed caching and added some useless arguments. Bear with me, let’s talk through this real quick.

Note that arguments are optional, so the method can still be called without passing anything. Let’s go back and see if we’ve addressed the problems with the original code.

1. Is caller still misled about the real impact of calling this?

No. The fact that this method name is now a verb retrieve_name makes it clear that when you call it, it will do things. That’s all it takes to send the correct signal.

2. Can the caller control cache invalidation?

Yes.

name = retrieve_name

# Name is now cached, feel free to reuse it.
do_something_with(name)
do_something_else_with(name)

# Get a fresh name whenever you want.
fresh_name = retrieve_name

3. Can the caller stop redundant work from happening?

Totally.

my_user = User.find(123)
name = retrieve_name(email: my_user.email) # Saves a database call.

In case it’s not obvious, we couldn’t accept arguments the same way in the original version, because we’re only caching one value, and even if we then passed a different user, we would still get back the first cached value.

Ultimately, with very little effort, we just gained 3 significant advantages in maintainability, reusability, and performance of our code.

FAQ

What if I need to call this method from different places, so I don’t have a variable to reuse?

I feel your pain. Unfortunately, if you must depend on this caching technique because you cannot assign a variable once, and pass it around, I have some bad news for you. Your abstractions need rethinking. There should be a top level routine in your code that tells the story of a particular transaction. Values that are reused need to be floated up into that context and passed into whatever needs them. In a vanilla Rails world the place like this would be your controller actions. If doing this makes your actions too long, you’re missing intermediary objects that give you a clean abstraction to write your routine. That said, this is a pretty big topic best left for future blog posts.

1. Mounted volumes are slow and error-prone
2. You need hacks and shortcuts to run any console/debug commands in containers
3. Live updates on code changes are unreliable in docker
4. Runtime is slower in docker
5. Dependency updates are slower in docker
6. Networking is more complicated with docker

What genuinely surprises me, is that often teams don’t consider the obvious: just run your app directly on your machine. I find it to be the sweet spot of dev setup. Let’s see what it would look like.

1. Use docker-compose for databases and external services

Create a docker-compose.yml file in your app’s root and only declare your databases in it. For example, this file gives you

• Postgres on localhost:5432
• Redis on localhost:6379
• Fake S3 on localhost:9000

version: '3'

services:
  postgres:
    image: postgres:10.3-alpine
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
  redis:
    image: redis:3.2.11-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
  minio:
    image: minio/minio
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
    entrypoint: sh
    command: -c "mkdir -p /data/dev /data/test && /usr/bin/minio server /data"
    environment:
      MINIO_ACCESS_KEY: access_key
      MINIO_SECRET_KEY: secret_key

volumes:
  postgres-data:
  redis-data:
  minio-data:

2. Use asdf for language runtimes

Create a .tool-versions file in app’s root. Here’s an example for elixir and node setup.

elixir 1.6.4-otp-20
erlang 20.2.4
nodejs 10.8.0

Asdf is like rvm, nvm, and other version managers combined. It has an extensive list of things it can manage.

3. Setup everything

Now you can bootstrap the application by running

asdf install
docker-compose up

and in another terminal you run the app itself:

mix phx.server

That’s it. Now you have the benefit of quick and simple dev setup without giving up all the convenience of interacting with your app directly, without containers in the middle.

Bonus: How to make local Rails work with dockerized Postgres?

The cool part is that your database.yml can be committed to the repo, it will always look the same:

default: &default
  adapter: postgresql
  username: postgres
  host: localhost

development:
  <<: *default
  database: myapp_dev

test:
  <<: *default
  database: myapp_test

production:
  <<: *default
  database: myapp

However, there’s a minor issue when using this setup in Rails. You might get an error when trying to install the pg gem or run a rake db:structure:dump command. Both of these actions rely on postgres being installed locally. To work around it simply add postgres to your .tool-versions — asdf supports it. You will not be actually running this postgres, only using its cli as a client, and satisfying pg’s dependencies.

Gem elasticsearch-transport

Provides a bare-bones HTTP client that doesn’t have any Elasticsearch-specific api methods, but knows how to discover and connect to multiple servers, rotate connections, and log things.

• readme: elastic/elasticsearch-ruby/elasticsearch-transport

Gem elasticsearch-api

Provides a module that adds elasticsearch-specific methods such as search, cluster, index to a generic HTTP client. Can be included in any class that implements method perform_request which returns an object responding to status, body, headers.

• readme: elastic/elasticsearch-ruby/elasticsearch-api

Gem elasticsearch

Depends on:

• elasticsearch-transport
• elasticsearch-api

All it does is it takes an HTTP client from elasticsearch-transport and includes the Elasticsearch::API module into it from elasticsearch-api, providing a more convenient client as a result.

• readme: elastic/elasticsearch-ruby/elasticsearch

Gem elasticsearch-dsl

Provides a tire-like syntax for defining queries. The resulting query object is useless on its own, but it supports to_hash, and therefore can easily be fed into any HTTP client by encoding Hash as JSON. If you feed this object to the client from elasticsearch-transport, it will be automatically dumped as JSON using the default MultiJson serializer.

• readme: elastic/elasticsearch-ruby/elasticsearch-dsl

Gem elasticsearch-watcher

Depends on:

• elasticsearch-api

Extends Elasticsearch::API with an extra method watcher, which in turn provides methods specific for the Watcher plugin, such as put_watch, get_watch, and others.

• readme: elastic/elasticsearch-ruby/elasticsearch-watcher

Gem elasticsearch-extensions

Depends on:

• elasticsearch

Adds contributor-friendly features like terminal colorizers and formatters for Elasticsearch responses, cluster start/stop for testing, and profiling features for testing.

• readme: elastic/elasticsearch-ruby/elasticsearch-extensions

Rails-specific

Gem elasticsearch-model

Depends on:

• elasticsearch

This gem contains various modules to be included into models. It does nothing without explicit includes.

• readme: elastic/elasticsearch-rails/elasticsearch-model

Module Elasticsearch::Model::Proxy

This module is useless on its own. It adds __elasticsearch__ method to a model at class and instance levels, which is supposed to isolate all elasticsearch functionality underneath it. However, the proxy object is actually empty, it has no methods, and it expects that all other modules will be manually included into it.

• module source/docs: elasticsearch-rails/elasticsearch-model/lib/elasticsearch/model/proxy.rb

Module Elasticsearch::Model::Client

Adds accessor client to the model at both class and instance level. Default client comes from elasticsearch-transport.

• module source/docs: elasticsearch-rails/elasticsearch-model/lib/elasticsearch/model/client.rb

Module Elasticsearch::Model::Naming

Adds accessors index_name and document_type to the model at both class and instance level. Defaults are inferred from the model name.

• module source/docs: elasticsearch-rails/elasticsearch-model/lib/elasticsearch/model/naming.rb

Module Elasticsearch::Model::Indexing

Adds class methods settings, mapping, create_index!, index_exists?, delete_index!, and refresh_index! which can be used to define and manage field mappings. Index methods would implicitly use the previously defined mapping, as well as implicitly inferred index/document_type names. The module also adds instance methods index_document, update_document, and delete_document that depend on model having as_indexed_json, and id to work.

• module source/docs: elasticsearch-rails/elasticsearch-model/lib/elasticsearch/model/indexing.rb

Module Elasticsearch::Model::Searching

Adds class level search method that accepts a to_hash-compatible object, delegates to the search method on the client. By default the client is coming from elasticsearch-transport.

• module source/docs: elasticsearch-rails/elasticsearch-model/lib/elasticsearch/model/searching.rb

Module Elasticsearch::Model::Serializing

Adds an instance method as_indexed_json to a model, which by default delegates to as_json with option root: false.

module source/docs: elasticsearch-rails/elasticsearch-model/lib/elasticsearch/model/serializing.rb

Module Elasticsearch::Model::Importing

Provides class-level method import allowing batches of records to be efficiently imported into Elasticsearch. This module automatically adapts for ActiveRecord and Mongoid.

module source/docs: elasticsearch-rails/elasticsearch-model/lib/elasticsearch/model/importing.rb

Module Elasticsearch::Model

When included, this module does 3 things.

1. Includes Elasticsearch::Model::Proxy into the model.
   
2. Includes the following modules into the __elasticsearch__ proxy object.

• Elasticsearch::Model::Client
• Elasticsearch::Model::Naming
• Elasticsearch::Model::Indexing
• Elasticsearch::Model::Searching
• Elasticsearch::Model::Serializing
• Elasticsearch::Model::Importing

3. Delegates some important methods from model class/instance to the __elasticsearch__ proxy object, namely search, mapping, settings, index_name, document_type, import.

module source/docs: elasticsearch-rails/elasticsearch-model/lib/elasticsearch/model.rb

Module Elasticsearch::Model::Callbacks

Adds callbacks that sync model and Elasticsearch representation on create/update/delete. The callbacks are blocking, if the syncing must be asynchronous it’s suggested to implement your own callbacks, and not use this module. This module automatically adapts for ActiveRecord and Mongoid.

module source/docs: elasticsearch-rails/elasticsearch-model/lib/elasticsearch/model/callbacks.rb

Gem elasticsearch-persistence

Depends on:

• elasticsearch
• elasticsearch-model

Provides a way to build models backed by Elasticsearch database, similar to ActiveRecord models being backed by SQL database. Additionally, provides a way of using Repository pattern to the same effect.

• readme: elastic/elasticsearch-rails/elasticsearch-persistence

Gem elasticsearch-rails

Provides rake tasks for importing data from Rails models into Elasticsearch, as well as instrumentation for displaying search requests and their stats in logs. Includes special support for Lograge. Both rake tasks and instrumentation features must be manually required to function (no railtie support). Comes with multiple Rails application templates which allow the user to generate example applications locally, starting from a very simple integration, to a full-blown Elasticsearc-powered application, to demonstrate the gem capabilities and common usage patterns.

• readme: elastic/elasticsearch-rails/elasticsearch-rails

• deterministic order of operations across hosts
• centralized configuration (no immediate need for the likes of etcd/consul)
• agent forwarding
• better control over host resources (no unnecessary periodic runs)

In essence, you have an entity that can see and orchestrate all the pieces in the system rather than having each piece trying to maintain itself by catching up to its surroundings.

Given the above points, this article is about running Ansible from your local machine. It assumes that the target hosts are only accessible via ssh, and helps setup Vagrant in the same way, as if it was a VPS.

Nevertheless, during my venture into Ansible I immediately ran into some sticking points, which I knew had to have elegant solutions, yet they were hard to search for online, or easy to miss in the docs. Naturally, they ate away my time and now I’d like to help you save yours.

1. Build a convenient local playground

Just a few servers that can talk to each other is all you want. Your multiple production machines, their interactions, their firewalls and dns config should all just be reproduced on a smaller scale. Is that really so hard? If your hosting provider is kind of like Digital Ocean it’s especially useful to get it all thoroughly mimicked, since you don’t get any security groups or virtual private clouds there, so all your ipconfig and dns stuff has to be configured by hand.

Well, turns out it’s easy, after some screwing around.

Path to failure

You start hooking up ansible provisioner in Vagrant. Don’t. It’s not even a good approximation of how you will run Ansible in production.

Path to success

There are 4 quick steps to having a very convenient setup.

1. Make it easy to sync your hosts file with your VMs
2. Automate adding your pub key to VMs
3. Configure your ssh client
4. Write your Vagrantfile

1. Make it easy to sync your hosts file with your VMs

This assumes you have vagrant installed. A very convenient vagrant plugin can automatically add and remove hosts every time you add or destroy VMs. Install as follows.

$ vagrant plugin install vagrant-hostsupdater

Now every time you boot or destroy a VM your /etc/hosts will have the hostname added/removed automatically. You will notice it asking you for your sudo password every time it tries to do that.

2. Automate adding your pub key to VMs

I wrote a small ruby script for Vagrant which lets you conveniently put your pub key into VM akin to how Digital Ocean would bootstrap your machine with your key. Assuming you’ve made a root dir for your Ansible project (I called mine stack), do this while in it.

$ mkdir vagrant
$ cd vagrant
$ curl -O https://gist.githubusercontent.com/maxim/dafc3b6da5754419babb/raw/7789793ed7e799dc22e6222c30c6130f34a055e7/key_authorization.rb
$ cd ..

Now you have a vagrant/key_authorization.rb file in there, I’ll show you how to use it in just a bit.

3. Configure your ssh client

SECURITY NOTICE: Absolutely do not do this for your production servers. This is only safe on a private vagrant network with your own VMs.

We will setup our machines on certain IP range, and I’d like them to be accessible just like Digital Ocean machines, directly as root. So this ~/.ssh/config makes it much more convenient.

# For vagrant virtual machines
Host 192.168.33.* *.myapp.dev
  StrictHostKeyChecking no
  UserKnownHostsFile=/dev/null
  User root
  LogLevel ERROR

With this one config you murdered a whole bunch of birds. Specifically,

• SSH won’t complain about non-matching keys for your ever-changing vagrant VMs
• SSH won’t try to remember and manage those keys via known_hosts
• You won’t have to specify root@… every time
• SSH will shut up about how you’re making it do such awful things

Just make sure you replace myapp with whatever local hostname you’d like for your app, and ip address with your desired vagrant ip range.

4. Write your Vagrantfile

Now that you have everything else in place, let’s add the Vagrantfile into your ansible dir.

require_relative './vagrant/key_authorization'

Vagrant.configure('2') do |config|
  config.vm.box = 'ubuntu/trusty64'
  authorize_key_for_root config, '~/.ssh/id_dsa.pub', '~/.ssh/id_rsa.pub'

  {
    'db1'    => '192.168.33.10',
    'app1'   => '192.168.33.11',
    'redis1' => '192.168.33.12',
  }.each do |short_name, ip|
    config.vm.define short_name do |host|
      host.vm.network 'private_network', ip: ip
      host.vm.hostname = "#{short_name}.myapp.dev"
    end
  end
end

This makes it super easy to add more machines into the ruby hash, specify their exact ips, and bring your whole stack up and down with vagrant up and vagrant suspend.

Also notice the require line on top, and the authorize_key_for_root command. This is a reference to my script you downloaded earlier. With this in place the first key it finds among the ones listed will go into the VM as one of root user’s authorized_keys. This way you can ssh as root without a password.

Also thanks to our ssh config, you now get to run the following, and it’ll just work.

$ vagrant up db1
$ ssh db1.myapp.dev
root@db1:~#

This might make you wonder, why not simply let Ansible setup a non-root user for you, and do everything via sudo? Based on my conversations with friendly neighborhood sysadmins, passwordless sudo gives you no more security than bootstrapping via root does. All it does is add an extra useless step to every operation. As far as using Ansible as Vagrant provisioner: as I mentioned in the intro, my goal is a very production-like environment. Vagrant shouldn’t play any role in it except leave me with a few blank machines similar to the ones my VPS provider would build for me. In essence, I want my starting point on Vagrant to be almost exactly like if I used an actual live VPS, and I like to keep it simple by making a good use of the default config. In my case it means a machine with a hostname, IP, and a root user with my key authorized. That’s exactly what we’re doing here.

2. Teach Ansible to talk to Github on your behalf

In an effort to keep things simple, I avoid having to create extra ssh keys on my servers and add them to Github. Instead there is a way to let servers access Github on your behalf without creating any extra identities. Ansible would take the identity of the user who initiated the playbook run, and forward it to the host, which in its turn will use it to talk to Github.

This mechanism is called agent forwarding. You might not want this if you have a complex deploy pipeline, where a deploy server acts autonomously and has its own identity, but Ansible makes it so easy to orchestrate various processes, that I decided not to build one for my setup.

So there is a setting for this. Create a file right here in the root dir called ansible.cfg with the following contents, and it will be automatically picked up when you run Ansible.

[ssh_connection]
ssh_args = -o ForwardAgent=yes

That’s it. No need to add new keys to github.

3. Add Github to known_hosts properly and securely

For those who are not sure what this is: a server like github can give you a key which your ssh client will use to ensure that you have a secure ssh connection. That key is easily obtained by using the following command.

ssh-keyscan -t rsa github.com

Path to failure

People out there suggest that you should run that command on your remote hosts in your Ansible playbooks to set the key dynamically. Don’t. That defeats the purpose of having the key. A man-in-the-middle attack could compromise the result you get, leaving you in the exact situation this measure was meant to prevent.

Path to success

Use Ansible feature called lookup. Here’s an example Ansible task that will set the key in a secure way.

- name: ensure github.com is a known host
  lineinfile:
    dest: /root/.ssh/known_hosts
    create: yes
    state: present
    line: ""
    regexp: "^github\\.com"

{:.notice} Careful: If you do this while having a large number of target servers, you’re gonna have a bad time. This might cause some serious bombardment of your control machine. In that case use accept_hostkeys=yes in your git task. I only have about 10-20 machines, so this isn’t a problem for me. (from the comment by mpdehaan)

You might wonder how is that different than the fail path above? First of all, this doesn’t run on a remote host, it runs on your control machine. Second of all, it only sets this key once per host. If github decides to change it you would have to write another play to update it, or modify this one. This is good because we don’t want a MITM attack to trigger a change of the real key.

Another advice out there is to actually hardcode this key in a variable. That’s also a good way to do it, but I don’t like having ugly strings pollute my var files.

4. Keep your secret vars separate

I’m personally not a fan of shit-work involved in placing variables in many different files. It’s more convenient to see the whole picture in one place. However, I do believe secret variables should be either git-ignored or encrypted, and for that you need to put them into their own file.

In my setup I use group_vars/all to keep all non-secret things. So now that this file is taken, how can you share secrets among all your hosts?

Path to failure

I spent a long time trying to figure this one out. I was recommended things like using lookups to fetch each individual variable from their own files elsewhere on my machine. I was also recommended to place these variables into vars file for each individual host, repeatedly. Both are fail. When I discovered the way, I admit I was kind of kicking myself.

Path to success

I simply didn’t know one little fact. Your group_vars/all can be a directory. All files in there can contain variables for all hosts. So I created 2 files in there, config and secrets. I also added group_vars/all/secrets to .gitignore and solved all my issues. Another approach would be to encrypt that file with ansible-vault and let it stay in your repo. I didn’t need that.

5. Avoid perpetually “changed” and “skipping” tasks

As a slightly obsessive-compulsive person, I didn’t like the fact that some tasks kept showing me “changed” or “skipping” status. Besides the fact that it feels wrong, various notification tools might end up bothering you about things changing while they actually aren’t. One such offender was the way to create a postgres extensions in your database.

Path to failure

This is the way a typical postgres create extension task looks.

- name: ensure postgresql hstore extension is created
  sudo: yes
  sudo_user: postgres
  shell: "psql my_database -c 'CREATE EXTENSION IF NOT EXISTS hstore;'"

Every time you run it, it will be detected as “changed” even though nothing actually changes.

Path to success

Instead we can leverage Ansible’s register, changed_when and failed_when to make this task report ok, as it should. Take a look at this version.

- name: ensure postgresql hstore extension is created
  sudo: yes
  sudo_user: postgres
  shell: "psql my_database -c 'CREATE EXTENSION hstore;'"
  register: psql_result
  failed_when: >
    psql_result.rc != 0 and ("already exists" not in psql_result.stderr)
  changed_when: "psql_result.rc == 0"

This clever trick takes advantage of psql exit codes and stderr output. Notice also that we removed IF NOT EXISTS part from the SQL to make sure we get an error if extension is already there. This is done on purpose, because we only consider the task failed if the exit code is not zero and the error is something other than “already exists”. If the error is actually “already exists”, then it’s not really a failure, it’s exactly what we want. The changed_when piece indicates that if there is no error and we exited successfully, then it means psql actually added the extension, and therefore changed. All neat now.

It’s worth noting that while it hurts an obsessive person like me, sometimes it’s hard to achieve an ok report on some tasks. For example, if you use the shell module with creates option, it might generate skipping instead of ok, and you should let it go. Instead focus on getting rid of changed reports, and leave skipping alone.

6. Separate your setup and deploy playbooks

Every time you use a package module in Ansible (like apt or npm) you have a choice between state=present and state=latest. The former will simply ensure that a desired package is installed, while the latter will, in addition to that, go ahead and update it if it’s not of the latest available version. When you are building your stack, my advice is to always prefer present. This also means that when using VCS modules like git set update: no. This is important because you need to be able to converge your server configuration without actually deploying and changing your software. A software update, whether it’s your app’s deploy, or a dependency version bump, has nothing to do with your server configuration, and could really break your production. Your updates have to be strict, purposeful, and well thought out, which is why I suggest to write separate playbooks for them. In those playbooks it would be acceptable to use state=latest, since you’d only run them when you’re ready to deal with the consequences. Chances are you would need to choreograph some data and configuration to get all the updated pieces working anyway, so having a different “convergence vector” for it is a much simpler approach.

Well, time to grab some coffee and dive back into building an awesome stack.

• Note: only file/dir owner can chmod it
• Note: scripts need both x and r permissions to execute (that’s because scripts are read into interpreter)
  (only r is enough if ran via ruby script.rb, sh script.sh)

files

Warning: setuid is dangerous

directories

• Note: having xw on a dir is enough to delete any file in it (unless it has sticky bit)

sticky on dirs

• only used when writable by group/everyone
• files in dir can only be edited/deleted by their owner (think /tmp)
• symlinks only work if target is within this dir

setgid on dirs

• all files/subdirs created by anyone in this dir inherit its group
• all subdirs inherit this bit when created

setuid on dirs

• no effect

sources

• https://en.wikipedia.org/wiki/Chmod
• https://wpollock.com/AUnix1/FilePermissions.htm
• https://major.io/2007/02/13/chmod-and-the-mysterious-first-octet/

The worst part? This was a minimal viable product for a newly-born startup.

Accidentally CMS

“Complexity” by nerovivo

The programmer’s nature encourages us to indulge ourselves in solving puzzles and modelling abstract concepts. It’s the passion that makes us lose sight of the danger looming ahead, the trap we’re edging towards thanks to our subjective assumptions and vague speculation, the trap of building a overdesigned and overcomplicated system for its own sake. A CMS trap. We suffer various consequences, ranging from burnouts, and loss of enthusiasm, to missed deadlines, and failed businesses, yet we never seem to speak of this mistake directly. Somewhere by a water cooler, an experienced colleague casually points out that you might be overcomplicating things. Somewhere in an IRC chat you get ridiculed for asking questions about a complex object model for a project that will most likely never see the light of day. Yet nobody can clearly explain exactly what is the underlying thought process. These casual remarks is all the education we get on the subject, and people end up learning this the hard way. That’s why I’d like to shine some light on this phenomenon. To start, here is my best shot at defining the CMS trap the way I see it.

A CMS Trap is a state of a web-application in which the development of content management systems is obstructing the development of the content.

If you are building a startup like me, you should know that this trap is especially dangerous in the early stage. Only a small percentage of companies get to play the long game, and by that time their problems have shifted onto a entirely different plane of existence. While these companies may also be subject to falling into the CMS trap, they would probably be able to afford it, if not even pursue this direction intentionally. Here, I’d like to focus on the much more abundant variety: the small companies. The problem would become apparent as soon as you’ve opened your doors to an influx of customers, who’d begin using your project, and providing you with real analytics and feedback. At this point your project would no longer be driven by your gut feeling, rather you’d have real data suggesting how to proceed, dictating which features to implement next. This would be the time when all of your initial architectural assumptions are being tested, and reality is beginning to set in. Reality has no tact, it doesn’t spare you any painful truths when dawning upon your hopeful application design. You’d wish you could refactor, but it’d be too late, as you’d be forced to keep up with new features instead, and implementing them would only be getting harder in this downwards spiral of dwindling productivity.

I’ll thank me later

“Most of our assumptions have outlived their uselessness.”
— Marshall McLuhan

Simply put, we love designing systems. As soon as we form some understanding of a problem, we rush to our /(?:whiteboards|moleskines|mindmaps|editors)/ and start passionately defining entities and their interactions. It feels good, it’s what we do best. We tackle some of the most fundamental decisions about the project. Then, having carefully outlined our assumptions, we commit to them. We like to think that we sow wisdom and flexibility with our early decisions, and we will thank ourselves later. With all of those useful points of extension and well-represented entities, what could possibly go wrong? The reality is, that most likely these early assumptions will restrict our future, not expand it. The day comes when we meet our old friend, the innocent “past self” staring back at us from the editor, smiling proudly. This well-meaning person spent hours, days, and weeks diverting our efforts into the abyss of speculative architecture, while having barely any idea about the real problems we’ll be facing. We are now stuck with all of that “helpful” code. It’s as if you decided to cook some salad, but instead of having separate ingredients laid out in front of you, all you have is another fully cooked salad given to you by a stranger, which you are now forced to dig through in hopes of fetching some of the pieces you need.

In programming, your past self is nothing but a stranger with boundary issues.

In the same spirit, imagine you come back to your computer only to find your app reorganized by some clueless stranger in ways that have little to do with reality. This isn’t very different to how we find ourselves looking at a system we’ve over-modelled in the past. Wouldn’t you wish that you didn’t have to deal with any of this garbage, and could instead simply greenfield your way ahead as dictated by your business needs?

To bring this back to my personal story, I eventually realized that with every new business feature I spent more time figuring out how to fit it into the existing framework I imposed on myself than actually designing the feature. As you may have guessed, I was thanking myself profusely for being so considerate.

C < RUD

“It’s harder to read code than to write it.”
— Joel Spolsky

Talking about architecture is a lot like talking about code itself. It’s not exactly that we can never be mindful of the future, it’s just that the odds are not in our favor. Code is easy to write and hard to change or remove. Every line we light-heartedly throw into the mix will eventually be taunting us with the timeless question: “guess what’s going to break if you touch me? ”. Architectural decisions, just like code, are easy to make and very hard to unmake. While in code this problem is alleviated with testing, in architecture we don’t have testing. The only measure of quality we have is really the measure of pain we feel when working on a new feature, and by that time it’s often too late. Bad architecture can suffocate your business even while your code is sporting 100% test coverage.

Alarm triggers

“Fire Alarm” by Fey Ilyas

As with most traps, there is no specific way of knowing when you are walking into one. The best you can hope for is to have some sort of “tells” that warn you of an upcoming danger. Below I list some of these tells from my own experience. Seeing these things in your early stage project should at least make you suspicious.

The early onset conservatism

“A state without the means of some change is without the means of its conservation.”
— Edmund Burke

Say you are faced with a new feature, and you find it to be a real yak shave. You realize that it will take a huge refactor, and you are arguing for ways to just avoid it. There is a fine line between negotiating feature requirements for reasons of efficiency, and negotiating them because you are stuck in the accidental buildup of legacy architecture. Could it be that your early speculative design decisions are starting to get in the way of today’s real business needs? Have you perhaps built too much too soon, and is it only a matter of time before the trap snaps, leaving your project effectively paralyzed? Don’t get me wrong, more often than not being conservative is a healthy defense against unnecessary complexity, it’s a standard practice of an experienced developer. The problem is when there is too much defense too early in the project’s life. It should definitely cause some suspicion.

The Drupal syndrome

“Hm, our pricing rules are different for various products, so we’ll need to find a way for an admin to define these rules in the admin panel. Maybe we should store code in the database and eval it?”

This is a classic sign of walking into the CMS trap. You are trying to come up with ways to let admin program some logic, which should then be saved to the database. If it wasn’t for the admin interface, it could’ve been done with only a few lines of code. However, now we are talking about creating price models associated with rules, and all the complexity emerging from this. Any further extensions to pricing capabilities, which could’ve been implemented with a line or two of code, would now have to take the shape of database migrations, forms, validations, and everything else down this rabbit hole. Do you really need admin UI for pricing rules at this point?

The seed is weak

How do you implement 10 categories to place your products into? Typical answer involves creating a Category model and then writing a script that will seed the 10 prescribed categories, which will be assigned to products. Then you’d make sure every developer runs this seed file. Also, don’t forget about running it in production of course. On every deploy. And on every pull. And when setting up a new machine. And when running tests. And naturally, if you change something in the seed file.

If your early-stage application relies on a lot of seed data, you are on a slippery slope. Things that can be assumed constant mustn’t need to be modelled as database-backed entities at this point yet, but I’ll get back to this later.

Every road leads to Mordor

“One does not simply implement business logic”
— Boromir

This is somewhat similar to the early onset conservatism, yet there is a difference. Ever found yourself intimidated by a trivial task? Ask yourself this: would this task be intimidating if it was to be implemented in isolation, without the rest of the app surrounding it? If the answer is yes, look at your feet, because you might be caught in the trap. Implementing a feature in a well architected system shouldn’t be any more difficult than implementing it in isolation.

The phantom pain

Sometimes a CMS trap can be recognized by the presence of phantom pains that stem from hidden implications of an emerging CMS. For example, in reality you would never need to delete your categories, but because you built them as admin-editable database-backed records, suddenly you are thinking about the non-existent scenario of having them deleted. Your architecture took the liberty of making you contemplate a scenario that isn’t real. You end up dealing with fake pains, the phantom pains.

Prevention

“Drawing A Line In The Sand” by Henry Burrows

All of the above symptoms have something in common. They are all a product of early assumptions that lead to a complex system. At this point it’s useful to answer two questions: “what’s a complex system?” and “how do you program without making assumptions?”.

Well, for the purposes of this essay let’s say that a complex system is a system of networked nodes which consists of more nodes and connections than you can generally track in your head. Obviously, to get a low-complexity system you need to reduce the number of nodes and connections. As for the latter question, that’s what brings me to the main point. In order to program without making early assumptions, you must avoid doing things at runtime.

Let me elaborate. Having been scarred by over-modelling, I found that there is a principle that should become fundamental in all decision making. Let’s call it: keep it static, stupid, which seems appropriate because it’s really nothing more than a slightly more architecturally-aware riff on keeping things simple.

Making things static is the architectural equivalent of avoiding premature optimization.

The beauty of this principle is that it’s applicable on every abstraction level, regardless of whether you are talking about views, database, or code. The idea itself is simple: if in doubt, do it statically. It’s easier to understand what this means by looking at some concrete examples on various levels of a typical Rails app.

Can it be solved with a class?

Earlier in the post I mentioned pricing rules. This is a common problem where each product might abide by a different pricing algorithm. Price could depend on quantity, current user (think loyalty programs), order history, coupons, and various other things. To avoid the CMS trap I urge you not to allow constructing these kinds algorithms at runtime at an early stage. Write a pricing scheme class. Use strategy pattern. Make the pricing algorithm swappable at the code level. Define pricing rules via your programming language, this way the complexities of this logic can be mapped directly to code, and not warrant a whole layer of abstraction.

Programming languages already come with many wonderful tools, such as conditions and loops. Why reimplement them at a higher level of abstraction? These tools are more than enough to allow you to build complex pricing logic by writing code, directly. Once you have multiple pricing algorithms written as pluggable objects, feel free to let admin choose one, perhaps even “fill in the blanks” by plugging in factors and key values into your algorithm, but evolve this functionality gradually, as needed. Build out your admin UI with time, injecting more and more runtime flexibility into your strategy objects. Remember, you can always make static/hardcoded things dynamic, but not so much the other way. Everything that you make adjustable at runtime introduces complexity into every decision you make from that point on, and increases chances of bugs you cannot foresee, even in seemingly unrelated parts of your app.

Can it be solved with a static page?

Say you are listing things on a page for customer to see. These things may very well be products, photos, or files, whatever else it is that you are doing. Now, you have probably decided that there would be a title, a description, a picture, perhaps author or brand on each of those elements. You’ve split up your entities into these data fields and you decided to build database-backed models. This is where I’d suggest to stop and consider whether you have any good reason for why it can’t be a static page. A static templated view means that in order to change things, you have to edit the view and deploy, yes, but it also means that you don’t have controllers, models, migrations, forms, admin UI, or anything else. In fact, you might kind of still have an admin UI if you’re using Github. It’s not as real time as it could’ve been, but decent nonetheless. People can edit views on Github directly without much issue.

This becomes more of an issue if the things you list are categorized and otherwise laid out based on certain rules. In the dynamic approach, this would immediately force you to create a network of associated models just to render this sort of a page. Consider how little you know at this point about your future needs, and how constrained you will be having speculated your way towards that future. Consider also how quick and easy it would be to just sit down and hardcode this page. Just like the case with strategy pattern, you can always inject dynamic content into this page going forward, when the real needs arise. If you build out a dynamic system right away, you will likely end up constrained by it. Err on the side of static.

Can it be solved with a constant?

Getting back to the seed data issue, this example is fairly simple. You are creating categories. These categories are predefined. Instead of adding models, tables, and seed data, why not simply make a constant with an array? Code allows you to carry static data without involvement of a database. Use that, and wait until you really need the editing of categories at runtime. When that time comes, you could always extract data from the constant into the seed file, without any issues. Moreover, even that isn’t necessary. If you have some categories that never change, and some that should be manipulated in admin panel, you shouldn’t even seed the former ones. You could leave them in the constant, always read them from there, and this way avoid seed data altogether. It’s actually a little secret of mine. I don’t like seed data. It’s been years now, and our app works right out of the box on any new developer’s machine. If you pull our code into your dev machine, the app will just run. This is why I say: when in doubt, hardcode.

Can it be solved with a string in the database?

Say, at this point the app is working just fine, and you have your necessary database-backed models. You need to display a free-form text that may differ from one entity to another (e.g. different per product), yet it might contain certain values interpolated from elsewhere. As per the principle, you should not think about modelling this text via classes. First, ask whether you could simply get by with letting admin type the text as a string. But wait, you’d say. If this text has values plugged in from elsewhere, why should admin be typing them by hand? Would she have to look them up every time? Seems wrong. Well, relax a bit and consider canned snippets. That’s right, perhaps you can simply setup a free form text field while providing some pre-written text for admin, which has appropriate values already plugged in. When you think you need structured data for storing something highly flexible, consider instead using a plain string with canned snippets.

Wrapping up

“For every complex problem there is an answer that is clear, simple, and wrong.”
— H. L. Mencken

While the above text is a good general principle, it cannot exactly apply to problems that are clearly asking for CMS-like solutions. When you are tasked with building a highly-flexible CMS, that’s what you do, naturally. When you are asked to build an app with something like Drupal, you are in a whole different realm, where the CMS trap is pretty much your perpetual state of being. However, even in those special cases questions will arise whether to make something more or less dynamic, and I encourage every developer to always lean towards static. You will be doing a service not only to your future self, but also to the next developer, who would much rather slice up a piece of static html and inject some dynamic content than attempt to understand a steaming pile of speculative architecture with many moving parts.

It’s also important to note that I’m not advocating entirely against architecting up front. It’s good to a healthy extent, yet there is a line we draw in the sand on a case by case basis. I encourage you to think carefully about where to draw that line every time you implement something.

Speaking of my story, it ended with a year-long stagnation and a very reluctant revamp of the entire app. In the end, the aforementioned “blueprints” have been downgraded to hardcoded classes, and over time they have become very declarative, thanks to a naturally-evolving internal DSL. Seeing these files today and imagining how I’d proceed implementing runtime admin UI for all the moving parts is nightmarish. Even though it ate away a year, I’m still glad that we bit the bullet and refactored. It was painful, but now this mistake is far behind us.

Unless you know exactly what you’re doing (which is unlikely), stay static. Try to put extra effort into determining which parts of your business can be left hardcoded. If in doubt, hardcode. While doing that, make sure you follow best practices: never put the same conditions in two places, never repeat constant data, use composition, dependency injection, inheritance, whatever you need to make sure you abide single responsibility principle, and maintain singular authority.

Most importantly, don’t get yourself tangled in too much speculation, let the story unfold naturally.

Do nothing. All files in this dir are eager loaded in production and lazy loaded in development by default.

If you add a dir under app/something/

(e.g. app/models/concerns/, app/models/products/)

Ask: do I want to namespace modules and classes inside my new dir? For example in app/models/products/ you would need to wrap your class in module Products.

• If the answer is yes, do nothing. It will just work.

• If the answer is no, append the exact path in your application.rb.

  config.autoload_paths += %W( #{config.root}/app/models/products )

In either case, everything will be eager loaded in production.

If you add code in your lib/ directory

Option 1

If you put something in the lib/ dir, what you are saying is: “I wrote this library, and I want to depend on it where I decide.” This means that if you use your library in a rake task, but not in a rails app, you just require it in your rake task. If you need this library to always be loaded for your rails app, you require it in an initializer. If you need this library for some of your models or controllers, you require_dependency (see below why) it in those files, and since everything under your app/ dir is already auto- and eager- loaded as needed, your library will only be “pulled-in” if something that requires it from app/ or rake, or your custom script, actually gets loaded.

Option 2 (bad)

Another option is to add your whole lib dir into autoload_paths.

config.autoload_paths += %W( #{config.root}/lib )

This means you shouldn’t explicitly require your lib anywhere. As soon as you hit the namespace of your dir in other classes, rails will require it. The problem with this is that in Rails 3 if you just add something to your autoload paths it won’t get eager loaded in production. You would need to add it to eager_load_paths instead, which causes a different problem (see below). And in ruby 1.9 autoload is not threadsafe. You probably want eager loading in production. Requiring your lib explicitly, like in option 1, is akin to eager loading it, which is threadsafe.

Option 3 (meh)

All the different things under your lib dir should be placed into their own directories, and those directories should be individually added to eager_load_paths.

config.eager_load_paths += %W(
  #{config.root}/lib/my_lib1
  #{config.root}/lib/my_lib2
)

This means that you can’t just throw files into your lib dir. If you have my_lib1.rb, you must put it under my_lib1/my_lib1.rb and my_lib1 should be added to eager load paths. This means that if you have more files in my_lib1, you should create a dir my_lib1/my_lib1/extra.rb. This is a bit annoying.

So why not just add lib/ into eager_load_paths?

If you add lib/ into eager_load_paths, everything will work great. Your files will be autoloaded in development, and eager-loaded in production. Except the problem is that eager_load_paths use globbing like lib/**/*.rb, meaning that everything in your lib dir will try to get loaded. Your tasks, your generators, everything. This is not what you want.

Organizing lib

Regardless of which option you pick (option 1, hint hint), in your lib/ dir you should structure your code as if you structure a gem. If you need more than 1 file, you could for example add a same-named directory where everything is properly namespaced, and let your 1 file relatively require files in that directory.

Why use require_dependency (auto-reloading)

If you use require_dependency, you are enabling auto-reload of your files in development across requests. require alone won’t do it. I suggested to use it in your rails app, but not in initializers or rake tasks because rake tasks only run once, and changing initializers always requires restart.

However, it won’t work without one additional piece of configuration. In application.rb you should add this:

config.watchable_dirs['lib'] = [:rb]

P.S. I originally posted this article in a gist.

class Product < ActiveRecord::Base
end

class Tee < Product
end

class Pen < Product
end

This inheritance looks reasonable, but now we have to come up with relational database structure. We need to find a way to store tee’s own attributes, pen’s own attributes, as well as their common (product’s) attributes without duplication. Some databases (PostgreSQL) provide support for table inheritance, but it’s a specialized feature which ties you down to the given db.

Single table inheritance

ActiveRecord provides only one way to handle a is_a relationship which is Single Table Inheritance. You’d have to create a table looking somewhat like the following.

The problem here is that all attributes are stored in the same table. It’s likely that soon the number of attributes will grow unmanageable, and most of them will always stay NULL since they’ll be specific to only one type.

Polymorphic has_one association

A has_one association allows us to split out tees, pens, and products into three different tables. In fact — as you’re about to see — this is the only way to get what we want. The problem is that it creates a has_a relationship, and we want is_a. Since there isn’t much choice, we can make it look like we have an is_a relationship, which I’m about to show.

Multiple table inheritance (simulated)

#railsbridge IRC channel about ways to achieve something like MTI with Active Record. He pointed me to a pastie where he implemented an MTI-like behavior and called it a “hydra” pattern, which I subsequently cleaned up a bit.

So we want to have 3 tables in the database.

• product_properties
• tees
• pens

class ProductProperties < ActiveRecord::Base
  belongs_to :sellable, :polymorphic => true, :dependent => :destroy
end

class Tee < ActiveRecord::Base
  has_one :product_properties, :as => :sellable, :autosave => true
end

class Pen < ActiveRecord::Base
  has_one :product_properties, :as => :sellable, :autosave => true
end

Immediately we can see duplicated code between Tee and Pen. This can be easily solved with a mixin.

module Sellable
  def self.included(base)
    base.has_one :product_properties, :as => :sellable, :autosave => true
  end
end

class Tee < ActiveRecord::Base
  include Sellable
end

class Pen < ActiveRecord::Base
  include Sellable
end

Now comes another issue. Every time we want to access price or title attributes (stored in product_properties) we have to call @tee.product_properties.price. This isn’t very convenient, especially considering that product_properties has to be built first in case it doesn’t exist. So let’s ensure it’s always built by updating the module.

module Sellable
  def self.included(base)
    base.has_one :product_properties, :as => :sellable, :autosave => true
    base.alias_method_chain :product_properties, :autobuild
  end
  
  def product_properties_with_autobuild
    product_properties_without_autobuild || build_product_properties
  end
end

Awesome, now product_properties is built automatically in case it doesn’t exist. We still have the method accessing issue though. For that I used method_missing.

module Sellable
  def self.included(base)
    base.has_one :product_properties, :as => :sellable, :autosave => true
    base.alias_method_chain :product_properties, :autobuild
  end
  
  def product_properties_with_autobuild
    product_properties_without_autobuild || build_product_properties
  end
  
  def method_missing(meth, *args, &blk)
    if product_properties.public_methods.include?(meth.to_s)
      product_properties.send(meth, *args, &blk)
    else
      super
    end
  end
end

Now if a method is missing from Tee or Pen instance it will be delegated to product_properties, which enables us to use @tee.price and @tee.title.

However, what about validations? Let’s say we want all products to always have a title, and we want to see an error appear on a Tee instance when ProductProperties#title is missing. Basically I want to completely remove product_properties from my sight as if it doesn’t exist, make it absolutely transparent. Let’s add the necessary validation in ProductProperties.

class ProductProperties < ActiveRecord::Base
  belongs_to :sellable, :polymorphic => true, :dependent => :destroy
  validates_presence_of :title
end

And now let’s make all Sellable models respect the validation as if it’s their own.

module Sellable
  def self.included(base)
    base.has_one :product_properties, :as => :sellable, :autosave => true
    base.validate :product_properties_must_be_valid
    base.alias_method_chain :product_properties, :autobuild
  end

  def product_properties_with_autobuild
    product_properties_without_autobuild || build_product_properties
  end

  def method_missing(meth, *args, &blk)
    if product_properties.public_methods.include?(meth.to_s)
      product_properties.send(meth, *args, &blk)
    else
      super
    end
  end

  protected

  def product_properties_must_be_valid
    unless product_properties.valid?
      product_properties.errors.each do |attr, message|
        errors.add(attr, message)
      end
    end
  end
end

Notice that I’m including an additional validator with the Sellable module. The validator collects all the errors on ProductProperties and adds them to parent class as if the errors are on a Tee or Pen itself.

As a nice finishing touch we can put this snippet into a Rails initializer.

class ActiveRecord::Base
  def self.acts_as_product
    include Sellable
  end
end

# now we can say

class Tee < ActiveRecord::Base
  acts_as_product
end

Although that’s a matter of taste.

Fixing method_missing

There is a problem with method_missing. It checks the array of public_methods on product_properties to find out if delegation should occur. This check will fail in cases like @tee.title_changed?. That’s a magic method and therefore will not be part of static method array. Well, this is an easy fix.

# Replace old method_missing with this one:
def method_missing(meth, *args, &blk)
  product_properties.send(meth, *args, &blk)
rescue NoMethodError
  super
end

As you can see, even magic methods will work this way. Only if a NoMethodError is thrown we withdraw back into super.

Handling attributes hash

In the comments Austin brought up a case where initializing new models like Tee.new(:title => "foo") will throw an unknown attribute error. That’s expected since we rely on method_missing for accessing ProductProperties attributes. Instead we should define accessor methods explicitly in our individual products. Thankfully, it’s not too hard to accomplish with our Sellable mixin. First we need to add a submodule ClassMethods with a class method that uses class_eval to magically generate missing attributes.

module ClassMethods
  def define_product_properties_accessors
    all_attributes = ProductProperties.content_columns.map(&:name)
    ignored_attributes = ["created_at", "updated_at", "sellable_type"]
    attributes_to_delegate = all_attributes - ignored_attributes
    attributes_to_delegate.each do |attrib|
      class_eval <<-RUBY
        def #{attrib}
          product_properties.#{attrib}
        end
        
        def #{attrib}=(value)
          self.product_properties.#{attrib} = value
        end
        
        def #{attrib}?
          self.product_properties.#{attrib}?
        end
      RUBY
    end
  end
end

I’ll walk through this code quickly. First we’re extracting only the columns that we want to access. When we call content_columns in the first line of the method, it already excludes a bunch of special columns such as id and type. We then manually subtract more columns we’d like to ignore, such as timestamps, and polymorphic type.

Next we iterate over each remaining attribute and creating instance methods for it, such as title, title= and (for completeness) title?. Having these accessors defined explicitly is enough for ActiveRecord to see them when performing mass assignment, etc. We can now do something like Tee.new(:title => "foo") without any problems. The extra cases such as @tee.title_changed? are still handled by method_missing so we’re good.

One more thing left. We need to run this method on the base class into which we include Sellable. Just need to add a couple of lines to the self.included hook.

def self.included(base)
  base.has_one :product_properties, :as => :sellable, :autosave => true
  base.validate :product_properties_must_be_valid
  base.alias_method_chain :product_properties, :autobuild
  
  # Add these two lines:
  base.extend ClassMethods
  base.define_product_properties_accessors
end

And we’re all set.

All together now

Here’s the full picture of everything we just did.

class ActiveRecord::Base
  def self.acts_as_product
    include Sellable
  end
end

class ProductProperties < ActiveRecord::Base
  belongs_to :sellable, :polymorphic => true, :dependent => :destroy
  validates_presence_of :title # for example
end

module Sellable
  def self.included(base)
    base.has_one :product_properties, :as => :sellable, :autosave => true
    base.validate :product_properties_must_be_valid
    base.alias_method_chain :product_properties, :autobuild
    base.extend ClassMethods
    base.define_product_properties_accessors
  end

  def product_properties_with_autobuild
    product_properties_without_autobuild || build_product_properties
  end

  def method_missing(meth, *args, &blk)
    product_properties.send(meth, *args, &blk)
  rescue NoMethodError
    super
  end

  module ClassMethods
    def define_product_properties_accessors
      all_attributes = ProductProperties.content_columns.map(&:name)
      ignored_attributes = ["created_at", "updated_at", "sellable_type"]
      attributes_to_delegate = all_attributes - ignored_attributes
      attributes_to_delegate.each do |attrib|
        class_eval <<-RUBY
          def #{attrib}
            product_properties.#{attrib}
          end

          def #{attrib}=(value)
            self.product_properties.#{attrib} = value
          end

          def #{attrib}?
            self.product_properties.#{attrib}?
          end
        RUBY
      end
    end
  end

  protected

  def product_properties_must_be_valid
    unless product_properties.valid?
      product_properties.errors.each do |attr, message|
        errors.add(attr, message)
      end
    end
  end
end

class Tee < ActiveRecord::Base
  acts_as_product
end

class Pen < ActiveRecord::Base
  acts_as_product
end

This can be easily adapted for any other use case besides products in a store. In fact, with some meta magic or code generation this can easily be made into a plugin which I encourage you to try and send me the link when you’re done. :)