tech

2026-04-08T19:49:00.000-07:00

The Prompt That Crossed Two Organizations

Engineering × AI

The Prompt That Crossed Two Organizations — And Got Sharper Each Time

How a product executive's pressure-testing framework traveled to systems engineering, and what happened when we pointed it at a real AWS workflow.

By Thushara Wijeratna · with Scott Johnson & Ryan Polley

There's a quiet revolution happening in how smart teams use AI — and it has nothing to do with the model. It has everything to do with the instructions.

A few weeks ago I borrowed a prompt template from a product owner at a large enterprise. He had built it in ChatGPT to do something powerful: give his leadership team a private space to pressure-test ideas before they ever entered a room. When executives were developing roadmaps, he'd run them through the model first — surfacing assumptions, stress-testing the logic, anticipating the hard questions a CFO or GM might raise. The result was a win on both sides of the table. The CEO could arrive at conversations with a sharper, more fully-formed point of view. And the product manager got to execute against a plan that had already survived serious scrutiny — no half-baked pivots, no surprises mid-flight.

Think of it less as critique and more as a rehearsal room. The tool doesn't challenge people — it challenges ideas, privately, before the stakes are high.

I took the same framework and ran it in Claude — the model we use at Solo. It worked just as well. Which raises something worth sitting with: the framework didn't just travel across organizations and domains. It traveled across AI models entirely. That's the tell. When the same set of instructions produces sharp, useful output regardless of which model is running them, the instructions are the asset. The model is increasingly the commodity.

I read those instructions and thought: this exact mental model applies to systems architecture.

So I adapted them. Same seven-step skeleton — identify the thesis, stress-test the portfolio balance, expose assumptions, map risk concentration, name the opportunity costs, simulate the leadership challenge, propose alternative shapes. I changed the vocabulary and the lens. Instead of asking what will Finance push back on, I asked what will the CEO and engineering team challenge. Instead of scoring for revenue potential, I scored for cost and time savings. The goal shifted from sharpening an executive's boardroom instincts to sharpening an engineer's thinking in peer and leadership conversations.

Then I pointed it at something real: the workflow our team at Solo uses to create new EventBridge scheduled rules in AWS.

— —

What the workflow looked like

The process isn't complicated on its face. You take a YAML template, swap in five dynamic variables, insert the block into a CloudFormation file, open a PR, get infra review, run a change set in the AWS console, verify, execute, merge. Clear enough. Solid IaC discipline — no console drift, changes tracked in git, peer review required.

What happened next is what makes this worth writing about.

— —

What Claude found

Step 1 — Strategic thesis

The thesis was absent. Claude's first move was to name this: defensive / constraint-driven, no clear thesis. The process enforces the right principle — infrastructure as code over console changes — but it's encoded as a manual human checklist rather than a system with guardrails. The implied thesis should be "infrastructure changes are safe, auditable, and low-friction." Claude's verdict: it's only partially achieving that.

Step 2 — Portfolio balance

The portfolio is completely unbalanced. Claude broke the workflow down across four dimensions: IaC discipline (present), automation and tooling (absent), validation guardrails (absent), developer autonomy (weak). Its summary was blunt: all weight is on human process, zero investment in tooling that would make this self-service and safe simultaneously.

Step 3 — Assumption exposure

The assumptions don't hold up. Claude identified four things that must be true for this process to succeed — and challenged each one:

"Engineers will follow the checklist correctly every time." There's no validation that catches a malformed cron expression, wrong snake_case, or a misaligned variable before it hits CloudFormation.

"Infra team review is a meaningful gate." Unclear. If it's checking syntactic correctness, a linter does this faster and more reliably. If it's checking strategic intent, that's not documented anywhere.

"The change set review step prevents mistakes." Partially — but it requires the engineer to know what to look for. No checklist defines what "looks good" actually means.

"Console changes never happen." The last line of the process says never change the rule from the AWS Console — but there's no enforcement mechanism. That's policy, not a guardrail.

Step 4 — Risk concentration

The risk concentration is real. One engineer executing the process incorrectly causes a production scheduling outage or a silent missed execution. The template has six-plus Fn::ImportValue calls — a single upstream stack name change silently breaks every rule, with no cross-reference validation documented. And the process assumes the engineer simultaneously knows valid cron syntax, CloudFormation change set semantics, ECS task override structure, and rake task naming conventions. That's a high knowledge bar with no scaffolding.

The most underappreciated risk lives in the last line: "Never change the rule from the AWS Console." A critical constraint buried where engineers are least likely to retain it.

If someone misses it — especially under time pressure — they introduce configuration drift that CloudFormation won't detect on the next deploy. The instruction exists. The enforcement doesn't.

— —

The leadership challenge simulation

Claude predicted exactly the questions that would land hardest in an executive or peer review:

CEO / Engineering lead

"How many engineer-hours does this take per rule addition, end to end? What's the error rate? Have we had production scheduling failures from this process?"

Probably no clean answers. That's the problem.

Finance

"If a scheduled job silently fails to be created or runs at the wrong time, what's the business impact?"

The process has no alerting or confirmation that a newly added rule is actually firing. Execution success is not the same as operational correctness.

Engineering

"Why do I need infra team review for a cron job? Why can't I validate this locally? Why is there no test environment path?"

Claude's assessment: these are legitimate objections. The current process treats every engineer as a potential misconfigurer rather than building systems that make misconfiguration hard.

— —

Three alternative shapes

Claude proposed three different strategic directions, each with honest tradeoffs:

Option A — Recommended near-term

Script-first

Build a small CLI that prompts for inputs, validates cron syntax, generates the YAML block, and inserts it correctly into the file. This eliminates the entire class of template variable substitution errors and removes the knowledge burden from the engineer.

Eliminates substitution error class entirely One-time build and maintenance investment

Option B

Separate stack per rule

Instead of one monolithic YAML file, each rule gets its own CloudFormation stack. Engineers own their rule's lifecycle. Merge conflicts disappear. Blast radius is isolated.

Eliminates merge conflicts, isolates blast radius Stack proliferation requires naming discipline

Option C — Longer horizon

Migrate to EventBridge Scheduler + CDK / Terraform

Replace CloudFormation-managed EventBridge Rules with the purpose-built newer service, managed through CDK or Terraform modules. Better DX, built-in retry policies, templated constructs reduce copy-paste risk significantly.

Better DX, built-in retries, less copy-paste risk Migration cost, team upskilling, short-term disruption

Claude's bottom line: This process enforces the right principle with the wrong mechanism. The risk isn't that engineers are careless — it's that the process provides no structural resistance to errors. A linter, a generator script, and a validation step in CI would eliminate the majority of failure modes at low cost. The highest-leverage immediate action: a script that generates the YAML block from inputs and validates cron syntax before the PR is opened. Everything else can wait.

— —

What this is really about

The genealogy of this critique is what I keep coming back to. A product owner at an enterprise company built a framework in ChatGPT to make product leaders sharper. I adapted it for Claude to make systems engineers sharper. The seven-step skeleton traveled across two organizations, two domains, two AI models, and two completely different problems — and produced something genuinely useful every time.

That last part matters more than it might seem. We're entering a moment where the major AI models are converging in capability. The choice between them is increasingly a matter of workflow preference, not raw power. What doesn't transfer automatically — what has to be deliberately designed — is how you instruct them. The same prompt that works in ChatGPT works in Claude. The same framework that sharpens a product roadmap sharpens an engineering workflow. The instructions are the portable, reusable, compounding asset. The model is the infrastructure underneath.

We spend a lot of time evaluating which AI model to use and almost no time designing how we instruct it. The difference between an AI that validates your thinking and one that challenges it isn't the model version — it's the instruction set. One framing decision, encoded in a project's system prompt, shifts the output from agreeable to adversarial, from a mirror to a pressure test.

The insight the enterprise product owner had — that you can force structured, sequential reasoning by encoding a multi-step framework as the operating instruction — turns out to be domain-agnostic and model-agnostic. The same architecture works on roadmaps, on engineering workflows, on financial models, on hiring processes. You change the vocabulary. The sharpness is the point.

The prompt that lives in our Claude project now means any engineer can walk in with a workflow, a design doc, or an architectural decision and get back something that will make them think harder — not feel better.

That's the unlock. And it cost nothing but the willingness to borrow a smart idea from someone doing a completely different job, on a completely different platform, solving a completely different problem.

The best prompts, it turns out, travel well.

Want to build your own pressure-testing project? The pattern is straightforward: pick an adversarial advisor persona, write a multi-step reasoning framework that forces each analytical lens to run in sequence, and explicitly ban validation as a default behavior. It works in Claude. It works in ChatGPT. The framework above has seven steps — but what makes it work isn't the number, and it isn't the model. It's the instruction not to let weak reasoning slide.

How We Migrated Sidekiq's Redis Without Losing a Single Job

2026-03-26T20:35:00.000-07:00

How We Migrated Sidekiq's Redis Without Losing a Single Job

Infrastructure · Redis · Sidekiq

How We Migrated Sidekiq's Redis Without Losing a Single Job (and Without Listening to AI)

Solo Engineering Team · March 2026 · 8 min read

We moved our Sidekiq backend from Redis Enterprise to AWS ElastiCache. The AI tools recommended a careful, expensive approach. We did something simpler — and it worked perfectly.

The Setup

Our app runs Sidekiq workers on ECS. Each process connects to Redis on startup to read and process jobs. We were moving from Redis Enterprise to ElastiCache — different host, different connection string, same protocol.

New jobs would start going to the new Redis as soon as we deployed. But existing jobs queued in the old Redis? They'd be orphaned the moment every worker switched over.

What the AI Tools Said

We asked around — Claude, ChatGPT, Gemini, Grok. They all landed in roughly the same place:

You should deploy a separate environment connected to the old Redis. Let it drain the queue over time, then decommission.

It's not wrong. But it's heavy. That approach meant new ECS task definitions, environment variable management across two sets of infra, coordinating the decommission, and extra cost while two clusters run in parallel.

When we pushed back, one tool offered an alternative: run two Sidekiq processes per Docker container — one pointed at old Redis, one at new. That would have required changes to CloudFormation templates, process supervision config inside the container, and careful cleanup afterward. Trading one complex migration for another.

But they missed something important: Sidekiq's backing store is completely external to the process. A job scheduled on Redis Enterprise doesn't belong to any particular Sidekiq process — it just sits there until a worker with a connection to that Redis comes along. The worker is stateless.

So the "debugging nightmare" scenario the AI tools described... wouldn't actually happen.

The Actual Solution

Our team came up with something much simpler. In config/initializers/sidekiq.rb, at startup, each Sidekiq process decides which Redis to connect to. We added one line:

config/initializers/sidekiq.rb — the one-liner

# Coin toss at startup — connects this process to one Redis for its entire lifetime
redis_url = rand < 0.5 ? LYMO_SIDEKIQ_NEW_REDIS_URL : LYMO_SIDEKIQ_OLD_REDIS_URL

That's it. On startup, each worker tosses a coin. Heads → new ElastiCache. Tails → old Redis Enterprise.

The result: roughly half the cluster continued draining the old queue, while the other half processed new jobs on ElastiCache. No new infra. No task definition changes. No separate environment to coordinate.

We also pointed all job producers (the code that enqueues jobs) at the new Redis immediately. So new work only ever went to ElastiCache. The old Redis just needed to drain.

This is where Sidekiq's initializer structure becomes the key enabler. Each configure_server and configure_client are can be wired seperately where the server (one that reads) uses the redis_url resolved at startup:

config/initializers/sidekiq.rb — full initializer

redis_url = rand < 0.5 ? LYMO_SIDEKIQ_NEW_REDIS_URL : LYMO_SIDEKIQ_OLD_REDIS_URL

Sidekiq.configure_server do |config|
  config.redis = { url: redis_url }
end

Sidekiq.configure_client do |config|
  config.redis = { url: new_redis_url }
end

One coin toss. One URL to pull from. That process reads and from the same Redis for its entire lifetime.

The clients (that push jobs) will always use the new url, and the reads would be split between the old and new url. In time, the old queue drains as it receives no further jobs. The old Redis processes were naturally left behind to drain, and as they cycled out, the cluster fully converged on the new setup with no intervention required.

How It Went

It worked exactly as expected. Within a day, roughly 90% of the old queue had drained naturally. Workers reading from old Redis gradually found less and less work, while ElastiCache handled all the new throughput.

The remaining jobs were a different story: scheduled jobs. These live in Sidekiq's sorted set and don't get picked up until their execution time arrives — which could be hours away. Waiting wasn't ideal, so we wrote a small script to move them from the old Redis to the new one manually. A few lines to iterate the scheduled (and retry) set, re-enqueue on ElastiCache, and delete from old Redis. Clean cutover.

Once that was done, we deployed the cleanup — removed the conditional and all references to the old Redis connection. Four lines of code deleted. Done.

Oh, and while all of this was happening? The rest of the team made a dozen normal deployments — which restarted servers, reshuffled which Redis each process landed on, and generally did everything the AI tools said would cause a debugging nightmare. Nothing broke. No jobs lost. The doom and gloom theories were disproven in the most practical way possible: by live testing.

Why the AI Advice Missed the Mark

The AI tools were technically cautious but operationally naive. They modeled the problem as "jobs are tied to a running process" — which isn't how Sidekiq works. Redis is the source of truth, not the worker. The worker is stateless.

They also defaulted to the safest, most conservative architecture: full environment isolation. That's sensible for high-stakes migrations. But for a queue drain, it's significant overengineering.

The human insight — the DB is external, the workers are stateless, so we can split them probabilistically — is the kind of lateral thinking that comes from actually understanding the system rather than pattern-matching to a template.

— ✦ —

Takeaways

01
Sidekiq workers are stateless. Redis is the state. This gives you more migration flexibility than you might think.
02
Probabilistic splits are underrated. You don't always need clean cutoffs. A coin toss at startup is simple, observable, and reversible.
03
AI tools are good at safe answers, not always good at efficient ones. They'll often recommend the conservative solution even when a simpler one exists. Treat their output as a starting point, not a final answer.
04
The cleanup should be as simple as the migration. If your migration leaves behind complex infra, you've done too much. Ours cleaned up with four deleted lines.

Dead Code Is a Cognitive Tax — Here's How AI Helps You Stop Paying It

2026-03-15T13:23:00.000-07:00

Dead Code Is a Cognitive Tax — Here's How AI Helps You Stop Paying It

Posted to Engineering · [Your Name] · [Date]

Every engineer knows the feeling. You open an unfamiliar part of the codebase, and you're immediately staring down a tangle of services, workers, models, and task entries — none of which come with a label saying "still matters" or "abandoned in 2023." You read the code carefully, try to trace the call graph, maybe even grep for usages — and only after 30 minutes do you realize: this thing hasn't run in production for over a year.

That tax on your attention has a name: cognitive load. And dead code is one of its most insidious sources.

What Is Cognitive Load in a Codebase?

Cognitive load, in the context of software engineering, is the total mental effort required to understand a system well enough to work in it safely. Every class, method, model, and background job you encounter is a unit of context you have to hold in your head.

The problem is that your brain doesn't automatically know which of those units are live and which are ghosts. If an EstimateWorker class exists in your repo, you have to assume it matters — until you prove otherwise. That proof takes time, attention, and often a distracting detour away from the actual work you sat down to do.

Dead code doesn't just waste disk space. It actively misleads you.

A Real-World Example: The Estimation Pipeline Cleanup

Recently, our team completed a cleanup effort across seven pull requests targeting a legacy estimation infrastructure — a suite of services originally built around Prophet forecasts and a Clair analysis pipeline — that had gone completely dark since late 2023.

Here's what was still sitting in the codebase, doing nothing:

EstimateService — fetched a CSV over HTTP, upserted records into the database, and refreshed an estimation cache. Silent for months.
EstimateWorker — a Sidekiq background job that uploaded files to S3, triggered the estimation flow, and posted Slack notifications. Long dead.
Estimation::Prophet::DownloadWorker — downloaded forecast CSVs from S3 and upserted them into a Prophet table. Never called.
Estimators::ClairAnalysis — computed hourly analysis records for a brief window in late 2023, then stopped.
ClairAnalysis model and its backing database table — zero writes since the pipeline went quiet.
Three SwitchBoard dispatch entries — events_collect_for_next_week, generate_weekly_user_report, estimate_v2 — all orphaned task names in a routing map.

Any engineer — or AI assistant — reading this codebase would reasonably assume all of the above was active production infrastructure. None of it was.

The Numbers

Pull Requests

Files Changed

943

Lines Deleted

−816

Net Lines Removed

PR	Branch	+Added	−Deleted	Files
#1	cleanup-tasks	13	16	2
#2	cleanup-unused-estimate	0	74	4
#3	remove-clair-analysis	0	314	2
#4	remove-prophet	0	210	5
#5	remove-clair-analysis-model	20	57	3
#6	rename-clair-v2s	94	68	13
#7	remove-estimate-unused	0	204	2
Total		127	943	31

The 127 additions are almost entirely the rename PR (#6) — migrations, updated references, and renamed specs. Every other PR was pure deletion.

The Cognitive Impact of the Cleanup

Cleaner model surface. Once EstimateService, EstimateWorker, and ClairAnalysis were gone, the remaining models — Clair, ClairDailyInterimResult, ClairSetting — actually reflected how the system works today.

Naming that signals intent. ClairV2 implies a versioning scheme. ClairDailyInterimResult tells you exactly what the thing is and why it exists.

A smaller SwitchBoard dispatch map. Removing the three orphaned entries made the dispatch map honest again.

A shorter test suite that still covers everything that matters. Several spec files covering deleted code were removed. The test suite got faster without losing any meaningful coverage.

Where AI Fits In: Finding Dead Code You Can't See

Here's the uncomfortable truth about dead code: it's often invisible to the people closest to it. If you wrote EstimateWorker two years ago and the team that decommissioned the upstream service never filed a ticket, you might not even know it's dead. The code looks fine. The tests pass. Nothing alerts you.

A Telling Real-World Example: Claude Gets Confused, Then Catches Itself

We recently asked Claude to generate a flow diagram of our pay guarantee process. Claude produced a diagram that looked plausible — tracing through services, models, and workers in a way that made logical sense.

The problem? Part of that diagram was wrong — because Claude had incorporated a module that was no longer active into its understanding of the flow. The dead code was so well-structured and apparently coherent that the AI read it as live infrastructure and wove it into the diagram without hesitation.

But here's what makes this story instructive rather than just cautionary: When an engineer removed this hopefully last bit of dead code, Claude immediately realized that the diagram she drew earlier relied on this bad signal, revised its understanding, and corrected the diagram.

That sequence — confidently wrong, then self-correcting — is a useful frame for thinking about AI and dead code. It fooled the AI for the same reason it fools engineers: it looks like it belongs.

What AI Can Do

Tracing call graphs at scale. AI can trace the full call graph of a function or class across an entire monorepo — answering not just with direct callers, but with the absence of callers.

Cross-referencing runtime signals with static code. When connected to observability data — logs, APM traces, queue metrics — an AI can compare what the code says it does with what actually runs in production.

Flagging stale patterns. Dead code has fingerprints: models with no recent migrations, task names absent from any scheduler config, service classes with no callers outside their own spec files.

Drafting cleanup PRs. Once dead code is identified, AI can help draft the actual removal — proposing what to delete, what to rename, and what specs to clean up.

What AI Can't Do (Yet)

AI isn't a replacement for engineering judgment. A worker might be "dead" in CI but still referenced by a cron job in an ops runbook nobody's touched in three years.

The right model is AI as a scout, engineer as the decision-maker. AI surfaces candidates. Engineers verify, contextualise, and own the deletion.

Making Dead Code Cleanup a Habit

Timestamp your decommissions. When you turn off a pipeline, leave a comment in the code with the date.
Review your task dispatch maps regularly. A quarterly review catches orphaned entries before they fossilise.
Use AI during onboarding and code review. AI tools can help new engineers quickly validate whether something is live — and surface it for cleanup if it isn't.
Treat deletion as a first-class deliverable. 816 lines removed is a meaningful engineering contribution. Make it visible in sprint planning, changelogs, and retros.

Conclusion

Large codebases accumulate cognitive debt quietly, continuously, and with compounding interest. Dead code is one of the most expensive line items: it misleads engineers, bloats test suites, and turns routine code reading into archaeology.

As we saw first-hand, it even misleads AI. Claude confidently incorporated a dead module into a flow diagram of our pay guarantee process — because the code looked live. That moment of confusion, and the self-correction that followed, is a perfect metaphor for where we are with AI-assisted engineering today: powerful, promising, and most effective when paired with good runtime context and human judgment.

The goal isn't a perfect codebase. It's a codebase where the code you're reading is the code that's actually running. That's a goal worth shipping toward.

When AI Sounds Right But Isn't: A Sidekiq Story

2026-02-26T16:39:00.000-08:00

I recently ran into a Redis migration problem with Sidekiq. I needed to point our workers at a new Redis instance while safely draining jobs still sitting in the old one. I turned to Claude for advice — and what followed is a good reminder of why GenAI output should always be treated as a starting point, not a final answer.

Step 1: My Initial Idea (and Why Claude Correctly Pushed Back)

My first instinct was a "coin toss" approach: configure the Sidekiq server to point to the old Redis with 50% probability and the new Redis otherwise, while the client always pushes to the new Redis. Once the old queue drained, I'd do a clean cutover.

Claude's response here was actually good:

Claude: The coin toss approach has some real problems... Non-deterministic behavior is hard to reason about in production. You lose retry guarantees. A job that fails while a worker is connected to the old Redis will be retried there — but if on the next attempt the worker happens to connect to the new Redis, it won't see that job. The job could get stuck or lost. Sidekiq server connects to one Redis at startup, not per-job...

Fair enough. The critique was accurate. Claude then suggested a cleaner alternative: use Sidekiq 7's config.capsules to run multiple Redis connections within a single process — one capsule for the new Redis, one to drain the old.

Step 2: A Confident, Detailed, Completely Wrong Answer

I was on Sidekiq Pro 7.3.9, so I asked Claude to show me how to configure capsules for this use case. It gave me this:

Sidekiq.configure_server do |config|
  # New Redis - primary capsule (default)
  config.redis = { url: new_redis_url }

  # Old Redis - drain capsule
  config.capsule("legacy") do |cap|
    cap.redis = { url: old_redis_url }
    cap.concurrency = 5
    cap.queues = %w[default mailers critical]
  end
end

Sidekiq.configure_client do |config|
  config.redis = { url: new_redis_url }
end

The response was detailed. It used the right vocabulary. It even included helpful caveats about thread pool sizing, checking the retry and scheduled sets before teardown, and queue enumeration via redis-cli SMEMBERS queues. It felt authoritative.

There was just one problem: it was wrong.

Step 3: Thirty Seconds in the Docs

Before implementing anything, I did a quick check of the official Sidekiq documentation. Here's what it says, plainly:

Sidekiq Docs: "There is still one iron-clad rule: a Sidekiq process only executes jobs from one Redis instance; all Capsules within a process must use the same Redis instance. If you want to process jobs from two separate Redis instances, you need to start two separate Sidekiq processes."

Capsules are for organizing queues and concurrency within a single Redis instance. They cannot span multiple Redis instances. The entire premise of Claude's solution was invalid.

When I pointed this out, Claude acknowledged it immediately:

Claude: You're right, I was wrong — capsules cannot span multiple Redis instances. They're for organizing queues/concurrency within a single Redis. So for your drain scenario, you actually need two separate Sidekiq processes with different config files.

No pushback. No hedging. Just a clean admission that the confident, detailed answer was incorrect.

What Makes This Failure Mode Dangerous

This isn't a story about AI being obviously broken. It's about AI being convincingly wrong — and that's a harder problem to guard against.

A few things made this particularly easy to fall for:

The answer was structurally sound. Capsules are a real Sidekiq 7 feature. The code was syntactically valid Ruby. The caveats about concurrency and retry sets were genuinely useful. Only the core assumption — that capsules can target different Redis instances — was wrong.
The fluency signals trust. When an answer uses the right terminology, references the right version numbers, and anticipates edge cases, it reads as expert. That fluency is a product of training on large amounts of text, not of verified understanding.
The model doesn't know what it doesn't know. Claude didn't say "I'm not certain about the multi-Redis constraint — check the docs." It presented the solution as if it were established fact.

A Simple Rule of Thumb

If you wouldn't ship code based solely on a Stack Overflow answer from 2019 without reading the docs, don't ship code based solely on a GenAI answer either. The bar should be the same — or higher, because at least the Stack Overflow answer has upvotes, comments, and a date stamp.

GenAI is genuinely useful for orientation: understanding an unfamiliar API surface, exploring options, getting unstuck. But any answer that involves a specific documented behavior — especially version-specific constraints — needs at least one authoritative source check before you act on it.

In this case, thirty seconds in the Sidekiq docs saved what could have been hours of debugging a fundamentally broken architecture. That's a pretty good return on thirty seconds.

The actual solution, if you're curious: two separate Sidekiq processes with separate config files, each pointing at a different Redis instance. One processes new work, one drains the old queues. When the old queue, retry set, and scheduled set are all empty, shut the old process down.

Death-Defying Sidekiq Jobs: Part 2

2025-11-28T19:23:00.000-08:00

In my previous post, I outlined the problem of parent jobs getting killed during Sidekiq shutdowns because they took too long to enqueue child jobs. We implemented a solution that used an active driver index instead of the expensive redis iterator, but the story doesn't end there.

The Data Revealed More

After deploying the active driver index, I gathered metrics on the parent job execution times. The good news: runtime dropped significantly. The bad news: even with the new index, the higher percentile execution times still hovered around 40 seconds.

That 40-second ceiling was a problem. Sidekiq's shutdown grace period is 25 seconds by default, and while we could extend it, we'd just be postponing the inevitable. Jobs that take 40 seconds to enqueue children are still vulnerable to being killed mid-execution during deployments or restarts.

Enter `bulk_perform`

The problem was that we had 100,000 jobs to push to sidekiq and while each push was in the order of a micro-second or less, the math adds up, and soon we are waiting close to a minute till all jobs were sent. I knew that this was a common problem with I/O bound systems where generally a "bulk" operation comes to the resuce. As in database writes, where we need to write a thousand records, we use a bulk insert, where through a single connection/call, the client sends a 1000 prepared statements that then are executed as a single batch in the database server (ex: postgres). A quick GenAI search hit upon bulk_perform - a method specifically designed for this exact scenario in the sidekiq world. Instead of enqueuing jobs one at a time, bulk_perform allows you to asynchronously submit up to 1,000 jobs to Sidekiq at once.

Here's what the refactored code looked like:

class ParentJob
  include Sidekiq::Job

  def perform(work_item_ids)
    # Prepare all job arguments
    job_args = work_item_ids.map { |id| [id] }
    
    ChildJob.perform_bulk(job_args)
  end
end

The key difference: perform_bulk pushes the jobs to Redis in a single pipelined operation rather than individual Redis calls. This dramatically reduces the network overhead that was causing our bottleneck.

The Results

The impact was immediate and dramatic. Parent job execution times dropped to just a few seconds, even for large batches. The 99th percentile went from 40 seconds down to under 5 seconds.

This shows the results of our incremental optimizations:

More importantly, the job now always finishes gracefully during a Sidekiq-initiated shutdown. No more interrupted enqueuing, no more orphaned work items, no more race conditions.

The overall time for job processing was reduced significantly, allowing for more efficient use of the cluster:

Lessons Learned

Measure first, optimize second: Premature optimization is still the root of at least some evil. Our goal here was to run the task under 20 seconds so that it would not get interrupted by sidekiq. If our first optimization got us there, we would not need to use bulk_perform. And bulk_perform is not a slam dunk. Since all the arguments for the jobs are marshaled at once, it can overwhelm your redis db if it is running high on memory already.
Deep dive when the situation demands it: bulk_perform has been in Sidekiq for years, but I'd never needed it until this specific use case pushed me to look deeper. Where else might we improve silent in-efficiencies with this technique? Time will tell.
Network calls are expensive: The difference between 1,000 individual Redis calls and one pipelined bulk operation was the difference between 40 seconds and 3 seconds.
Graceful shutdowns matter: Taking the time to handle shutdowns properly means deployments are smoother and data integrity is maintained.

Conclusion

What started as a critical bug during deployments became an opportunity to understand Sidekiq's internals more deeply. The journey from "jobs getting killed" to "graceful shutdowns every time" involved measuring performance, understanding bottlenecks, and discovering the right tool for the job.

If you're enqueuing large numbers of child jobs from a parent job, bulk_perform may just be the ticket.

Death-defying sidekiq jobs

2025-11-13T15:10:00.000-08:00

As promised in my earlier post, I'm thrilled to announce that the changes to prevent Sidekiq job termination have been successfully deployed, and the results look promising!

But before I get ahead of myself, let me break down the problem again. (If you haven't read the previous posts, you might want to check them out for context.)

The Problem

We have a parent job that spawns child jobs for mileage calculation for each user
The parent job runs longer than 30 seconds and occasionally gets killed by Sidekiq
Why does this happen? Sidekiq restarts every time we deploy new code (several times a day—we are a startup, after all!). Auto-scaling rules on the cluster can also reboot Sidekiq
Generally, this parent job is idempotent when interrupted during the time series iteration (where 99% of the time is spent), so it doesn't usually cause data corruption—just an annoying inefficiency
In the unlucky 1% of cases, we could spawn two jobs for each user, causing each to compute mileage independently and doubling the count
We can't handle concurrent invocations (which happen at the end of an outage) because it's hard to differentiate between a scheduled invocation and one triggered by a service restart

The Solution (Deployed Methodically)

First, I tackled these steps:

Deployed metrics to track how long the parent job takes. We now have over a day's worth of data. Notice it takes way longer than 30 seconds—if our new approach succeeds, this graph should flatten out in the coming days
Deployed code that builds a parallel data structure to hold driver IDs
Tested to ensure both the old and new approaches return the same set of users

Testing Challenges

Step #3 proved harder than expected. Testing against a live system means the numbers never match exactly. I wrote code to examine the differences, built a hypothesis about why/how the numbers would differ, and tested it against the data.

users = []
GeoTimeseries.iterate do |user_id|
  users << user_id if GeoTimeseries.recently_driven?(user_id) 
end
orig_set = Set.new(users)

current_time = Time.current
new_set = 
  6.times.reduce(Set.new) do |user_ids, i|
    bucket_time = current_time - (i * Geo::LastHourSink::BUCKET_DURATION)
    bucket_key = Geo::LastHourSink.bucket_key_for(bucket_time)
    members = $redis_aws.smembers(bucket_key).map(&:to_i)
    user_ids.merge(members)
  end

Analyzing the Differences

To understand the discrepancies:

• orig_set - new_set shows users our new technique missed
• new_set - orig_set shows users who appear with the new technique but were absent before

Users We Missed (orig_set - new_set)

Spot-checking the last timestamp of several users showed they'd last driven slightly over an hour ago. This makes sense—our new technique runs about a minute after the time series iteration, by which point we'd already expired some early drivers.

Running the time delta across the complete set revealed two patterns:

Users who stopped driving slightly before the 1-hour mark
Users who started driving a few seconds ago

I hypothesized that users who hadn't driven for the past hour must have just started driving. If correct, these users should now be present in our new data structure—which I validated.

New Drivers (new_set - orig_set)

Everyone in this set had just started driving, so it made sense we missed them during the iteration that happened a minute earlier. (This screenshot shows -- second column --how long they have been driving and they are mostly under 60 seconds )

With these validations complete, I'm confident in the new approach. Stay tuned for follow-up metrics showing the flattened execution times!

When Your Fix Becomes the Problem: A Tale of AWS Outages, Redis Flags, and Performance Scaling

2025-11-07T22:02:00.000-08:00

The Original Problem: AWS Outage Chaos

During the recent Oct 20th 2025 AWS outage, our team discovered an uncomfortable truth about our scheduled jobs. We had jobs configured to run exactly once per schedule via AWS EventBridge Scheduler. Simple enough, right?

Wrong.

When AWS came back online after an extended outage, EventBridge released a flood of queued job triggers that had accumulated during the downtime. Our "run once" job suddenly ran multiple times in rapid succession, causing data inconsistencies and duplicate operations. Check out my previous post on how we recovered user data after the outage here.

The Solution That Worked (Too Well)

The fix seemed straightforward: implement a Redis-based distributed lock to prevent concurrent executions. Before each job execution, we'd set a flag in Redis. If the flag was already set, the job would recognize a concurrent execution was in progress and gracefully bail out.

def perform
  return if concurrent_execution_detected?
  
  set_execution_flag
  
  begin
    # Iterate over driver TimeSeries data to find active drivers
    process_active_drivers
  ensure
    clear_execution_flag
  end
end

def concurrent_execution_detected?
  !REDIS.set("job:#{job_id}:running", "1", nx: true, ex: 300)
end

We deployed this with confidence. Problem solved!

The Problem With the Solution

Except... it wasn't.

Shortly after deployment, we noticed something odd: some scheduled slots had no job execution at all. The job simply didn't run when it was supposed to. This was arguably worse than running multiple times—at least duplicate runs were noisy and obvious.

The Real Culprit: Death by a Thousand Drivers

After digging through logs and tracing job lifecycles, we found the smoking gun: Sidekiq's graceful shutdown mechanism combined with our job's growing execution time.

Here's what was happening:

A scheduled job starts executing
The job iterates over TimeSeries data for all our drivers' geospatial data
Kubernetes scales down our Sidekiq cluster (or a pod gets replaced during deployment)
Sidekiq begins its graceful shutdown, giving jobs 30 seconds to complete
Our job takes longer than 30 seconds (sometimes over a minute!)
Sidekiq hard-kills the job
The Redis flag remains set (because the ensure block never runs)
Sidekiq automatically retries the job on another worker
The retry sees the Redis flag and thinks "concurrent execution detected!"
The retry bails out immediately
No job completes for that scheduled slot

The Hidden Performance Regression

What made this particularly insidious was that our job used to be fast. When we first launched, iterating through driver TimeSeries data took milliseconds. But as our traffic surged and our driver count grew, the Redis keyspace for the TimeSeries structure expanded significantly.

What was once a quick scan became a slow crawl through thousands of driver records, filtering for those who had driven in the last hour. We only actually needed the geospatial data from the last 10 minutes, but we were scanning everything.

The job had slowly, imperceptibly degraded from sub-second execution to over a minute—crossing that critical 30-second Sidekiq shutdown threshold.

The Real Fix: Performance First, Then Locking

We realized the Redis lock wasn't wrong—it was just unable to work correctly with a slow job. The real problem was that we couldn't distinguish between two scenarios:

Truly concurrent jobs (from AWS outage flooding) → Should be blocked
Retry after Sidekiq kill (legitimate recovery) → Should proceed

When the job took 60+ seconds, Sidekiq would kill it and spawn a retry. But the Redis lock was still held, so the retry would see it as a concurrent execution and bail out. The lock was working as designed; the job was just too slow to survive Sidekiq's shutdown process.

The solution wasn't to remove the lock—we still need it to handle AWS outage scenarios. The solution was to make the job fast enough that it would never be killed mid-execution.

The Performance Bottleneck

Our original implementation looked something like this:

def process_active_drivers
  all_drivers = Driver.all
  
  all_drivers.each do |driver|
    # Fetch and scan the entire TimeSeries for this driver
    timeseries = REDIS.zrange("driver:#{driver.id}:locations", 0, -1)
    
    # Filter for entries from the last hour
    recent_locations = timeseries.select do |entry|
      entry.timestamp > 1.hour.ago
    end
    
    # We only needed the last 10 minutes anyway!
    process_recent_activity(recent_locations.select { |e| e.timestamp > 10.minutes.ago })
  end
end

This meant:

Fetching potentially thousands of driver records from the database
For each driver, pulling their entire geospatial TimeSeries from Redis
Filtering in Ruby to find recent activity
All to find maybe a few dozen drivers who were actually active

The Solution: An Active Driver Index

Instead of scanning all drivers and their complete history, we built a lightweight index structure in Redis that tracked only the drivers who had been active in the last hour:

# When a driver's location is recorded (happens frequently)
def record_driver_location(driver_id, location_data)
  # Store in the main TimeSeries (as before)
  REDIS.zadd("driver:#{driver.id}:locations", timestamp, location_data)
  
  # NEW: Add driver to the active drivers set with expiry
  REDIS.zadd("active_drivers", Time.now.to_i, driver_id)
  
  # Clean up entries older than 1 hour
  REDIS.zremrangebyscore("active_drivers", 0, 1.hour.ago.to_i)
end

Now our scheduled job became:

def process_active_drivers
  # Get only drivers active in the last hour (sorted set query is O(log N))
  cutoff = 1.hour.ago.to_i
  active_driver_ids = REDIS.zrangebyscore("active_drivers", cutoff, "+inf")
  
  # Only fetch the data we need
  active_driver_ids.each do |driver_id|
    # Get just the last 10 minutes of data using ZRANGEBYSCORE
    recent_locations = REDIS.zrangebyscore(
      "driver:#{driver_id}:locations",
      10.minutes.ago.to_i,
      "+inf"
    )
    
    process_recent_activity(recent_locations)
  end
end

The Expected Results

Based on benchmarking, the performance improvement should be dramatic:

Before: 60+ seconds (and growing with scale)
After: <1 second consistently (in testing)

By maintaining a parallel index of active drivers, we:

Eliminated the need to scan all drivers
Eliminated the need to fetch and filter complete TimeSeries data
Reduced the job from O(N×M) to O(A×K) where A is active drivers (tiny compared to N) and K is recent locations per driver

If the benchmarks hold in production, with the job completing in under a second:

Sidekiq's 30-second shutdown window will no longer be a concern
The Redis lock will finally work as intended—preventing duplicate jobs from AWS outages without blocking legitimate retries
We can distinguish between truly concurrent jobs (which should be blocked) and retry jobs (which should proceed)

Update: I'll be deploying this solution soon and will follow up with a part 2 covering the actual production results and any surprises we encounter along the way.

Lessons Learned

Performance problems masquerade as concurrency problems - Our Redis lock was correct, but it couldn't work with a job that took longer than Sidekiq's shutdown window. We couldn't distinguish between "truly concurrent" and "legitimate retry."
What works at 10x doesn't work at 100x - Our original implementation was fine for dozens of drivers. With thousands, it became a bottleneck that made our concurrency control unworkable.
Maintain the right indices - Scanning complete datasets to find recent activity is a code smell. Build lightweight indices that track what you actually need.
Use Redis data structures wisely - Sorted sets (ZSET) with time-based scores are perfect for "recently active" tracking with automatic time-based filtering.
Measure, don't assume - We didn't notice the job slowing down because it happened gradually. Better monitoring would have caught this before it became critical.
Fix root causes, not symptoms - The Redis lock wasn't the problem—it was exactly what we needed for AWS outages. The problem was the job being too slow to work with the lock correctly.

The Architecture Pattern

This pattern of maintaining a "recently active" index alongside your main data structure is broadly applicable:

# Pattern: Active Entity Index
# Main data: Complete history for each entity
# Index: Set of entities active in time window

class ActivityTracker
  def record_activity(entity_id, data)
    timestamp = Time.now.to_i
    
    # Store complete data
    REDIS.zadd("#{entity_type}:#{entity_id}:history", timestamp, data)
    
    # Update active index
    REDIS.zadd("active_#{entity_type}", timestamp, entity_id)
    
    # Periodic cleanup (or use Redis expiry)
    cleanup_old_entries if rand < 0.01
  end
  
  def get_recently_active(time_window = 1.hour)
    cutoff = time_window.ago.to_i
    REDIS.zrangebyscore("active_#{entity_type}", cutoff, "+inf")
  end
end

This trades a small amount of additional write overhead for massive read performance gains when you need to find "what's active right now?"

Conclusion

Distributed systems problems often look like they require coordination primitives when they really require performance optimization. Our Redis lock was the right solution for preventing duplicate jobs during AWS outages—but it could only work correctly once the job was fast enough to complete before Sidekiq's shutdown timeout.

The key insight: you can't distinguish between concurrent execution and legitimate retry if your job doesn't finish before the system kills it. By making the job 60× faster, we enabled our concurrency control to work as designed.

Sometimes the best fix for a distributed systems problem isn't better coordination—it's making operations fast enough that edge cases become rare and recoverable.

When AWS Went Down, Our Users Didn’t Lose Their Miles

2025-10-26T15:43:00.000-07:00

On Oct 20, 2025 UTC, AWS experienced a significant regional service disruption that affected several of our core components — specifically the EventBridge scheduling layer that powers our mileage pipeline.

For several hours, our data ingestion flow couldn’t persist new trip events from the Redis TimeSeries. This temporarily paused mileage calculations, leading to incorrect user summaries in the app.

But here’s what didn’t happen:
We didn’t lose a single record, and no user lost a mile.

How Our System Works

Our pipeline captures anonymized telemetry events from users’ mobile devices, processes them through AWS infrastructure, and stores aggregated trip summaries in a relational database.

Data Flow: Mobile App → Ingestion API → Redis TimeSeries → EventBridge Scheduler → RDS

Event flow:

Mobile App captures GPS telemetry and trip start/stop events.
Ingestion API authenticates and sanitizes the data before writing to Redis TimeSeries.
Redis TimeSeries stores short-term data points with fine-grained timestamps for quick replay (a background job backs up the stream to S3 for longer term storage).
EventBridge Scheduler triggers aggregation and processing jobs every few minutes.
RDS stores validated, aggregated trip records for long-term analytics and reporting.

What Happened During the Outage

When AWS degraded services in our primary region (us-east-1), the EventBridge Scheduler stopped firing events from 6:50 - 10:07 UTC halting our pipeline smack in the middle. While we were still capturing and ingesting the user geo data, the Redis Timeseries was not being processed as this job did not get scheduled for little over 3 hours.

Remarkably, the Redis Timeseries held up. Our Redis cloud is with Redis Enterprise, but the instances are hosted in the AWS cloud. Even though Redis Enterprise noted that some customers would be impacted, we did not see a significant degradation.

Understanding the AWS architecture helps explain why this was the case.

AWS separates each service into a control plane and a data plane. The control plane is responsible for resource monitoring, allocation, scaling whereas once resources have been allocated, the data plane takes over - the data plane is more reliable/resilient while the control plane is known to fail.

Here from the horses's mouth:

Since we provision our Redis cluster for expected usage, manually adding more nodes/memory as our volumes increase, we were not relying on the AWS control plane for scaling - our instances continued to hum (we saw the same with RDS as well -- again, it is customary to provision the RDS for your needs and perform upgrades manually as traffic increases)

This was not the case for our web/job servers, that were configured with Auto scaling rules. We had set a lower limit on the number of machines for each cluster, and we were running hard on this reduced capacity until recovery.

When services recovered, we started processing events from the Timeseries, creating trips for users. But since we generate incremental trips for the last few minutes, we were still missing trips for the last 3 hours and 7 minutes.

We could easily tell how many trips we missed as we track this closely using a Cloud Watch metric. Each bar shows a completion of the job that is responsible for incrementally processing the timeseries.

When services recovered, EventBridge Scheduler fired all the events in the backlog.

This caused a different problem as our trips processor was designed to handle the time series data in real time. We did not anticipate serving more than a single event during a ten minute window. So we got 21 late-fired triggers but effectively could process just one, for the last ten minutes. More on this later!

The critical task was to update the user data for the missing three hours. I had written a script to patch trips that I had used earlier for a less severe outage (5 minutes). With some minor modification to account for the partial data towards the tail, I was able to correct mileage for all our users who happened to be driving during the outage (luckily they were not on self-driving cars powered by AWS. Ok - bad joke)

There was still something I couldn't explain - CW told me ~ 2,000 jobs completed after jobs started flowing again. I expected 21 jobs, but I was puzzled by the much larger volume that ran at the tail. What amounted to that, and would they cause a different type of mis-calculation? Indeed, some interesting things did take place with those 21 EventBridge triggers, let me explain.

When a trigger fires on the tenth minute, we launch a job per user who have likely been driving recently. These jobs run concurrently, and we need to track the completion of the last job to know that all users have been processed and the window can be marked "complete".

This is done with a Redis Set that keeps track of users who are still being processed. So when the trigger fires, it first determines all recent drivers, adds them to the set, before spawning a worker per user. Then each worker removes an element from the set, and if it is the last item, notifies the completion of the run.

When 21 triggers fired in rapid succession, they all forked a job per user, resulting in many workers racing to compute the same job, and may workers hitting an empty set. And of course this meant that these jobs "up counted" miles for the drivers in that time window.

So the last data cleanup was to figure out where we added more miles to users during the end of the outage. I first thought this might be really hard as we had already updated these records, which then kept getting updated further as the users kept driving. But fortunately, we update both start and end times for each trip in the record, so it was possible to compute the miles driven in this specific range for each user from the raw timeseries data.

To verify that we have been up-counting, I queried for records in descending order of speed. And I saw speeds of 600 mph, which confirmed the hypothesis quite fast [no pun intended]

I could re-use a method from the earlier script for patching the data, and write a bit of code for the update. So finally, after a very long day, our users' data was fully corrected.

Improvements made:

We are making improvements on how we handle the tail of an outage going forward. The idea here is to not let more than a single processor to run in a given time window. This can be done with a simple Redis SET command with option "NX", which sets a flag (lock) if it is not already set, thus guaranteeing that only a single process can acquire the lock. We set the TTL to be below the time window (7 minutes in this case) so that the lock naturally expires before the next trigger.

Our Approach: Fairness and Fidelity

Our principle is simple: if you drove it, we count it.

We don’t approximate, and we don’t drop data due to transient infrastructure issues. Each pipeline component is designed for eventual consistency and idempotent replay, so every record can be reconstructed safely and accurately.

What We’re Building Next

Resilience isn’t just uptime — it’s graceful recovery. We’re implementing several next steps to strengthen this architecture:

Buffer device data: As users go through areas with low mobile reception, we want to buffer the location data and deliver it when reception improves.
Adjust inaccurate device signal : Use techniques like Kanman filtering to adjust the location for high fidelity, when the device accuracy is low
De-couple RDS for real time updates: We will store running trips in Redis, with archiving to take place later. This makes us resilient on an event when the RDS is un-responsive, as we only need it at the latter step of archiving
Monitoring for anomalies: Add speed as a tracked metric and alert over 200 mph.
Chaos testing & fault injection: Monthly simulated outages to validate that our recovery flow remains reliable.

What It Means for Our Users

When something breaks in the cloud, we don’t panic — we verify, replay, and reconcile.
Because behind every data point is a real person trusting us to get it right.

Outages happen, but trust shouldn’t. And that’s what we’re building for.

Posted by [Thushara Wijeratna], [Head of Eng] at [WorkSolo]

Rails 8: - can't cast RSpec::Mocks::Double

2025-10-16T15:39:00.000-07:00

One of the first unit test failures I encountered on a Rails 8.0 upgrade was:

can't cast RSpec::Mocks::Double

The error happens on saving an ActiveRecord object to the database.

#  connected                   :boolean          default(FALSE)
#  last_connected_at           :datetime

class LymoAccount < ApplicationRecord

  after_commit :update_last_connected_at, if: :saved_change_to_connected?

  def update_last_connected_at
    update!(last_connected_at: Time.current) if connected?
  end

Turns out, this is related to stricter validation in Active Record, that refuses to save a mock object.

Generally speaking, you should be using FactoryBot methods to create real ActiveRecord objects from your unit tests. And we were. So it puzzled me why we would get this error, as it did not seem like we were storing anything that was a mock.

ChatGPT got pretty confused as well -- It got confused as the exception was thrown from an after_commit hook and its assumption was that there were attributes already set in the model that are being re-serialized and this was causing the issue.

We went through a listing of all the attributes of the record, examining their type (class) and none of them was a mock.

This was the point when I gave up on Gen AI and took another look at the model.

I quickly eliminated that this has anything to do with the connected attribute that we are checking, by updating the test to save without the check. It didn't help, so I knew that the update itself was throwing.

Then I wondered if updating any column would throw the same error.

I updated a text column, and it worked - progress. Now I knew this might have something to do with either:

1. the specific column - maybe it has some validation that is behaving weird

2. the column type - maybe datetime columns behave differently with updates

Luckily there were a few other datetime columns, and when I updated any one of them with Time.current, they threw the same error. We were getting somewhere.

On a hunch, that this was to do with Time.current , I took a look at the unit test, to see if Time was being mocked - normally, we use Timecop as the best practice, but it was always possible that an engineer decided to be "creative".

This is what I found on the unit test:


  let(:data_sync_at) { Time.current.to_i }
  before do
    allow(Time).to receive_message_chain(:current, :to_i).and_return(data_sync_at)
  end

So what this does is stub the intermediate Time.current call, and it is not that obvious! So when we used Time.current to update the record, it tried to use a Mock object and failed.

A simple Timecop.travel(data_sync_at) fixed the issue.

Here is the Claude transcript, which happened later, when I was better able to guide the conversation along a more productive route!

Debugging Rails RSpec Error: RSpec::Mocks::Double Casting Issue

Q: On a Rails model, why would this code throw an error on a unit test saying: can't cast RSpec::Mocks::Double?

def update_last_connected_at
    update!(last_connected_at: Time.current) if connected?
end

Answer: The error occurs because connected? is returning an RSpec mock double instead of a boolean value, and Rails is trying to cast it to a boolean in the if conditional.

Common causes:

Unmocked association — connected? might depend on an association that's mocked but not configured properly
Missing stub — The connected? method itself might be mocked but not given a return value

Solutions:

1. Stub the method explicitly with a boolean:

allow(model).to receive(:connected?).and_return(true)

2. Build a proper test object instead of relying on mocks:

let(:model) { build(:your_model, connected: true) }

Q: Happens without connected?

Answer: If you're getting the error without even calling connected?, the problem is likely that update! itself is being called on a mock.

Most common cause: You're probably testing this method on a mocked or partially mocked model instance. When you stub the model without fully setting it up, calling update! on it returns a double instead of actually updating the record.

Solutions:

1. Use create or build instead of double:

let(:model) { create(:your_model) }
model.update_last_connected_at

2. If you must use a double, stub update!:

let(:model) { double(:your_model) }
allow(model).to receive(:update!).and_return(true)

Q: Model is not stubbed, it is a FactoryBot created one

Answer: If it's a real FactoryBot instance, then the issue is different. The error suggests that update! is being called but something in the conditional logic is going wrong.

Most likely causes:

connected? method doesn't exist or returns a mock
The model instance itself is getting wrapped in a double
Database transaction/rollback issues

Debug steps: Add some debugging to see what's actually happening:

def update_last_connected_at
    puts "connected? returns: #{connected?.inspect}"
    puts "connected?.class: #{connected?.class}"
    update!(last_connected_at: Time.current) if connected?
end

Q: Connected is a column in the table (model). Here's my test and model:

let(:data_sync_at) { Time.current.to_i }
allow(Time).to receive_message_chain(:current, :to_i).and_return(data_sync_at)
// call method that ends up saving the record

# Model:
after_commit :update_last_connected_at

def update_last_connected_at     
  update!(last_connected_at: Time.current)   
end

Answer: Ah, there's the issue! You have an infinite loop of callbacks:

update! is called in update_last_connected_at
update! triggers after_commit
after_commit calls update_last_connected_at again
Which calls update! again... and so on

This creates a recursive callback that RSpec is having trouble with, and the mock you set up on Time.current might be interfering.

The real problem: You're missing the connected? check that should prevent this recursion:

def update_last_connected_at
    update!(last_connected_at: Time.current) if connected?
end

Or use update_column instead (skips callbacks):

def update_last_connected_at
    update_column(:last_connected_at, Time.current) if connected?
end

Q: My after_commit is like this so should not recurse:

after_commit :update_last_connected_at, if: :saved_change_to_connected?

Answer: Good catch! With that guard clause, it shouldn't recurse. But there's still an issue:

The problem: When update_last_connected_at calls update!, it triggers another after_commit. Even though the guard if: :saved_change_to_connected? prevents it from running again, RSpec is still having issues.

The real culprit is likely your Time mock:

allow(Time).to receive_message_chain(:current, :to_i).and_return(data_sync_at)

This is mocking Time.current in a way that might be interfering with the after_commit callback execution in your test. When you call update!(last_connected_at: Time.current) inside the callback, the mocked Time.current might be returning a double.

Try this instead:

Option 1: Mock just the .to_i part, not the whole chain:

allow_any_instance_of(Time).to receive(:to_i).and_return(data_sync_at)

Option 2: Use travel_to:

travel_to(Time.at(data_sync_at)) do
  # your test code
end

Option 3: Simplify by not mocking Time at all:

let(:data_sync_at) { Time.current.to_i }
# Just use the actual Time.current in your model

Real time location of drivers : a tale of repurposing a Jupyter Notebook

2025-05-17T07:04:00.000-07:00

At Solo, we automatically track driver location to calculate mileage—making it easy for our users to deduct travel expenses when filing taxes with the IRS. But we go far beyond basic mileage tracking. Our app breaks each day into individual “trips,” so drivers can see their full driving route in detail. Wondering where you lost the most time in traffic? Trying to remember that tricky apartment complex with no parking? Or the day you circled a golf course for 30 minutes? We capture all of it—and turn those moments into insights, delivered through an interactive map that helps you drive smarter.

To visualize driving routes, we use OpenStreetMap rendered through a Jupyter Notebook, all powered by a lightweight Flask server. The Flask server handles two core tasks:

Given a list of [latitude, longitude] coordinates, it plots the route on interactive map tiles.
It animates the route by syncing movement with timestamps associated with each coordinate pair.

We chose OpenStreetMap over Google Maps for a few key reasons that make it especially startup-friendly:

Cost-effective: OpenStreetMap is significantly more affordable than Google Maps, with no steep API pricing.
Highly customizable: From tile colors and custom markers to layer controls, the map styling is incredibly flexible.
Frequently updated: The map data is refreshed several times a day, ensuring accuracy and relevance.

On the backend, our Flask server handles dynamic map rendering. The render_map() function below takes in location data, timestamps, and speeds, then visualizes the route using Folium and branca—a powerful combo for interactive mapping in Python.

Here's how it works:

If a transition_duration is set, the function animates the trip using TimestampedGeoJson, syncing movement with time.
If no animation is requested, it renders a color-coded route based on speed, using folium.ColorLine.

  def render_map(locations, epochs, speeds, transition_duration):
    if transition_duration:
        print("animating the path!")
        return TimestampedGeoJson(
            {
                "type": "FeatureCollection",
                "features": [
                    {
                        "type": "Feature",
                        "geometry": {
                            "type": "LineString",
                            "coordinates": [
                                [point[1], point[0]] for point in locations
                            ],
                        },
                        "properties": {"times": epochs},
                    }
                ],
            },
            period="PT1M",
            transition_time=int(transition_duration),
        )
    else:
        colormap = branca.colormap.StepColormap(
            ["black", "#DB3A2D", "#003057", "#00BE6E"],
            vmin=0,
            vmax=120,
            index=[0, 1, 5, 30, 1000],
            caption="Speed (mph)",
        )
        return folium.ColorLine(
            positions=locations,
            colormap=colormap,
            weight=4,
            colors=speeds,
        )

This is how an animation looks like:

I didn’t begin building production features in Jupyter notebooks. In fact, this all started three years ago with a much simpler goal: to understand our traffic patterns. As we expanded the product into new cities—and into different segments within larger metro areas—we needed answers. Where is traffic volume increasing? Where are drivers earning more in tips? These kinds of questions required a flexible geospatial analytics setup. Jupyter notebooks turned out to be the perfect environment to explore this growing volume of location-based data.

This is a rough look at our early days as we launched in Seattle:

That early exploration eventually evolved into a lightweight geospatial analytics pipeline—one that could handle real driver data at scale. Using Jupyter notebooks gave us the flexibility to prototype quickly, visualize patterns, and iterate. But as the insights proved useful, we started formalizing parts of that workflow. What began as an experiment matured into a production-grade service: powered by a Flask backend, drawing from location check-ins, and rendering driver routes with OpenStreetMap tiles—all orchestrated from within the same notebook-driven environment we started with.

This is exactly what makes working at a startup so much fun. At a smaller scale, we can take something like a Jupyter notebook—a tool meant for exploration—and ship a real feature to users through the mobile app. I know some of you engineers at Amazon or Meta might be shaking your heads, but that’s the beauty of it: tools that would never even be considered in a big-company tech stack become fair game at a startup. And sometimes, that unconventional choice turns out to be the right one.

This is the route of a driver that is plotted using folium on a React-native web-view:

And what happens when we do have millions of eyeballs on these maps? That’s not a crisis—it’s an opportunity. There are several clear paths to optimize for lower latency and scalability (hello, Leaflet and tile caching). But the key difference is this: we’ll be solving a real, validated need—not one we only thought users might have. That’s the advantage of moving fast at a startup. We don’t prematurely optimize—we ship, we listen, and we scale when it actually matters.

Bringing it back to maps—our rendering is handled by the Folium library, running within a Python Flask server. What’s nice about Folium is that it provides the same visual output in a Jupyter notebook as it does in production. This lets us prototype and test the map layout directly in the notebook before moving the code over to a Flask endpoint.

Here’s how it works: a web server sends the Flask server a list of GPS points to plot. The Flask server then renders the route using Folium and returns the HTML map back to the web server, which in turn passes it along to the mobile app.

For individual routes, this approach works surprisingly well. It’s not the fastest setup—since the full map is rendered server-side and sent to the client—but for shorter routes, the latency is acceptable and the experience is smooth enough.

Eventually, we wanted to display a real-time map of all our active drivers on a flat panel in our Seattle office. The simplest (and fastest) way to do that? Leverage the same system we’d already built.

So I added a new endpoint to the Flask server—one that accepts a list of GPS points and renders a small icon at each location. Different driver events are visualized using different icons. For example: a driver's current location appears as a yellow circle with a red outline; a new sign-up shows up as a gold star; and when a driver swipes a Solo cash card, a dollar sign icon pops into view.

Well, that was the easy part. The real challenge was figuring out how to track all these events in real time so we could continuously update the map every few minutes.

To manage this, I used several Redis sorted sets, grouped by event type:

EVENT_DRIVING
EVENT_SIGNUP
EVENT_SWIPE

Each set holds user IDs as members, with their current <latitude, longitude> stored using Redis' GEOADD command. These sets guarantee that each user has only one location entry, so as we receive location updates, we simply overwrite the previous value—giving us the user's most recent location at any given time.

But there's a catch: if a driver stops moving or goes offline, their entry becomes stale. Redis does support TTLs (time-to-live), but it doesn't allow expiring individual members of a set. So I had to get creative.

To work around this, I store a separate key for each active user using a naming pattern like LIVE_EVENTS_<user_id>, and assign a 5-minute TTL to each. Then, every 10 minutes, I scan through the geo sets and prune out any user IDs that no longer have a corresponding LIVE_EVENTS_* key—effectively cleaning up stale locations.

And that map you see at the top of this post? It was built exactly this way—stitched together from Redis geo sets, rendered by a Flask server, and piped straight from a Jupyter notebook prototype into production.

Postgres indices on large tables : gotchas

2024-03-13T14:55:00.000-07:00

When you run a migration to create an index on Postgres, to allow queries to run, we create the index using the `CONCURRENTLY` flag.

But if the migration fails for any reason, the index will be created but unusable. Say, you were adding a UNIQUE index, and the migration fails since it encounters a duplicate. So the migration fails with something like:

Now imagine that your migration needed to make an existing index UNIQUE. It is usually done by first removing the index, and adding it back with the UNIQUE constraint.

Since the migration is aborted at the point of creating the new index with the constraint, now there is no index. The app will be running without an index.

If you were to check the db, you might get confused as the index *seems* to be there - it is just not operable.

To verify, run an `EXPLAIN` and you will see it does not use the index.

Now the queries that regularly use the index will be running quite slowly, as the index is not quite ready. This could add a lot of pressure to the db and make the app pretty much unusable / inoperable.

Ruby lazy collections and with_index

2024-01-10T14:39:00.000-08:00

The lazy collections don't quite give us a way to use `with_index` , but, we can use `each_with_index`.

It took me a moment to figure out the problem was with the collection being lazy.

Using Redis for web request counting

2022-08-25T11:33:00.001-07:00

Problem Statement

With the growth of our user base and increasing traffic to the web servers, we wanted to come up with some realtime counters to measure traffic.

We wanted the system to give us an idea of:

The total web requests the site is receiving each minute
These totals aggregated hourly and displayed for the day (24 entries)
The counts broken up by the specific request path
The number of unique visitors broken up by the minute, hour and day

Limitations of available solutions

We use Heroku for deploying the app. The Heroku dashboard provides us a rough idea of the request counts, but it is difficult to incorporate that into our own operational dashboards. Also Heroku doesn't break down the counts by the request path, which was important to us in understanding the source of traffic and how we could scale resources based on the source. Unique visitor counts are similarly an important metric not readily available.

Counters help scale the app

Breaking down the traffic by the source was important in coming up with a way to efficiently scale our service. Currently most of our application sits in a single service. So scaling the app means we add machines to this single fleet, even though only requests from one or two sources really benefit from this added capacity.

We have 3 main sources of traffic:

Our users hit the service for various features provided from their mobile app
Our users send us their geo location from the mobile app
We receive near real-time data of our users gig work from a third party

Our primary application performance goal is providing a sub second experience to users as they use our mobile app; thus mainly, we want to optimize resourcing on the backend with a focus on 1.

However, we get so much more traffic from 2. and 3. which consume most of the server bandwidth. Keeping all three services as a single service degrades the experience for the user.

Mind you, 2. and 3. do not have a real time processing requirement. While a web server is needed to accept these requests, the actual processing is handled by an asynchronous worker outside of the web server.

But still, since there are so many of these web requests, for the few milliseconds each of them sit on the web server, it takes away bandwidth from the user requests.

Why Redis?

Redis provides the ideal data structures for counting requests with minimal overhead. (And must I say that it is fast, as in really, really fast) A few keys can be used to keep a running total for each minute per request type, then a sorted set can be used to aggregate the last N minutes of requests. (For our purposes, we decided to keep 60 counts, thus providing a picture of activity for the last hour, but you can choose to keep counts for longer than that.)

The same idea can be extended to measure days worth of traffic broken by the hour.

Choice of the Sorted Set

Why did we decide on the sorted set for aggregating the counters? Well, the sorted set allows us to have the counters sorted by time. This way, we can quickly get the list of counts for the last hour ordered from minute 1 down. To be fair, it is a bit overkill to use a set for this, as we are never going to be mutating an element (since the timestamp is always increasing), but it does suit our purposes just fine!

Before going any further, let us briefly recap the salient features of the sorted set. It allows us to insert elements along with a score, and the elements are sorted in real time by the score. It scales really well for even millions (or more) of elements as each insert operation takes O(log(n)) time -- much like a binary tree. While we do not need that level of scale, one can think of keeping extremely granular counts for a long period of time, which could come in handy for debugging bizzare performance problems after the fact!

We can use the timestamp as the score. Redis will then always sort the set by the timestamp. This has the advantage that if you wanted to change the counter later (imagine a design where you quickly provide a rough estimate of the count, but later do a second pass to provide exact counts), you can simply add a new count to the set with the same timestamp and the position of the element will not change.

The counters will need to be reset at the start of the minute. I first made the mistake of using the expiry time of the key set to 1 minute, but realized that this introduces a race at the point of aggregating the count on to the sorted set. Which is that we may be unlucky that before the aggregation, redis could have expired the key, resulting in a substantial undercount in the set. (This was a difficult bug to track down, and of course the most obvious, I had a face-palm moment as you can imagine.)

There is a slight difficulty we need to work around here w.r.t the sorted set. If we keep the count as the element in the set, a count that happens to be the same as one already stored will replace the previous count (with the score modified). Since we are using the timestamp as the score, this will essentially remove the entry we had for the earlier timestamp. This is how sets work after all - it is a data structure suited for keeping unique members. But we can easily overcome this by prepending the timestamp of the count to the count and storing that as the element of the set. To read the count, we merely have to split the element by the delimiter (we used the colon here for the delimited, which is somewhat of a standard in redis for splitting keys), and use the second part.

A look at the running counters

A ZRANGE command can retrieve the counts like so:

Counting the unique visitors is only slightly more involved. First we need to keep a regular Redis set and update it for each request. In our case, the user id is encoded in the request header, we decode it and add it to the Redis set. Now if the same user visits the site again, since we are using a set, we will not add it again, and we still have just one element in the set. This way, we can take the length of the set at any point and know how many unique visitors we have from the time we started writing to the set.

The only thing left to do is, create the set at the start of the time interval we need to measure the count, and reset it at the end of the time interval. So we can set this up to reset every minute for a minute by minute count of unique visitors. Then we can use the infrastructure we built above to aggregate the count over to the sorted set, so we have a running count of unique visitors for the past N minutes.

(You may have a different technique for figuring out the ID of the user making the request. Once the ID is extracted, you can use a Redis set to keep track of the visit.)

Here is how we see the unique visitor count changing dynamically:

We can just as easily use another excellent Redis command to see all the user ids in this set. Here is a snippet in our case :

Implementation

We implemented the counters using Ruby, with the redis gem as our client. This involves several steps:

Initializing the counters
Resetting the counters at minute, hour, day intervals
Incrementing the appropriate counters for each request
Aggregating the count onto the set

The first two steps can be combined. We used a scheduler that sits within the app via the ruby clock gem. Heroku allows provisioning a single process that runs a scheduler based on the schedule we set via the ruby clock. This is pretty similar to how one would use cron on a Unix machine to schedule a task.

Heroku does provide a scheduler as well. We did not use it as it does not have the same reliability guarantees as the ruby clock gem. I have seen cases where the Heroku scheduler does not fire and fires very late, all documented behaviors.

Since we use Rails for our app, we utilized its framework built on top of controllers to track request counts.

A controller encapsulates serving requests for a specific web route (think of this as having one controller for yoursite.com/user/login and another controller for yoursite.com/reports/account). Now each controller is a subclass of a class we implement called ApplicationController which itself is a subclass of the Rails class ActionController::Base.

Rails allows us to hook all requests at the ApplicationController level with a simple before_action hook. We implemented the request counting using this hook, and it looks something like this:

class ApplicationController < ActionController::Base before_action :update_usage_counters def update_usage_counters PerfCounters.increment(user_id: current_user&.id, request_path: request&.fullpath) end end

Now each request goes through update_usage_counters, which delegates the work to the PerfCounters class we wrote. request is an object provided by the Rails routing framework, and request.fullpath contains the URL. The method current_user (not shown) extracts the logged in user's ID from the request headers.

I will reproduce pieces of a simplified version of PerfCounters that will illustrate the logic:

The incrementing logic looks like this:

class PerfCounters def self.increment(user_id:, request_path:) $redis.pipelined do |pipeline| if user_id.present? pipeline.incr('USER_REQUESTS_MINUTE') pipeline.sadd('UNIQUE_VISITORS_MINUTE', user_id) if request_path&.include?("/geo/send") pipeline.incr('GEO_REQUESTS_MINUTE') end else pipeline.incr('OTHER_REQUESTS_MINUTE') end end end end

Notice that a request made on behalf of a logged in user will have user_id parameter set. The request_path is the path of the URL, and we use it here to separate the counts made to track the location of the user.

Another neat redis feature we use here is pipelining. The idea is that if we need to make a number of independent requests to redis, we can open a socket to the redis server and send all that data and close the socket at the end. Redis server will return an array of replies in order after it processes all requests. This is a powerful feature that is more efficient than creating a socket for each separate request. It is not without cost - as the server has to buffer each request, technically blocking the request thread from processing other requests. The rule of thumb is to make sure that each request is pretty fast - O(1) would be ideal, and to not pipeline too many requests in a single call. As with everything, you must test this against all the other traffic you serve and compromise if you must!

Also notice that we are demonstrating the use of three counters USER_REQUESTS_MINUTE, GEO_REQUESTS_MINUTE and OTHER_REQUESTS_MINUTE, alongside a set called UNIQUE_VISITORS_MINUTE. This last one actually keeps the user ids of all visitors. The sadd command adds the visitor id to the set upon the first time we see them.

The ruby clock gem takes its inputs via a file named Clockfile. This is in fact a file that uses ruby syntax, i:e it is evaluated by the ruby interpreter. All we do is define the aggregator to run every minute, like so:

schedule.cron '* * * * *' do PerfCounters.aggregate_minute end

This is what the minute aggregation looks like:

def self.aggregate_minute tm_obj = Time.current - 1.minute # aggregate last minute's stats tm = tm_obj.to_i # get all current minute counters, add them to the hour list before zeroing them out user_rpm, other_rpm, geo_rpm, unique_visitors_last_minute = $redis.pipelined do |pipeline| pipeline.get('USER_REQUESTS_MINUTE') pipeline.get('OTHER_REQUESTS_MINUTE') pipeline.get('GEO_REQUESTS_MINUTE') pipeline.scard('UNIQUE_VISITORS_MINUTE') end $redis.pipelined do |pipeline| # ZADD key score value : keep timestamp as score so we get the counters sorted by time # append the timestamp to the counter to make sure entries don't overwrite. pipeline.zadd('USER_REQUESTS_LAST_HOUR', tm, "#{user_rpm}:#{tm}") pipeline.zadd('OTHER_REQUESTS_LAST_HOUR', tm, "#{other_rpm}:#{tm}") pipeline.zadd('GEO_REQUESTS_LAST_HOUR', tm, "#{geo_rpm}:#{tm}") pipeline.zadd('UNIQUE_VISITORS_LAST_HOUR', tm, "#{unique_visitors_last_minute}:#{tm}") pipeline.del('USER_REQUESTS_MINUTE') pipeline.del('OTHER_REQUESTS_MINUTE') pipeline.del('GEO_REQUESTS_MINUTE') pipeline.del('UNIQUE_VISITORS_MINUTE') end end

As you can see, there are two types of counters. One tracks the count for each minute, the other aggregates it for the hour. So for example, take USER_REQUESTS_MINUTE. This is incremented for each request made on behalf of a logged in user. Then upon the dawn of the minute, this counter is added to the sorted set USER_REQUESTS_LAST_HOUR and then immediately deleted.

You can chop the aggregations every Mth hour since otherwise these sets will keep growing eventually taking all of Redis memory! I won't show that code, but it should be fairly straightforward to implement.

After having implemented this solution and writing this article, I have come to see that there are other ways to implement counting using Redis. Redis provides such a versatile set of data structures and algorithms that there is always a simpler or more elegant technique somewhere!

For example, when we use a container like a sorted set or a list, we must set its bounds, clearing it at certain time intervals and thus restricting its memory usage. But if you use the Redis stack, there is an excellent data structure - the Redis Timeseries that does much of this bookkeeping for you. Basically, you can configure the time series to expire old entries (something that most other Redis data structures do not do for you - you can expire the complete key or nothing at all). Besides that it has commands very similar to the set or the sorted set.

Another advantage of a time series vs a sorted set would be the trivially simple management of a "rolling" window of counts. This is typical in performance monitoring that you want the "last 72 hours" or the last "30 days" of performance data, which is more useful than data "for all of today" or "for the current hour".

I leave this as an exercise to the reader. Maybe I can talk about this in greater detail in a future article as well!

This post is in collaboration with Redis.

Learn more:

Try Redis Cloud for free
Watch this video on the benefits of Redis Cloud over other Redis providers

(Embed this video if possible)

Redis Developer Hub - tools, guides, and tutorials about Redis
RedisInsight Desktop GUI

Set GIT Hash on Heroku Deploys

2020-12-18T11:27:00.001-08:00

There are several ways to set the GIT hash for heroku. I prefer setting this just before pushing the main branch to the heroku remote.

Also, I prefer to automate this with a `pre-push` hook.

One issue I ran into was that it was not straightforward to know if there is anything to push. Meaning, if the remote was up-to-date or not. This is important, as we don't want heroku deploying a version when there are no changes to be deployed (which it will if we set the APP_VERSION).

The simplest way to know if the remote was already up-to-date seemed to be to do a `--dry-run` of `git push`.

But of course that runs the hook again, so this is an endless loop situation.

It does not seem that git allows us to see the git argument on the hook. If it did, we could explicitly not run the hook on a dry run.

But there is a `--no-verify` option that bypasses the hooks, and can be used when we do our dry run.

Here is the complete script:

Rails database migrations on a cluster with AWS CodeDeploy

2019-11-30T17:47:00.001-08:00

A typical Ruby / Rails solution will comprise of a number of web servers behind a load balancer. Each of the web servers will read/write from a central database. In the course of new features being added to the application, the database schema goes through changes, what we refer to as "migrations".

When new code is deployed, the migrations that are needed for new code needs to be run first. If AWS Code Deploy is used for deployment of new code, we can set up the AfterInstall hooks to run the migrations before re-starting the web server.

So the usual flow in a deployment goes something like this:

Stop the web server
Migrate the database
Start the web server

However, our application is hosted on a number of web servers. We don't want to bring down all servers at once. A typical blue/green deployment will have us deploy to just one third of the server fleet at once.

So if we have 27 web servers, we will be running the above steps on 9 of them at the same time. The main problem with this is that when the Rails migrate runs on multiple servers at once, it is likely to fail on a number of them. This is because Rails takes an advisory lock on the database and throws an exception on concurrent migrations. You can read more about the advisory locking here as well as a way to work around the problem.

But the solution is not without its drawbacks. If you prevent the migrations running on all but one machine, it is possible that the new code will be deployed sooner on those machines before the migration has finished. This is specially true for long running migrations. Then there is potential for new code to be running against an old database schema. New features that depend on the new schema will likely fail.

A better solution would be:

Run the migration on a single instance - this could be one of the web servers, or a dynamically provisioned EC2 instance that can access the database.
For all web servers

2.1 Stop the server
2.2 Deploy new code to it
2.3 Start the server

The advantage of this solution is

We side-step the concurrent migration issue. We run the migrations on a single instance and then do the rest of the deployment without incurring any database issues.
We bring up the new code only after the database is migrated, so the new features work reliably from the start.

So the new database schema changes need to be backward compatible. But this is a general constraint we have anyway since on a blue/green deployment some part of the code is old and will be hitting the new database.

While this solution is pretty straightforward, it requires some effort to implement this in the AWS CodeDeploy environment.

What I ended up doing was to use a new Deployment Group (called staging) to bring up a single EC2 instance, change the start up code to only run the migration on that deployment group. Then I hooked this deployment group right after the deployment to a test instance, but before the code is deployed to the production servers.

In the startup code, we can check the current deployment group via ENV['DEPLOYMENT_GROUP_NAME']. In our scripts, we set the RAILS_ENV equal to the Deployment Group. This allows code to take different paths based on where it runs (in a local dev environment, a staging server or like in this case on a migrator server).

This is what our migrate script now looks like:

It is important to set the inequality, as we want the migrations to run on our test servers - we just don't want them running on the production web servers.

We add this to our database.yml, notice the environment is staging, to match the deployment group. Notice the database is the production instance.

In our case, we read credentials from AWS Secrets Manager. You don't have to.

This is how our staging step in CodePipleline looks like:

On the last CodeDeploy step, hit the edit button and set the Application Name and the Deployment Group correctly.

Now before the code is deployed to the production servers, the database migration has been completed on the staging instance. If the migration fails, CodeDeploy won't advance to the deploy stage. When the production servers start with new code, they will all use the new database schema.

After the migration has finished and before code is deployed, the old code will start using the new database schema. As long as the new schema is backward compatible, this will not cause a problem.

You may have to run the release pipline a few times till AWS co-operates with the new changes. But it should eventually start working.

Kinesis/Firehose/Athena - Creating queryable data from logs

2019-07-08T23:33:00.003-07:00

Amazon has a data transformation pipeline that allows log data to be queried with a SQL like syntax. This can be useful to gain insight that is buried in log data generally thought of as temporary. When was the last time you went over 6 months of logs? Right, just what I thought.

Wading through logs is painful and with the growth of data all the more so. No wonder that when confronted with the task of gleaning information from past data, engineers build specialized table structures with relational queries in mind or provision specialized map/reduce jobs to crunch over the data for detailed answers to specific questions.

But this time consuming exercise can be done away with by using the Amazon Kinesis pipeline. The flow looks something like this -> The application writes a JSON formatted record that captures a particular item of interest to a Kinesis data stream. A Firehose instance is attached to the output of the data stream. This Firehose instance converts the data to a "JSON like" format and writes them into a S3 bucket at a specified folder. Another Amazon service Glue provides a crawler that can then process new files that get uploaded to the S3 bucket. The Glue crawler infers the schema from the JSON it finds in the S3 files and creates a Glue table. To query the data, Amazon provides yet another service - Athena, which sports a SQL syntax and a user friendly query console. Phew, yeah, it is quite the mother of all pipelines.

This is all pretty straightforward to set up starting from Kinesis console itself. You should start with the Data streams tab in Kinesis, create a data stream, then create a Kinesis Firehose with source equal to the data stream you just created. Specify that firehose data will be written with the API like so:

Since we are writing JSON to Kinesis, there is no need to convert record format, and we will use the data as is without transformation to Firehose, well more on this later, but we can leave the default settings for Source record transformation and Record format conversion.

Finally you need to specify where you want this data to live, in our case S3:

Now head over to Glue and add a crawler. Specify the S3 folder that you used above for the "include path". For simplicity, I would start with a single schema for all S3 records, under "Grouping Behaviors".

Now head over to your favorite editor and let's write some code - finally!

It's up to you how you want to structure the code to do this. In the application I'm building, it is literally the logs that I want sent over to Kinesis. Anticipating this, I wrote a function that the app calls for writing logs, and this function was the ideal place to add in the write to Kinesis. It looks something like this:

That will be all to it, except there is an annoying bug in the pipeline that we need to work around. The issue is that Firehose writes "JSON like" data to S3 that is all a single line. The Glue crawler expects each record to exist in a single line. So when all the records are squished into a single line in S3, the crawler processes the first and throws away the rest. Imagine my surprise when only 1 out of 17 of my log records appeared in the Athena queries.

The workaround is to write a Lambda function with a Kinesis trigger. What this does is that every time a Kinesis record is written, the Lambda gets triggered. Well, that is not strictly true - Kinesis will batch a bunch of records and invoke the lambda once per batch. The batch size (or time for trigger) can be specified from the console.

Or if you are using serverless, this can be specified in the serverless.yml like so:

Without further ado, here's the Lambda that adds the newline:

This is written in node.js, and I used the serverless framework with the node.js template to write it. I'm exporting a single function named newlines. This is triggered when there is a batch of records in the Kinesis data stream. We map over the records, transforming each record by adding a new line. This is done in the add_new_line function.

To let the node engine know what we did, we use the callback. It is standard node.js to pass an error object for errors and null when there are no errors (we succeeded).

firehose.putBatchRecord is for efficiency - we could just as well have used firehose.PutRecord and the results would be the same besides throughput.

JSON to objects in a few languages

2018-06-04T15:26:00.000-07:00

When working with data services, we often have a need to convert JSON strings to objects that model our data. Here is a list of code snippets in different languages that convert this Facebook Graph JSON data.

The list is presented in ascending order of verbosity. Predictably, Javascript most succinctly expresses its wishes, whereas Java uses a copious amount of code. Scala avoids some of that verbosity by using case classes that remove boilerplate code for constructors. Jackson (Java) requires getters and setters to identify which attributes of the object are to be serialized, causing code bloat.

JSON:
Javascript:
Ruby:
Python:
Scala:
Java:

Goldbach Conjecture

2018-04-20T13:10:00.000-07:00

In the 18th century, two mathematicians came up with a conjecture - known by its original creator - named Goldbach conjecture. It says that any even number greater than 2 can be expressed as a sum of two primes. There is no theoretical proof for this yet, but it is said to hold up to 400 trillion.

A program to test Golbach conjecture for a given integer:

This program demonstrates two algorithms that are well known.

The sieve of Eratosthenes to calculate all primes upto a given number
A linear algorithm to find if two numbers in a list sum to a given number.

To prove the Goldbach conjecture for a given n, we use the sieve to find all prime numbers up to n, then use the linear algorithm to find two primes from this list that sums up to n.

Timing with Jupyter notebook

2018-04-06T15:58:00.001-07:00

Pieces of code can be timed within the Jupyter notebook using the %timeit magic.

Here is an example where a grid walk algorithm is implemented three times with progressively better run time, timed with %timeit and graphed using bokeh:

Code:

def num_paths(n):
    M = [[0] * n for i in range(n)]
    for i in range(n):
        M[n-1][i] = 1

    for r in range(n-2, -1, -1):
        for c in range(n-r-1, n):
            M[r][c] = M[r][c-1] + M[r+1][c]
    return M[0][n-1]

def num_paths_from(r, c, n, M):
    if M[r][c] > 0:
        return M[r][c]
    if r == 0 and c == n-1:
        return 1
    paths = ([(x,y) for (x,y) in 
              [(r-1, c), (r, c+1)] if y >= n-x-1 
                                   and y<n])
    npaths = 0
    for x,y in paths:
        npaths += num_paths_from(x,y,n,M)
    M[r][c] = npaths
    return npaths

def num_pathz_from(r, c, n):
    if r == 0 and c == n-1:
        return 1
    paths = ([(x,y) for (x,y) in 
              [(r-1, c), (r, c+1)] if y >= n-x-1 
                                   and y<n])
    npaths = 0
    for x,y in paths:
        npaths += num_pathz_from(x,y,n)
    return npaths

def num_paths_slow(n):
    M = [[0] * n for i in range(n)]
    return num_paths_from(n-1, 0, n, M)

def num_paths_super_slow(n):
    return num_pathz_from(n-1, 0, n)


for sz in range(5,15):
    %timeit num_paths(sz)
    %timeit num_paths_slow(sz)
    %timeit num_paths_super_slow(sz)

Timing:

100000 loops, best of 3: 7.74 µs per loop
10000 loops, best of 3: 26.2 µs per loop
10000 loops, best of 3: 62.1 µs per loop
100000 loops, best of 3: 9.27 µs per loop
10000 loops, best of 3: 32.9 µs per loop
10000 loops, best of 3: 200 µs per loop
100000 loops, best of 3: 11.3 µs per loop
10000 loops, best of 3: 43 µs per loop
1000 loops, best of 3: 615 µs per loop
100000 loops, best of 3: 13.9 µs per loop
10000 loops, best of 3: 56.9 µs per loop
100 loops, best of 3: 2.05 ms per loop
100000 loops, best of 3: 16.6 µs per loop
10000 loops, best of 3: 70.9 µs per loop
100 loops, best of 3: 6.67 ms per loop
100000 loops, best of 3: 19.4 µs per loop
10000 loops, best of 3: 97.4 µs per loop
10 loops, best of 3: 23.7 ms per loop
10000 loops, best of 3: 22.1 µs per loop
10000 loops, best of 3: 105 µs per loop
10 loops, best of 3: 80.2 ms per loop
10000 loops, best of 3: 25.6 µs per loop
10000 loops, best of 3: 135 µs per loop
1 loop, best of 3: 287 ms per loop
10000 loops, best of 3: 29.8 µs per loop
10000 loops, best of 3: 149 µs per loop
1 loop, best of 3: 1.05 s per loop
10000 loops, best of 3: 32.7 µs per loop
10000 loops, best of 3: 171 µs per loop
1 loop, best of 3: 3.78 s per loop

Chart:

Code for the plot:

from bokeh.palettes import Spectral11
from bokeh.plotting import figure, show, output_file

p = figure(plot_width=300, plot_height=300)
slowest = [62,200,615,2050,6670,23700,80200,287000,1050000,3780000]
slower = [26,32,43,56,70,97,105,135,149,171]
fast = [7,9,11,13,16,19,22,25,29,32]
st = 5
end = 8
mypalette=Spectral11[0:3]
p.multi_line(xs=[list(range(st,end)), list(range(st,end)), list(range(st,end))], 
             ys=[slowest[:end-st], 
                 slower[:end-st],
                 fast[:end-st]
                ],
             line_color=mypalette,
             line_width=5
             )

show(p)

This shows how the algorithm with exponential time complexity deteriorates for higher values of n:

Now that I've shown you a bunch of performance numbers and visualization, if you are curious about the algorithm, it is a contrived example of finding the number of paths from one corner of a grid to another, here the squares to the north of the diagonal from top right to bottom left are out of bounds - that is, the path is restricted to the right of the diagonal. In this image, we show the problem for n = 5.

The exponential algorithm recursively finds the number of paths from each point to the end point (the top right corner). But since you can reach a single point by a number of paths (and this number increases exponentially with n), the same computation of finding the number of paths from this point to the grid corner is repeated, causing the slowdown.

The next improvement is to remember the number of paths once calculated. Say if we are on [4,2], we will calculate the path to the grid end from here and mark it in M[4][2]. Next time we are at [4,2], we no longer need to calculate again, as the result can be looked up from M[4][2].

The last algorithm uses dynamic programming to do even less work. It works based on the simple observation that a cell (i,j) can only be reached from just 2 cells. Those are the cell to its immediate left, (i,j-1) and the cell right below it, (i+1,j). Then there is just a single path from these two to (i,j). So if we know the number of paths to those two cells, we can add them up to find the number of paths to (i,j). Then we can keep calculating the paths to each cell, walking from bottom row up, going right on the columns and eventually, we will fill the cell at the top right (0, n -1).

Pandas snippets

2018-04-04T17:08:00.000-07:00

Here are some useful snippets that can come in handy when cleaning data with pandas. This was useful for me in completing the coursework for python data science course.

Extract a subset of columns from the dataframe based on a regular expression:
Code:

persona1 = pd.Series({
                        'Last Post On': '02/04/2017',
                        'Friends-2015': 10,
                        'Friends-2016': 20,
                        'Friends-2017': 300
})

persona2 = pd.Series({
                        'Last Post On': '02/04/2018',
                        'Friends-2015': 100,
                        'Friends-2016': 240,
                        'Friends-2017': 560
})

persona3 = pd.Series({
                        'Last Post On': '02/04/2014',
                        'Friends-2015': 120,
                        'Friends-2016': 120,
                        'Friends-2017': 120
})

df = pd.DataFrame([persona1, persona2, persona3], 
                  index=['Chris', 'Bella', 'Laura'])
df.filter(regex=("Friends-\d{4}"))

Output:

	Friends-2015	Friends-2016	Friends-2017
Chris	10	20	300
Bella	100	240	560
Laura	120	120	120

Set a column based on the value of both the current row and adjacent rows:

For this example, we define regulars to the gym as those who have gone to the gym last year at least 3 months in a row:
Code:

import datetime
df = pd.DataFrame({'Month': 
                   [datetime.date(2008, i, 1).strftime('%B')
                             for i in range(1,13)] * 3, 
                   'visited': [False]*36},
                   index=['Alice']*12 + 
                         ['Bob']*12 + 
                         ['Bridgett']*12)

df = df.reset_index()

def make_regular(r, name):
    r['visited'] = (r['visited'] or (r['index'] == name) and 
                  ((r['Month'] == 'February') or
                   (r['Month'] == 'March') or
                   (r['Month'] == 'April')))
    return r

df = df.apply(make_regular, axis=1, args=('Alice',))
df = df.apply(make_regular, axis=1, args=('Bob',))
regular = ((df['visited'] == True) & 
          (df['visited'].shift(-1) == True) & 
          (df['visited'].shift(-2) == True))
df[regular]['index'].values .tolist()

Output:

1	['Alice', 'Bob']

Pushing your code to pypi

2018-03-23T19:32:00.001-07:00

Here is a good document that describes how to push your code to the Pypi repository.

A URL has changed slightly. In your ~/.pypirc set the URL as follows:

[pypitest]

repository=https://test.pypi.org/legacy/

The register step is no longer required. All you need to do is upload the files.

python setup.py sdist upload -r pypitest

Each time you initiate an upload, you'd need to change the version number and the URL.

While this uploaded the package to test.pypi.org, the upload steps had changed for pypi.org:

thushara@ figleaf (master)$ python setup.py sdist upload -r pypi

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'

warnings.warn(msg)

running sdist

running check

warning: sdist: manifest template 'MANIFEST.in' does not exist (using default file list)

warning: sdist: standard file not found: should have one of README, README.txt

writing manifest file 'MANIFEST'

creating figleaf-0.2

creating figleaf-0.2/figleaf

making hard links in figleaf-0.2...

hard linking setup.cfg -> figleaf-0.2

hard linking setup.py -> figleaf-0.2

hard linking figleaf/__init__.py -> figleaf-0.2/figleaf

hard linking figleaf/graph.py -> figleaf-0.2/figleaf

Creating tar archive

removing 'figleaf-0.2' (and everything under it)

running upload

Submitting dist/figleaf-0.2.tar.gz to https://pypi.python.org/pypi

Upload failed (410): Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at https://pypi.org/). For more information about migrating your use of this API to PyPI.org, please see https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html)

error: Upload failed (410): Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at https://pypi.org/). For more information about migrating your use of this API to PyPI.org, please see https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html)

To upload to pypi I used twine. Installing that on MacOS High Sierra required the removal of SIP.

In ~/.pypirc, I removed the repository line under [pypi]

python setup.py sdist

Remove old tars under dist, and

twine upload dist/*

Now I could see the project under pypi

Installing Twine on MacOS High Sierra

2018-03-23T18:47:00.002-07:00

thushara@ wildhops (master)*$ sudo -H pip install twine

Password:

Collecting twine

Downloading twine-1.11.0-py2.py3-none-any.whl

Collecting pkginfo>=1.4.2 (from twine)

Downloading pkginfo-1.4.2-py2.py3-none-any.whl

Requirement already satisfied: setuptools>=0.7.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from twine)

Collecting tqdm>=4.14 (from twine)

Downloading tqdm-4.19.8-py2.py3-none-any.whl (52kB)

100% |████████████████████████████████| 61kB 2.1MB/s

Collecting requests-toolbelt>=0.8.0 (from twine)

Downloading requests_toolbelt-0.8.0-py2.py3-none-any.whl (54kB)

100% |████████████████████████████████| 61kB 1.6MB/s

Requirement already satisfied: requests!=2.15,!=2.16,>=2.5.0 in /Library/Python/2.7/site-packages (from twine)

Installing collected packages: pkginfo, tqdm, requests-toolbelt, twine

Exception:

Traceback (most recent call last):

File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 215, in main

status = self.run(options, args)

File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 342, in run

prefix=options.prefix_path,

File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 784, in install

**kwargs

File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 851, in install

self.move_wheel_files(self.source_dir, root=root, prefix=prefix)

File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files

isolated=self.isolated,

File "/Library/Python/2.7/site-packages/pip/wheel.py", line 377, in move_wheel_files

clobber(source, dest, False, fixer=fixer, filter=filter)

File "/Library/Python/2.7/site-packages/pip/wheel.py", line 316, in clobber

ensure_dir(destdir)

File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 83, in ensure_dir

os.makedirs(path)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 150, in makedirs

makedirs(head, mode)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs

mkdir(name, mode)

OSError: [Errno 1] Operation not permitted: '/System/Library/Frameworks/Python.framework/Versions/2.7/man'

The only way to get write access under /System is to boot into Recovery Mode and run this command on the Terminal:

csrutil disable

Reboot, install again

A Graph in Python - and pythonic surprises

2018-03-22T16:03:00.000-07:00

I started implementing a Graph in python for a project and I encountered an unexpected behavior. See if you can spot the problem.

Code for the graph is here:

However this is buggy. Each time an edge is added to one node, it gets added to all the nodes. Adding an edge from 'bellevue' to 'lynwood' added the edge to both vertices 'bellevue' and 'lynwood'.

Code/Output:

g.add_node(GraphNode('seattle', [Edge('seattle', 'bellevue', 'dist', 10), Edge('seattle', 'lynwood', 'dist', 20)]))

g.add_edge(('bellevue', 'lynwood', 'dist', 5))

print (g)

bellevue -> bellevue:lynwood:dist:5
lynwood -> bellevue:lynwood:dist:5
seattle -> seattle:bellevue:dist:10 seattle:lynwood:dist:20

After a lengthy debugging stint, the issue was identified to be the way Python evaluates default argument values to functions.

Python : Common pitfalls

2017-12-07T20:36:00.001-08:00

Join a list if and only if all values in the list are strings:
Code:

print ("running %s" % ' '.join(cmd))

Error:

Traceback (most recent call last):

  File "test.py", line 29, in <module>

    print ("running %s" % ' '.join(cmd))

TypeError: sequence item 4: expected string, int found


Cause:

There are non-strings in the list cmd.


Ex:

cmd = ["runthis.py", "--host", host, "--port", port]



To make join  happy:

cmd = ["runthis.py", "--host", host, "--port", str(port)]



Avoid default values in mutable arguments:

Code:

class A:

    def __init__(self, lst=[]):

        self.lst = lst



a = A()

b = A()

a.lst.append('crocs')


print (b.lst)


Output:

['crocs']


Cause:

Python evaluates default arguments to a function at the time the function

is defined, not each time it is called. So all instances will mutate a single

list.

A Beautiful Soupy Exercise in Scraping Interesting Integers

2017-11-05T12:19:00.000-08:00

Integers can be very interesting at least if you are a mathematician, and even for a lay person like me, interesting integers can be used to spice up some data that is of interest. My interest here being the bike counter installed on the Fremont Bridge sidewalk that counts the number of bicycles crossing the bridge in both directions.

Inspired by this idea of blogjects and twittering houses, I wanted to send out an early morning tweet of the number of cyclists who braved the streets across Fremont the day before. The data is uploaded by SDOT early morning, and a cron task would request for this and tweet it.

Simple and a tad boring. Now what if I could map the count to something interesting? Researching on interesting integers, I came up on the theorem that there are no uninteresting integers because after all if there were a bunch of these, and one of them must be the smallest of the lot, and the fact itself makes this number interesting.

Energized in no small measure by this revelation, I sallied forth to find a list of interesting integers in the thousands range, as every day there were ~ 3000 cyclists being logged. It didn't take long for me to reach a comprehensive page of integers. The pattern is an integer within a tag, followed by a phrase that describes it.

0 is the <a href="http://mathworld.wolfram.com/AdditiveIdentity.html">additive identity</a>. 1 is the <a href="http://mathworld.wolfram.com/MultiplicativeIdentity.html">multiplicative identity</a>. 2 is the only even <a href="http://mathworld.wolfram.com/PrimeNumber.html">prime</a>.

At first glance, it seemed a simple matter of using Beautiful Soup to get each font tag, extract its text, then look for the font's sibling to extract the phrase.

However, the font tag has multiple siblings that make up the complete phrase. In the soup these are represented as NavigableString objects. It's a matter of moving across the document until we hit a tag, collecting all the text as we go along.

Now since all of this needs to be part of the tweet, I quickly realized that not much can be said in 140 characters. So I didn't bother keeping the URLs. I used a jupyter notebook to quickly prototype the outline, and I can't stress enough how useful this is, specially when you are dealing with an unfamiliar API (which Beautiful soup was to me).

Here is how I used the notebook to understand the basic structure of the page:

So getting to the integers was quite trivial as BeautifuSoup provides a way to search for a specific tag (font) with a specific value for a given attribute (size=+3). Since the phrase for the integer is in a number of contiguous elements, we need to construct it by visiting siblings of the font tag until we hit a tag.

unexpected_tags = {}
def get_text_to_eol(font_section):
    text_parts = []
    section = font_section.next_sibling
    while section.name != 'br':
        if section.name == 'a':
            text_parts.append(section.string)
        elif section.name is None:
            text_parts.append(str(section))
        else:
            print ("found %s tag" % section.name)
            unexpected_tags[section.name] = unexpected_tags.get(section.name, 0)+1
        section = section.next_sibling    
    return ' '.join(text_parts)

Now I ran through the results in jupyter, and the first few are shown below:

for number, text in map(lambda section: (section.get_text(), get_text_to_eol(section)), integer_sections):
    print (number, text)

0  is the  additive identity .
1  is the  multiplicative identity .
2  is the only even  prime .
3  is the number of spatial dimensions we live in.
4  is the smallest number of colors sufficient to color all planar maps.
5  is the number of  Platonic solids .
6  is the smallest  perfect number .
7  is the smallest number of sides of a  regular  polygon that is not  constructible  by straightedge and compass.
8  is the largest  cube  in the  Fibonacci sequence .
9  is the maximum number of  cubes  that are needed to sum to any positive  integer .
10  is the base of our number system.
11  is the largest known  multiplicative persistence .
12  is the smallest  abundant number .
13  is the number of  Archimedean solids .
14  is the smallest even number n with no solutions to  φ (m) = n.
15  is the smallest  composite number  n with the property that there is only one  group  of order n.
found sup tag
found sup tag
16  is the only number of the form x  = y  with x and y being different  integers .
17  is the number of  wallpaper groups .
18  is the only positive number that is twice the sum of its digits.
found sup tag
19  is the maximum number of 4  powers needed to sum to any number.
20  is the number of  rooted trees  with 6 vertices.
21  is the smallest number of distinct  squares  needed to tile a  square .
22  is the number of  partitions  of 8.

Since I collected the unknown tags, I could see what they were:

Here is an example of a superscript being used:
16 is the only number of the form xy = yx with x and y being different <a href="http://mathworld.wolfram.com/Integer.html">integers</a>.

Now isn't that interesting, out of all the numbers that this is the only one?

Here is how the subscript is being used:

126 = 9<a href="http://mathworld.wolfram.com/Combination.html">C</a>4.

With this insight, I augmented the superscripts with the ^ symbol, and left the subscripts as is.

def get_text_to_eol(font_section):
    text_parts = []
    section = font_section.next_sibling
    while section.name != 'br':
        if section.name == 'a':
            text_parts.append(section.string)
        elif section.name is None:
            text_parts.append(str(section))
        else:
            if section.name == 'sup':
                text_parts.append('^')
            text_parts.append(section.string)
        section = section.next_sibling    
    return ' '.join(text_parts)

And I again digressed on a merry tangent where people were using non-ascii characters to tweet subscripts and superscripts.

However, somewhat surprisingly, the sibling list did not always end in a . I hit a None for a section for integer 248 with the html:

248 is the smallest number n>1 for which the <a href="http://mathworld.wolfram.com/ArithmeticMean.html">arithmetic</a>, <a href="http://mathworld.wolfram.com/GeometricMean.html">geometric</a>, and <a href="http://mathworld.wolfram.com/HarmonicMean.html">harmonic means<a/> of <a href="http://mathworld.wolfram.com/TotientFunction.html">φ</a>(n) and <a href="http://mathworld.wolfram.com/DivisorFunction.html">σ</a>(n) are all <a href="http://mathworld.wolfram.com/Integer.html">integers</a>. 

Can you spot the problem, it is subtle?

Notice that harmonic means<a/> is not the correct encoding. Beautiful soup replaces this dangling tag with a beautiful pair <a></a>:

<a href="http://mathworld.wolfram.com/HarmonicMean.html">harmonic means<a></a> of <a href="http://mathworld.wolfram.com/TotientFunction.html">φ</a>(n) and <a href="http://mathworld.wolfram.com/DivisorFunction.html">σ</a>(n) are all <a href="http://mathworld.wolfram.com/Integer.html">integers</a>.<br/></a>

This is all very nice, except that we were relying on a tag to be an eventual sibling, and Beautiful soup is on a soupy wake trying to find the matching </a> to the tag it started with, finally finding it at:

1351 has the property that <a href="http://mathworld.wolfram.com/e.html">e</a>1351</a> is within .0009 of an <a href="http://mathworld.wolfram.com/Integer.html">integer</a>. 

We are getting all the integers from 248 through 1351 in one unbroken block.

Is there then an easier way to solve this problem? It's tempting to think regular expressions when it comes to html parsing issues of this sort. What if we use a regular expression to split apart the sections containing the integer and phrase? After all, using a top down parser snagged on a mismatched tag, but maybe a regular expression can give us a better behaved set of html tags which we can then parse individually with Beautiful Soup.

We can get the html lines with a reg exp split.

Since we decided to forego the URLs, we could construct a BeautifulSoup instance for each line, and call the get_text() method to strip all tags.

import re
lines = re.split("<br>[\r\n\s]*", html)
list_lines = list(filter(lambda x: x is not None, [re.search(r"<font size=\+3 .*", line) for line in lines]))
text_lines = [BeautifulSoup(l.group(0), 'html.parser').get_text() for l in list_lines]

['0 is the additive identity.',
 '1 is the multiplicative identity.',
 '2 is the only even prime.',
 '3 is the number of spatial dimensions we live in.',
 '4 is the smallest number of colors sufficient to color all planar maps.',
 '5 is the number of Platonic solids.',
 '6 is the smallest perfect number.',
 '7 is the smallest number of sides of a regular polygon that is not constructible by straightedge and compass.',
 '8 is the largest cube in the Fibonacci sequence.',
 '9 is the maximum number of cubes that are needed to sum to any positive integer.',
 '10 is the base of our number system.',
 '11 is the largest known multiplicative persistence.',
 '12 is the smallest abundant number.',
 '13 is the number of Archimedean solids.',
 '14 is the smallest even number n with no solutions to φ(m) = n.',
 '15 is the smallest composite number n with the property that there is only one group of order n.',
 '16 is the only number of the form xy = yx with x and y being different integers.',
 '17 is the number of wallpaper groups.',

But now we are not identifying the superscripts, subscripts, as you can see from the output for integer 16.

What we should do is to then use the regular expression to get the lines, apply the parser for each line, then use the function we wrote earlier to get the text within each line. Now since the parser can't go over a , it just might result in a better extraction of phrases.

import re
lines = re.split("<br>[\r\n\s]*", html)
list_lines = list(filter(lambda x: x is not None, [re.search(r"<font size=\+3 .*", line) for line in lines]))
soups = [BeautifulSoup(l.group(0), 'html.parser').font for l in list_lines]
np_list = [(int(s.get_text()), get_text_to_eol(s)) for s in soups]

(7,
  ' is the smallest number of sides of a  regular  polygon that is not  constructible  by straightedge and compass.'),
 (8, ' is the largest  cube  in the  Fibonacci sequence .'),
 (9,
  ' is the maximum number of  cubes  that are needed to sum to any positive  integer .'),
 (10, ' is the base of our number system.'),
 (11, ' is the largest known  multiplicative persistence .'),
 (12, ' is the smallest  abundant number .'),
 (13, ' is the number of  Archimedean solids .'),
 (14, ' is the smallest even number n with no solutions to  φ (m) = n.'),
 (15,
  ' is the smallest  composite number  n with the property that there is only one  group  of order n.'),
 (16,
  ' is the only number of the form x ^ y  = y ^ x  with x and y being different  integers .'),
 (17, ' is the number of  wallpaper groups .'),

Now we convert this list of pairs to a dictionary, so we can quickly look up the integer:

hash = dict(np_list)

{0: ' is the  additive identity .',
 1: ' is the  multiplicative identity .',
 2: ' is the only even  prime .',
 3: ' is the number of spatial dimensions we live in.',
 4: ' is the smallest number of colors sufficient to color all planar maps.',
 5: ' is the number of  Platonic solids .',
 6: ' is the smallest  perfect number .',
 7: ' is the smallest number of sides of a  regular  polygon that is not  constructible  by straightedge and compass.',
 8: ' is the largest  cube  in the  Fibonacci sequence .',
 9: ' is the maximum number of  cubes  that are needed to sum to any positive  integer .',
 10: ' is the base of our number system.',
 11: ' is the largest known  multiplicative persistence .',
 12: ' is the smallest  abundant number .',
 13: ' is the number of  Archimedean solids .',
 14: ' is the smallest even number n with no solutions to  φ (m) = n.',
 15: ' is the smallest  composite number  n with the property that there is only one  group  of order n.',
 16: ' is the only number of the form x ^ y  = y ^ x  with x and y being different  integers .',

The get_text_to_eol() was modified to handle hitting the end of the sibling list without hitting a . Also, we keep all strings unicode up until they need to be output, at which point a conversion to utf-8 is done.

def get_text_to_eol(font_section):
    text_parts = []
    section = font_section.next_sibling
    while section is not None:
        if section.name == 'a':
            text_parts.append(section.get_text())
        elif section.name is None:
            text_parts.append(unicode(section))
        else:
            if section.name == 'sup':
                text_parts.append('^')
            text_parts.append(section.get_text())
        section = section.next_sibling    
    return ' '.join(text_parts)

I think this will do for our purposes. In the next post I will show how this was used along with the bike counter uploads to tweet early morning updates to twitter.

The full source code, along with the twitter updates can be found here.

tech

What the workflow looked like

What Claude found

The leadership challenge simulation

Three alternative shapes

Script-first

Separate stack per rule

Migrate to EventBridge Scheduler + CDK / Terraform

What this is really about

How We Migrated Sidekiq's Redis Without Losing a Single Job

The Setup

What the AI Tools Said

The Actual Solution

How It Went

Why the AI Advice Missed the Mark

Takeaways

Dead Code Is a Cognitive Tax — Here's How AI Helps You Stop Paying It

Dead Code Is a Cognitive Tax — Here's How AI Helps You Stop Paying It

What Is Cognitive Load in a Codebase?

A Real-World Example: The Estimation Pipeline Cleanup

The Numbers

The Cognitive Impact of the Cleanup

Where AI Fits In: Finding Dead Code You Can't See

A Telling Real-World Example: Claude Gets Confused, Then Catches Itself

What AI Can Do

What AI Can't Do (Yet)

Making Dead Code Cleanup a Habit

Conclusion

When AI Sounds Right But Isn't: A Sidekiq Story

Step 1: My Initial Idea (and Why Claude Correctly Pushed Back)

Step 2: A Confident, Detailed, Completely Wrong Answer

Step 3: Thirty Seconds in the Docs

What Makes This Failure Mode Dangerous

A Simple Rule of Thumb

Death-Defying Sidekiq Jobs: Part 2

The Data Revealed More

Enter bulk_perform

The Results

Lessons Learned

Conclusion

Death-defying sidekiq jobs

The Problem

The Solution (Deployed Methodically)

Testing Challenges

Analyzing the Differences

Users We Missed (orig_set - new_set)

New Drivers (new_set - orig_set)

When Your Fix Becomes the Problem: A Tale of AWS Outages, Redis Flags, and Performance Scaling

The Original Problem: AWS Outage Chaos

The Solution That Worked (Too Well)

The Problem With the Solution

The Real Culprit: Death by a Thousand Drivers

The Hidden Performance Regression

The Real Fix: Performance First, Then Locking

The Performance Bottleneck

The Solution: An Active Driver Index

The Expected Results

Lessons Learned

The Architecture Pattern

Conclusion

When AWS Went Down, Our Users Didn’t Lose Their Miles

How Our System Works

What Happened During the Outage

Improvements made:

Our Approach: Fairness and Fidelity

What We’re Building Next

What It Means for Our Users

Rails 8: - can't cast RSpec::Mocks::Double

Debugging Rails RSpec Error: RSpec::Mocks::Double Casting Issue

Real time location of drivers : a tale of repurposing a Jupyter Notebook

Postgres indices on large tables : gotchas

Ruby lazy collections and with_index

Using Redis for web request counting

Problem Statement

Limitations of available solutions

Counters help scale the app

Why Redis?

Choice of the Sorted Set

A look at the running counters

Implementation

Enter `bulk_perform`