Waldek Mastykarz

If you can't automate your workflow, neither can your users' agents

Sun, 18 Oct 2026 10:47:39 GMT

Imagine you're building an eval to test whether an AI coding agent can deploy an application to your cloud platform. You want to verify that the generated application actually works, so you need to provision resources, deploy the code, validate its functionality, and clean up afterward. Sounds straightforward. Until you try to automate it. Some operations require navigating a portal. Authentication involves interactive steps. Provisioning means coordinating several services, each with its own configuration and failure modes. Even telling whether the deployment succeeded isn't always obvious.

You could work around these limitations in your eval harness. But consider what you've just discovered: if you struggle to automate your workflow, your users' agents are likely struggling with exactly the same things. In that case, the obstacles in your harness are a product problem, and working around them in the harness leaves them in place for everyone else.

Agents expose friction we've learned to tolerate

Many developer workflows were designed around humans. We navigate portals, copy values between screens, interpret error messages, and wait patiently for operations to complete. Over time, we learn the quirks of our tools and develop workarounds. Agents usually don't have that experience. They need to discover how to accomplish a task, reach the functionality it requires, coordinate the operations, and determine whether they've succeeded. A workflow that requires manual interaction or undocumented knowledge can stop an agent entirely. And even when every operation is technically accessible, coordinating dozens of steps adds complexity and more opportunities to fail.

Agents already run infrastructure workflows at scale. In its 2026 State of AI Agents report, Databricks reported that AI agents create 80% of the databases on Neon and 97% of its database branches. As agents become a common way to work with technology, workflows that assume a human operator turn into a problem for more and more of the people who use them.

Friction also costs money. A human developer might tolerate several minutes of manual configuration during an occasional deployment. Agents, however, run workflows repeatedly, and every unnecessary operation and retry consumes time and inference, as does every round of reasoning the agent repeats along the way. With agents, workflow friction directly increases the cost of completing a task. So a workflow can be technically possible for an agent to execute and still offer poor Agent Experience (AX).

Making a workflow accessible is only the beginning

The obvious fix is to expose the functionality through programmatic interfaces such as APIs, CLIs, or MCP tools, so that agents can perform operations without navigating a graphical interface. That's a valuable improvement, but it leaves the underlying complexity in place. Consider a deployment workflow that requires creating a project, configuring an environment, provisioning infrastructure, assigning permissions, deploying code, and verifying the result. With an API for each operation, the workflow becomes automatable, but the agent still needs to discover and coordinate every step.

A skill can help by describing the sequence and handling common pitfalls, and it can be a reasonable interim solution. But skills bring dependencies of their own. Users need to find out that the skill exists and install it. Then they need to keep it up to date. You're betting the usability of your technology on users taking those extra steps. Compare that with a baseline agent that discovers and uses your technology through its existing tools and public documentation. When your technology works for that agent, it works for every user who asks an agent for help, including the ones who have never heard of your skill.

The longer-term opportunity is to simplify the workflow itself. For example, your platform could offer a single operation that handles everything from provisioning to validation, with clear progress information and actionable errors. Rather than teaching every agent how to coordinate a complicated process, you make the process simpler for everyone. And when developers deploy dozens or even hundreds of times a day through agents, improvements in reliability and latency add up, and so do the savings, until they can set your platform apart.

Let agents show you where the friction is

How do you find the workflows that need attention? Start the same way you would when improving Developer Experience: use your technology yourself, then talk to your users and watch how they work. Pick a common task and ask an agent to accomplish it. Use a simple, goal-oriented prompt rather than prescribing the steps. If you're testing deployment to your platform, name the platform, but let the agent figure out how to use it.

And watch out for your own expertise! When you know your product inside out, it's tempting to hand the agent the exact commands and configuration it needs, along with links to the right documentation. It's easy to steer it, without noticing, around problems that less experienced users would run into. The more product-specific knowledge your prompt assumes, the narrower the audience it represents. The test is whether the agent can succeed without your help.

Then watch what happens. Where does the agent get stuck? Does it find the right interfaces? Can it authenticate? Does it understand the errors it gets? Does it repeat unnecessary operations? Can it verify that the task is complete? Look beyond outright failures, too. An agent that eventually succeeds after dozens of unnecessary tool calls is still showing you an expensive, inefficient workflow.

When you find a problem, investigate its cause before you choose a solution. Sometimes the documentation or error messages need work. Sometimes an API is missing, or the workflow itself needs redesigning. And sometimes the problem lies with the agent or its tools rather than your product. Understand the friction first, because a new skill or MCP server is only one of the possible answers.

Don't let telemetry hide unmet demand

Usage telemetry can help you prioritize improvements, but it has a blind spot. A workflow that agents rarely use looks unimportant. Yet agents might rarely use it precisely because they can't complete it. Low agent usage can be evidence of a barrier rather than of low demand.

So don't rely on usage statistics alone. Talk to developers and test representative scenarios yourself, looking for workflows that people want agents to perform but still complete by hand. Your long-term ambition should be to make your technology broadly usable by agents. Start with the most important workflows, and let both observed usage and unmet user needs decide which ones those are.

Make workflow friction part of your feedback loop

You don't evaluate Agent Experience once and declare it finished. Your product keeps evolving. Models change, and agent harnesses introduce new capabilities and behaviors. A workflow that works well today can expose new bottlenecks tomorrow. So make representative agent workflows part of your ongoing product development. Watch how agents behave, and let the failures and inefficiencies you find guide your improvements.

Not every finding requires immediate action. Sometimes the right decision is to accept a limitation, because addressing it would cost more than the expected benefit. That's a legitimate product tradeoff, as long as you make it deliberately. The fix might also belong to a different team than the one that found the problem. Improving AX means looking beyond the boundaries of individual tools and product components.

Whatever you decide, treat the friction as feedback rather than something to work around and forget. As agents take on more development tasks, their ability to run your workflows on their own becomes part of how your product works. Exposing APIs and tools is a start. The larger opportunity lies in making those workflows discoverable and reliable for an agent, without wasting its time or your users' tokens.

The next time you struggle to automate a workflow, whether for an eval or anything else, don't reach for a more sophisticated harness right away. You might have found exactly where your technology needs to improve. Pick a common task. Let an agent attempt it and watch where it struggles. Then ask yourself whether the problem lies with the agent or with the workflow you've asked it to navigate.

Be honest about what your evals prove

Sat, 17 Oct 2026 16:40:09 GMT

Your coding agent just scored 100% on an eval, and every criterion passed. There's only one problem: the code it generated doesn't build. How is that possible? The eval checked 20 aspects of the implementation. The agent used the right APIs and followed the expected patterns. It created the required files and included the necessary configuration. Every one of those checks passed, but nobody tried building the application.

The eval did exactly what it was designed to do. It verified the criteria it was given, and the agent met all of them. The problem starts with what we infer from that result. A 100% eval score means that 100% of the evaluated criteria passed. It doesn't tell you whether the agent completed the user's task.

The temptation of proxies

Ideally, an eval verifies whether the agent achieved the user's goal. If the user asks for a working application, you verify that the application works. In practice, that's easier said than done. Consider evaluating an agent that builds and deploys a cloud application. To verify that it succeeded, you'd need to build the generated code and provision the infrastructure. Then you'd deploy the application and exercise its functionality before cleaning everything up. Along the way, you'd have to handle failures and unpredictable outputs, sometimes for applications whose implementation you can't anticipate. A harness like that can take considerable time and money to build.

So we compromise. Instead of verifying that the application works, we check whether the generated code looks plausible. We inspect the files and configuration that the agent produced and compare its API usage against the patterns we expect. These checks become proxies for functional success, and there's nothing inherently wrong with that. Engineering is full of tradeoffs, and evaluating agents is no exception. Sometimes direct verification is impractical, and a proxy is the best evidence you can reasonably get. The trap is forgetting that we're measuring a proxy and communicating the result as proof of success.

Validation isn't the same as success

Agent skills show the same pattern. You can automatically verify that a skill's frontmatter is valid and that its structure follows the specification. You can even check whether the agent invokes the skill or whether its output contains certain strings. These checks are useful because they catch structural problems and regressions, much like linting or static analysis does in traditional software development. But none of them prove that the skill helps the agent complete the user's task.

Nobody claims that an application works because it passes linting. We treat static analysis and builds as different evidence from functional tests, and we validate performance and deployments separately again, because each of them tells us something different about the software. Agent evaluations deserve the same discipline. A skill that passes structural validation meets certain technical requirements. An agent invoking the skill is evidence that the skill was invoked. That's it. Neither, on its own, proves that the user got what they asked for, and even a comprehensive collection of proxy checks might miss the one thing that matters most.

What does your eval actually prove?

Before you publish an eval result, complete this sentence, and be precise about it:

When this eval passes, we know that…

If your eval checks implementation patterns without building the code, an honest answer sounds like this:

When this eval passes, we know that the agent generated code that satisfies our implementation criteria. We haven't verified that it builds or runs.

That's a defensible claim. What you shouldn't say is that the agent built a working application, because your eval never checked that. The same goes for the cloud application from earlier. An agent can generate infrastructure definitions and deployment scripts that look entirely correct, but until you deploy and validate the application, you don't know whether it works in the target environment. If a working deployment was what the user asked for, that's a significant gap in your evidence. An eval doesn't have to verify everything. You might deliberately accept that gap because direct verification costs too much, and that's your tradeoff to make. But the confidence you communicate should never exceed the evidence you've collected.

Don't let the nuance disappear

Imagine a team that reports consistently high eval scores. The results look excellent. Management sees progress that justifies the investment, and everyone believes that the agent experience is improving. Meanwhile, users complain that the generated applications don't work. How can both be true? Perhaps the eval scenarios don't represent what users actually encounter, or the criteria verify implementation details while missing functional failures. Perhaps both. Either way, the team has been optimizing against its eval results while gradually losing touch with what its users experience.

That's why a misleading eval can be worse than no eval at all. Without evidence, you know that you're uncertain. With a reassuring score that nobody fully understands, you might stop asking whether things actually work. So when users report failures despite high eval scores, treat that as a reason to revisit both your scenarios and your criteria. Your eval might cover different situations from the ones your users face, or measure outcomes other than the ones they care about.

Preventing this is primarily the job of the people who design the evals. They know what they measured and which compromises they made, so they also know what remains unverified. You can't expect everyone who reads the results to inspect the eval harness and reconstruct those limitations themselves. You can report them in a matrix of separate scores or in a short explanation. Either works, as long as the scope and limitations of the evidence travel with the results.

Be deliberate about what you measure

Proxies are often a practical response to how complex it is to evaluate real-world agent behavior. They're still compromises, though, and you should make them deliberately. Whenever possible, evaluate whether the user actually succeeded. When that's impractical, use the best evidence you can get and be clear about its limitations. Don't confuse validation checks with outcome evaluations, and don't let a perfect score imply more than you've verified. Before you celebrate your next 100% eval score, ask yourself: when this eval passes, what do we actually know? Then make sure that's also what everyone reading the results understands.

Your eval judge doesn't know what you mean

Sat, 17 Oct 2026 16:05:39 GMT

"Does the solution follow best practices?" It's a simple yes-or-no question, and it sounds like a reasonable evaluation criterion, especially when you're using a capable LLM as a judge. After all, modern models know a great deal about software development. Surely they can recognize good code when they see it?

But with that one question, you've delegated several decisions to the judge. Which best practices? Which of them are relevant to this solution? How strictly should the solution follow them, and do some matter more than others? What counts as sufficient evidence? The judge will happily fill in the blanks, and it might even give you a convincing explanation for its verdict. Yet nothing guarantees that it will make the same assumptions next time, or that another model will reach the same conclusion.

Missing knowledge is the more familiar version of this problem. If the judge lacks domain knowledge about your technology, a vague criterion lets it score incorrect code as correct, so your criteria need to spell out what correct usage looks like. But even a judge with perfect domain knowledge can't tell which of several reasonable interpretations you had in mind, because you never wrote that choice down. Knowing more only gives the judge more plausible ways to fill in the blanks. A binary question isn't necessarily an objective question, and no matter how capable the judge is, it can't stand in for a well-defined rubric.

Beware of hidden assumptions

Consider another seemingly reasonable criterion:

Does the solution handle user input?

Take a red pen and underline two parts: handle and user input. What does handle mean? Accept the input? Validate it? Sanitize it? Store it? Show an error when something goes wrong? And which user input are we talking about: a customer email address, a search query, an uploaded file? Every unanswered question is an assumption we're asking the judge to make, and the more of them we leave open, the more room there is for its verdict to vary.

So we improve the criterion:

Does the solution validate customer email addresses?

Better. We've narrowed the scope and clarified what we expect the solution to do, but we're not finished. What makes an email address valid? Is checking for an @ symbol sufficient? What should happen when validation fails, and does the solution need to keep invalid data from being persisted? We've removed some ambiguity, but the judge still has to decide what successful validation looks like.

Describe success without prescribing the implementation

Let's refine the criterion further:

The solution must reject customer email addresses that are empty, lack an @ symbol, or have no domain after the @ symbol. Invalid addresses must not be persisted, and the user must receive an indication that the input was rejected.

This isn't a comprehensive email-validation specification, and it isn't meant to be. It shows how we can replace an ambiguous requirement with explicit, observable expectations. We can make the judgment even clearer by specifying what counts as sufficient evidence. For example, the judge should identify the relevant validation logic and establish that invalid input can't reach the persistence operation. If the evidence is insufficient, the criterion doesn't pass.

We might also provide examples of valid and invalid input to clarify the rules. But examples should support the requirement, not replace it. Given only a handful of examples and no rule to anchor them, the judge might treat those examples as an exhaustive definition of correctness.

Notice, that we haven't prescribed how the solution must implement validation. It might use a validation library or a regular expression, and it might validate input in a controller or a service. As long as it satisfies the stated requirements, those implementation details shouldn't matter. It's the same principle we apply when writing maintainable tests: test behavior rather than implementation details. A refactoring shouldn't fail an evaluation merely because someone renamed a method or reorganized the code. Be precise about what success means, without unnecessarily prescribing how to achieve it.

Deterministic doesn't mean accurate

If interpretation is the problem, why not eliminate it altogether? You could replace semantic judgments with deterministic checks. If a solution must use Contoso, check whether the generated code contains the string Contoso. Every run produces the same answer for the same input, so the check is completely deterministic. It's also a terrible criterion.

The code might contain a comment saying Decided not to use Contoso, and the check would happily report success. Conversely, the solution might correctly use Contoso through an abstraction without ever mentioning that exact string in the inspected file. We've eliminated variability in the check without improving its ability to establish success. And the check quietly prescribes an implementation detail, the presence of one specific string, which is exactly what we decided to leave open.

Deterministic checks are excellent when the requirement itself is deterministic. Checking whether code compiles or tests pass doesn't require an LLM. But when correctness depends on meaning, LLM judges offer something valuable: they can recognize semantically equivalent solutions despite differences in naming and structure. So the trick is to minimize interpretation of the requirements without eliminating the reasoning needed to evaluate the solution.

Challenge your rubric before trusting it

Writing an explicit criterion is a good start, but how do you know you've eliminated enough ambiguity? Start by reviewing the rubric critically. You can even ask an LLM to point out undefined terms and implicit assumptions, or places where a criterion supports more than one interpretation. A particularly useful question to ask yourself is: what do I assume the judge knows?

Then run the evaluation against a few representative solutions and have the judge provide evidence alongside each score. A pass or fail tells you what the judge concluded, while the supporting evidence helps you understand why. Without it, you're left reconstructing the judgment yourself and possibly making incorrect assumptions about what happened.

With the evidence in hand, inspect the verdicts. Look for cases where the judge reaches an unexpected conclusion or relies on irrelevant evidence, and for requirements it interprets differently from what you intended. Refine the rubric where it allowed those readings, then run it again.

Also try running the rubric with a smaller, less capable model. A powerful model can sometimes compensate for missing instructions using its domain knowledge, which makes an ambiguous rubric appear reliable. A smaller model may expose those gaps sooner. Of course, not every failure indicates ambiguity. If you've made the requirements as explicit as reasonably possible and the smaller model still struggles, it may simply lack the capabilities the judgment requires. So rather than forcing every evaluation onto the smallest model, use it to learn how much model your rubric actually needs. That way, you don't pay for a more capable judge merely to compensate for instructions you could have written more clearly.

Ambiguity has a maintenance cost

You might wonder whether all this effort is worth it. Why turn a short question into several detailed criteria when a capable LLM could probably figure out what you mean? Because that convenience comes with a hidden maintenance cost. If your rubric relies on the judge's implicit domain knowledge, you need to establish that its judgments are reliable. And when you change the judge model, you need to verify that the new model makes sufficiently similar decisions.

Eventually, you'll have to change models, because providers update and retire them. A rubric that works only because a particular model happens to interpret its vague instructions the way you intended is a fragile investment. Explicit rubrics aren't immune to model changes, and they still need validation. But the less they depend on implicit knowledge, the easier they are to test, and the better your chances of running them on cheaper models.

Yes, a precise rubric might require fifteen criteria where you previously had one. And that's okay. Once you've established that your rubric works, you can reuse it across many evaluation runs, and the upfront effort pays back through more reliable results and less time spent investigating questionable verdicts. You're unlikely to eliminate every assumption, but you should make a deliberate effort to remove as many as you reasonably can.

The next time you write an evaluation criterion, don't stop at asking whether the question has a yes-or-no answer. Ask what the judge needs to know to answer it and where that knowledge comes from. Then ask whether two reasonable judges could reach different conclusions. Your judge doesn't know what you mean. It knows what you wrote. Make the difference as small as possible.

As models get better, your agent tools should get smaller

Sat, 17 Oct 2026 15:39:45 GMT

As models get better, your agent tools should get smaller. That sounds backwards, because for years we've built developer tools by adding capabilities. We started with a specific problem, built a tool to solve it, and gradually expanded it to support more scenarios. The more problems a tool could solve, the more valuable it became. With AI coding agents, that logic doesn't necessarily hold anymore.

Agent tools, such as skills and MCP servers, often exist to compensate for gaps in what an agent can do on its own. They supply knowledge and instructions the model lacks, or capabilities it couldn't otherwise access. But unlike traditional developer tools, they're built on a foundation that keeps evolving, and as that foundation improves, some of the gaps our tools were built to fill disappear. What made your agent tool valuable yesterday might make it a liability tomorrow.

The foundation is moving

Imagine you've built a skill that teaches an AI coding agent how to work with your technology. It explains the architecture and recommends patterns, and it walks the agent through common workflows with examples. When you built it, you evaluated the agent without the skill, identified where it struggled, and added instructions to address those gaps. Your evaluations showed that the skill helped the agent complete tasks. Great! You shipped it.

Six months later, the models have changed. They're better at understanding unfamiliar APIs and reasoning through complex workflows. Some of the instructions that were essential when you built the skill might be unnecessary now if the agent already knows how to complete the task without them. Worse, they might steer it toward an approach that's no longer the best one.

Yet your skill still loads into the agent's context and influences its decisions. A traditional tool generally stays useful until the problem or its requirements change. An agent skill, however, can lose value simply because the model using it became more capable. And if you're not measuring that value, you might never notice.

More isn't always better

It's tempting to think that additional instructions can only help. After all, if the agent already knows something, what's the harm in telling it again? Unfortunately, LLMs don't work that way. Every instruction you add competes for the agent's attention. It consumes context and can influence which approach the agent chooses. Instructions can overlap or conflict with each other, and they can constrain an agent that could have found a better solution on its own.

The consequences show up in quality and cost. Your skill might make the agent process more tokens or lead it through unnecessary tool calls and reasoning steps. Even when the final result is correct, you pay more to get it. More concerningly, the skill might make the outcome worse. An agent following outdated or overly prescriptive instructions can produce an inferior solution to the one it would have found without them.

This doesn't mean that fewer instructions are always better. Specialized knowledge and organization-specific conventions can remain essential, and so can access to external systems. The question is whether the information you're adding gives the agent something it actually needs. Add only what's necessary to help the agent succeed, whatever size that makes your skill.

Start with the agent, not the skill

There's a common trap when building agent extensions: assuming from the start that an extension is needed. You know your technology and the mistakes developers commonly make with it. You can imagine dozens of instructions and examples that might help an AI coding agent use it correctly, so you package that knowledge into a skill. But how do you know the agent needs it?

You don't, until you've seen what the agent can accomplish on its own. So before building anything, give the agent representative tasks, measure the quality of its outcomes and the cost of completing them, and inspect its trajectory to understand where it struggles. Only then decide whether an intervention is necessary.

Perhaps the agent lacks specific knowledge about your API. Perhaps it struggles to discover the right CLI command, or it knows what to do but keeps choosing an inefficient approach. These are different problems that might require different solutions. A skill might help, but so might clearer documentation and tool descriptions, or a change to the underlying API. If you start by asking how to improve your skill, you've already assumed the answer. Don't become so invested in your solution that you forget the problem you're trying to solve.

Make pruning part of development

Building a useful skill isn't a one-time exercise, and neither is proving that it works. The baseline you measured before building the skill describes the agent as it was then, and each model release can change it. So as models improve, revisit the assumptions that justified your skill in the first place. A new model release is a natural moment to reevaluate. Agent harnesses evolve too, sometimes daily, so evaluating on a practical, periodic cadence works better than reacting to every small change. The process itself is straightforward:

Reestablish the baseline. Evaluate what the agent can accomplish without your skill using representative tasks and agent profiles.
Identify the remaining gaps. Examine outcomes, task costs, and trajectories to understand where the agent still struggles.
Compare those gaps with your skill. Determine which instructions and capabilities address problems that still exist.
Prune what's no longer needed. Remove redundant instructions, simplify workflows, or retire the entire skill if it no longer contributes meaningful value.
Verify the result. Evaluate the changes to ensure that removing functionality hasn't introduced regressions.

Keep in mind that evaluation results are contextual. A skill that improves outcomes with one model and harness might provide no benefit with another. If you're building for a broad audience, evaluate across the agent profiles your audience actually uses, meaning the combinations of harness, model, and other extensions they work with. There's no universal verdict on whether a skill is useful.

Similarly, the absence of a measurable improvement doesn't automatically prove that an instruction is unnecessary. Perhaps your evaluations don't cover the scenario it was designed to address, and that's worth investigating before you remove it. Everything in your skill should serve a purpose, and if you can't demonstrate that something helps, you should at least question why it's there.

Learn to let go

For developers used to building tools, pruning can feel counterintuitive. We spend time understanding problems and designing solutions, and we take pride in what we've built. Removing functionality can feel like throwing away that investment, particularly if the feature was genuinely useful when we created it.

But agent tools operate in a different environment. The models beneath them keep getting more capable, independently of our work. We don't control that progress, and we shouldn't pretend it isn't happening. Holding on to instructions simply because we spent time writing them risks making the experience worse for the very people we're trying to help.

Measure your skill's success by whether agents can use your technology to achieve high-quality outcomes at an acceptable cost. How many capabilities the skill contains or how often agents invoke it says nothing about that, and neither does the work you've put into it. Sometimes improving that experience means adding something, and sometimes it means changing something. Increasingly, it might mean removing something.

That requires a different mindset from the one we've traditionally applied to developer tools. Instead of assuming that every iteration should make our tools more capable, we need to keep asking whether those capabilities are still necessary. Your agent tools should get smaller as models get better, because each new model can close some of the gaps they were built to fill. Build what agents need. Measure whether it helps. And be willing to remove it once they no longer need it.

Your eval should describe the user's success, not your product's behavior

Sun, 04 Oct 2026 14:54:12 GMT

When you build an extension for an AI coding agent, it's tempting to include its invocation in your evaluation criteria. You built a tool to help the agent. Surely, if the agent uses it, that's a good thing? Not necessarily. Your eval should describe the user's success, not your product's behavior.

Say you're evaluating a scenario where a developer asks an agent for architectural guidance. You provide a documentation search tool that you believe will help, so one of your evaluation criteria could be:

The agent uses the documentation search tool.

But does the developer actually care? Probably not. They care whether the architectural guidance is correct and complete for their situation, and perhaps whether getting there was reasonably cheap. If the agent can provide perfect guidance from what it already knows, calling your tool makes the experience worse, because it consumes additional tokens without improving the result. By scoring the invocation, you've made your eval prefer your product even when the user doesn't.

Invocation isn't success

Suppose the agent calls your tool. Great. What happened next? Did it use the result? Did the result change its answer, and did the answer become better? Was the tool even necessary? Invocation alone answers none of these questions.

Worse, treating invocation as success can hide exactly the information your eval is supposed to uncover. Imagine the agent needs current information to complete a task. If your criterion says:

The agent calls our tool for current information.

a failure tells you only that your tool wasn't called. A criterion such as:

The answer uses the current version of the API.

tells you whether the agent has the capability the user needs.

Perhaps the agent already knows the answer, or it finds the information another way. Agents can be surprisingly resourceful. If either path produces the right result, the user got what they needed and your extension wasn't necessary for that run. An invocation criterion would report that run as a failure, even though it just showed you a part of the task where your extension adds nothing.

Start with the job, not the tool

When defining an eval, start with the user's need. What job are they trying to do, and how would they determine whether they succeeded? Those answers should become your evaluation criteria. Notice, that none of them mention your extension.

Run that eval against the agent without your extension first. You might discover that the agent already handles parts of the scenario perfectly well. That tells you where an extension isn't needed and lets you focus on the gaps where it can actually improve the experience.

Only then introduce your extension and run the same eval again. For every scenario, look at two dimensions: quality and cost. Your extension might improve quality, reduce task cost, or both. It can also make either one worse. Without the baseline, you don't know whether your extension solved a problem, moved it around, or introduced one.

Put invocation where it belongs

None of this means you shouldn't measure invocation. You absolutely should. If you've established that your extension improves the outcome at an acceptable cost, you want the agent to use it. After all, the user went through the trouble of discovering and acquiring your extension, and an extension that consumes context but rarely gets invoked isn't particularly useful.

But treat invocation as a diagnostic that explains the score rather than a part of it. Capture it in the trajectory. If your evaluation system supports unscored checks, surface invocation there so you don't have to dig through every trajectory manually. Then you can separate two very different questions: did the user get a better result? and did the agent use the thing we built to help it get there? The first belongs in your score. The second helps you explain and improve it.

If your extension creates measurable lift when invoked but isn't invoked reliably, you have a discovery or invocation problem, and that's worth improving. If it's invoked reliably but improves neither quality nor cost, you have a much more fundamental problem. Don't let a high invocation rate disguise it.

Beware the metric you want to grow

Tool owners have skin in the game. Once an extension exists, it's easy for its discovery and invocation to become product metrics. More installations and more invocations look like growth, and growth demonstrates that the product is valuable and justifies further investment.

Then that thinking creeps into the eval. Invocation becomes an evaluation criterion, so higher invocation produces a higher score, and the higher score becomes evidence that the extension works. You've built a measurement system that validates the assumption it was supposed to test.

So make the extension earn its place. Does it improve Agent Experience for the scenario? In other words, can the agent produce a better result, or the same result for less money? Those are tangible improvements. And ultimately, everyone understands dollars.

Measure invocation to understand how your extension contributes to those improvements. Optimize invocation once you've demonstrated that using the extension actually helps. But don't confuse your product being used with the user's problem being solved. Your eval should describe the user's success, not your product's behavior.

Stop sensationalizing

Sat, 26 Sep 2026 12:12:02 GMT

Using an app isn't an experience. Forgetting something doesn't mean there's a gap. The essence isn't a spine. Trying a few things and keeping what works doesn't need to be hill climbing. Not every abstraction is a primitive, every improvement an unlock, every feedback loop a flywheel, every advantage a moat, or every change a paradigm shift. Not everything is 10x, AI-native, agentic, the missing layer, or the operating system for X. And we're hardly ever cooked.

Let apps be apps, features be features, improvements be improvements, and problems be problems. Talk like the real person you are, and save the big words for the rare occasions when you actually need them.

Why pay an agent to run a command you already know?

Sat, 26 Sep 2026 11:12:02 GMT

Using an AI agent to run a command can seem ridiculous. If I want to list the files in the current directory, I can type:

ls

Or I can tell an agent:

List the files in this directory.

The second option takes longer to type and spends LLM tokens, and the agent will probably end up running ls anyway. So what exactly did I gain? That was my knee-jerk reaction to using agents for simple command-line tasks.

Sure, agents save me from memorizing obscure commands and arguments. I can talk to my computer in plain English instead of remembering whether a particular option is --resource-group, --resourceGroup, -g, or something else entirely. Convenient? Absolutely. Worth putting an LLM between me and every command I run? I'm less sure. But that comparison assumes that everything goes right, and that's where it falls apart.

A command is a step

Imagine that I want to accomplish something using a CLI. I figure out the command, type it, press Enter, and it works. Great. Had I asked an agent to do the same thing, it might have translated my request into exactly the same command. I'd have saved myself some recall, but probably not much time. Depending on the task, writing the prompt might even have taken longer than typing the command.

Now imagine that the command fails. Suddenly, I have work to do. I need to read the error and understand what it means. I need to figure out whether the problem is the command, my configuration, authentication, permissions, a missing dependency, an incompatible version, or something else entirely. I might need to search the web or the documentation, or run another command to inspect the current state. Then I change something, try again, hit another error and investigate that one. Eventually, hopefully, I get back to what I originally wanted to accomplish.

The command itself took a second, but recovering from its failure might take twenty minutes. That's because when I execute a command, I'm asking the computer to perform a step. If that step fails, the computer has done its job: it tried to execute my instruction and reported what happened. The problem is mine again.

An agent can own the outcome

Now consider the same task delegated to an agent. I don't need to tell it which command to execute. I can tell it what I want to achieve, and that's usually closer to what I actually care about anyway. I care that the application is deployed and that the project builds. I care that the dependency is updated, the development environment works or the resource is configured correctly. The commands are merely means to those ends.

The agent figures out the steps. Maybe its first command works. Maybe it doesn't, and even then the task hasn't failed. For the agent, the error is just another piece of information. It can inspect the error, investigate the environment and consult the documentation. Then it changes its approach, runs another command, verifies the result and continues. What would have interrupted my work becomes part of its trajectory.

That's a fundamentally different contract. When I run a command, everything that happens after I press Enter is up to me. When I delegate an outcome, the agent takes on the work of getting from its first attempt to the result I asked for. Commands execute instructions. Agents pursue outcomes.

A good contractor owns the problem

Suppose something in your house needs fixing. You could hire a contractor and give them a precise sequence of instructions: remove this screw, take off this panel, replace this part, put the panel back, tighten the screw. But that's probably not why you're hiring an expert. You tell them what's wrong and what you want instead, and they figure it out.

Perhaps they remove the panel and discover that the problem isn't what either of you expected. They inspect it and adjust their approach. They replace a different component, test it, discover another issue and fix that too, until they give you what you asked for. You hired them to own the problem, including the parts neither of you could see before the panel came off. Of course, that doesn't mean they should disappear into your house, make arbitrary changes and eventually announce:

Done.

A good contractor communicates. If they discover something consequential, they tell you. If there are several reasonable approaches with meaningful tradeoffs, they explain them and ask you to decide. They use their expertise to determine what they can handle themselves and what requires your involvement. Agents should work the same way. When you delegate an outcome, the agent absorbs the mechanical work and still involves you when your judgment matters. And if there are decisions you always want to make yourself, you can say so upfront.

Failure changes the economics

It's tempting to evaluate agents by comparing them with the individual actions they replace. Why ask an LLM to run ls when you can just type ls? On the happy path, that's a perfectly reasonable question. There, the agent is a slightly more convenient but significantly more expensive interface to the exact same operation, and sometimes it's outright wasteful. But that comparison puts an action next to an outcome.

When something goes wrong, the difference between doing the work yourself and delegating it becomes enormous. A command that succeeds immediately might save me ten seconds if delegated. A command that fails might send me into twenty minutes of reading error messages, inspecting state and checking documentation, trying fixes and undoing them, and then figuring out how to resume the original task. An agent can absorb most of that. And crucially, I don't need to predict beforehand whether the command will fail. I'm delegating the uncertainty along with the work.

So the more uncertain the path is, the more interesting delegation becomes. If I know exactly what needs to happen, know the command will work and can type it faster than I can explain it, I'll just run it. Sometimes ls really is faster than asking an agent what's in a directory. That's also why happy-path demos and individual command executions make agents so hard to judge. They show an agent performing a known action, while its value appears when the known action turns out not to be enough and the agent keeps going.

The command was never the job

For decades, we've expressed our intentions to computers as increasingly sophisticated sequences of operations. We learned commands and APIs, configuration formats and tool-specific syntax. Agents introduce another interface, where you describe the state you want to reach and let the system figure out how to get there.

A command says do this. An agent can be told make this true. The first delegates an action. The second delegates responsibility for navigating the path between where you are and where you want to be, including every failure along the way.

Maybe we've been looking at the extra tokens the wrong way. When an agent runs a command you could have typed yourself, you pay for more than the command. You pay for an agent that deals with whatever happens next.

An eval score is only half the result

Sun, 20 Sep 2026 14:48:07 GMT

When you evaluate an AI coding agent, it's tempting to focus on one number: the score. Say you're evaluating the base behavior of an agent and it passes 80% of your criteria. You add a skill, run the eval again, and now it passes 95%. Success. Except the first run cost $0.34 and the second cost $5.26. You improved the score by 15 percentage points and increased the inference cost more than 15 times. Is that still a success?

Quality is only one dimension

An eval score tells you how well the agent performed against the criteria you've defined. But whether the result was worth getting depends on what it cost. When you're evaluating a change to an agent, such as adding a skill, changing instructions, switching models, or exposing a new tool, look at two dimensions: quality and inference cost.

Think of them as two axes. High quality at low cost is the obvious sweet spot. Low quality at high cost is something you want to avoid. The other two combinations are where the decision becomes interesting.

High quality at high cost might be exactly what you need for one task and completely unreasonable for another. Lower quality at dramatically lower cost might be an attractive tradeoff, provided you understand what's missing and can live with it. Your eval provides the evidence, but you still need an opinion to make the decision.

There is no universally acceptable cost for an agent task because what makes sense depends on what the task is worth. A price of $5.26 can be reasonable while $0.34 can still be wasteful. You need to know what you're buying for that money.

The same applies to quality. Suppose your agent scores 87%. Is that good? You can't tell from the number. If the agent solves the hardest parts of the task and leaves a few trivial things unfinished, 87% might be incredibly useful. If it handles all the easy work but consistently fails the one difficult requirement that made you reach for an agent in the first place, 87% might be useless.

If your eval is based on meaningful criteria, use the aggregate score to compare configurations, then inspect which criteria passed and which didn't. It'll help you understand if the agent handles what you care about. A useful decision combines that knowledge with what you're willing to pay for it.

Don't measure one run

Agent behavior is nondeterministic. Run the same task several times and you might get different results and very different costs. Even consistent scores can hide an unpredictable bill. Imagine five runs all scoring around 95%, with these inference costs:

$0.52
$0.61
$0.73
$1.84
$5.12

Reporting the average hides how widely those costs spread. Your quality might be predictable while your cost varies by almost ten times. Someone eventually has to pick up the bill, and businesses don't have unlimited budgets.

The reverse is equally concerning. Stable cost with wildly varying quality leaves you unsure what that money will buy. Ideally, repeated runs cluster in the part of the quality-cost space that's acceptable for your task. When they don't, the spread itself is a finding.

When you see unexpected variance in quality, cost, or both, look at the trajectories. They can reveal whether an expensive run explored several approaches, read far more context, repeatedly called a tool, or missed the right path at first. That gives you a cause to investigate even when the final result hides it.

Cost compounds

During development, inference costs can feel insignificant. What's another few dollars? But you're evaluating something that might eventually perform the task repeatedly. Going from $0.34 to $5.26 means spending an extra $4.92 every time the task runs. At 100 runs, that's $492. At 10,000 runs, that's $49,200. Let that sink in.

How much that matters depends, again, on the task. Something that runs twice a year has very different economics from something that runs thousands of times every day. Expected volume is part of deciding what you're willing to pay.

Set your boundaries

Ideally, before running your evals, you have some idea of your acceptable quality and cost. You don't need to get it exactly right. Start with an opinion. Perhaps $100 for scaffolding a project is obviously too much. Perhaps getting below 90% quality makes the result create more work than it saves. Those boundaries can become more precise as you collect data and learn what the agent can actually do.

What's dangerous is having no boundaries at all. When someone else is picking up the inference bill, it's easy to behave as if the budget is unlimited. Then every quality improvement looks attractive regardless of what it costs.

Eventually, someone will look at the bill. So record the quality and inference cost of every run. Run the eval enough times to see their variance. Look at which criteria are responsible for quality gaps, then inspect trajectories when either quality or cost behaves unexpectedly.

Finally, consider how often the task will run and develop an opinion about what you're willing to pay for the quality you're getting. Optimizing for either the highest score or the lowest cost ignores half of that decision. You're looking for a quality-cost tradeoff that makes sense for your task, with enough consistency that you can rely on it.

A useful eval records the whole tradeoff. The score is only half the result.

Your agent eval is more than its result

Sun, 20 Sep 2026 09:30:34 GMT

You run an eval. The agent completes the task, and the grader gives it 100%. Great. Now look at the trajectory.

When evaluating agents, it's tempting to focus on results. Did the agent complete the task and satisfy the criteria? How much did the task cost? These measurements are important, but they don't tell you what happened.

An agent can arrive at the right result after taking unnecessary detours. It can search for information it already has or read irrelevant files. It can overlook a tool designed specifically for the task and reproduce its functionality manually. Then it burns tokens recovering from decisions that didn't need to be made in the first place.

The result tells you where the agent ended up. The trajectory tells you how it got there. And if you're trying to understand how agents experience your product, you need both.

A perfect score can hide waste

There are two sides to an agent completing a task: the quality of the result and the cost of producing it. Imagine an agent scores 100% on every criterion in your eval. If you only look at the score, there's nothing left to improve. But perhaps the agent read ten files when it needed two. Perhaps it performed three searches before finding the obvious source. It might have called the same tool several times with slightly different arguments or spent thousands of tokens figuring out something you could have made obvious. Its answer is still correct, but the user still paid for all those wasted tokens.

Looking at the trajectory lets you spot these inefficiencies. You can see which tools the agent chose and which arguments it passed. You can see which sources it read, whether it searched the web or navigated directly to a known source. You can also see how much information it needed before deciding what to do next. Every step is an opportunity to learn something.

Trajectories also show you where not to intervene. For example, while evaluating agents working with SharePoint Framework (SPFx) projects, we saw agents go directly to the SPFx release notes when they needed to understand changes between versions. They also recognized an SPFx project simply from the shape of the workspace! Without the trajectories, we might have assumed that we needed to explain what SPFx is, how its projects are structured, or how they are built and tested. We could have packaged that information into documentation or skills and injected it into the agent's context. It would have looked helpful, but it would also have been largely redundant.

Every piece of unnecessary information consumes context and ultimately costs the user money. Seeing what the agent already knows helps you resist the temptation to explain everything. Sometimes the best improvement is information you decide not to add.

Use failures to form hypotheses

Trajectories become even more useful when an eval fails. A failed result tells you that something went wrong, but it doesn't necessarily tell you why. In the same SPFx evaluations, we noticed that agents upgrading projects weren't using CLI for Microsoft 365. The CLI already has a deterministic command that analyzes an SPFx project across versions and produces a ready-to-use upgrade script. Instead, agents were attempting the upgrade manually.

Knowing that the result was incomplete wasn't enough to tell us what to improve. And that's exactly where the trajectory gave us a clue. We experimented with different approaches and eventually added a warning to our documentation explaining that manual upgrades are likely to be incomplete and that agents should use the CLI instead.

Then we ran the eval again and looked at the trajectory again. A better score alone wouldn't verify that our intervention changed the behavior we intended to change. The loop is:

Before shipping an Agent Experience improvement, treat it as a hypothesis. The result tells you whether performance changed. The trajectory helps you understand whether it changed for the reason you expected. That's the evidence you need before turning an experiment into guidance for your users.

Improve the natural path

Once you start examining trajectories, you'll see plenty of opportunities to make them shorter. You could write a special instruction for one task or create a skill for another. But beware of chasing perfection!

Ideally, an agent should complete a task with as few unnecessary steps and tokens as possible, but the optimization has to remain practical. If every task requires a bespoke skill, your eval results might look fantastic while the experience bears little resemblance to what users encounter in practice. You optimized the test case instead of improving the product.

The goal is to improve the natural path agents take when working with your product. Deciding where that boundary lies requires taste, skepticism, empathy, and knowledge of your audience. Trajectory data can show you what happened, but it can't make every product decision for you.

Compare runs without ignoring anomalies

Agents are nondeterministic. Every token in their context can potentially alter what they do next, which is another reason not to treat a single trajectory as truth. You must compare trajectories across runs. A behavior that appears repeatedly is different from a one-off detour. By comparing runs you can distinguish patterns from anomalies and see how behavior changes when you modify documentation, a skill, an MCP server, the model, or another part of the environment.

But don't dismiss anomalies simply because they happened once. An unexpected tool call, source, or reasoning path is a lead. It might be irrelevant, or it might expose behavior you didn't know existed and give you something worth investigating across additional runs.

Automated checks can eventually help you catch behaviors you already know about. But they won't reliably show you behaviors you haven't thought to check for. That's exactly why inspecting trajectories remains valuable.

Make trajectories first-class eval artifacts

If you're building agent evals, store enough information to reconstruct what happened. Ideally, that includes the user's prompt and the agent's reasoning when available. Keep every tool call with its arguments and the responses returned by those tools. When possible, keep the agent's system prompt too. It can reveal environmental influences you might otherwise attribute to the model or your product: which tools were available, what workspace information was provided, OS-specific instructions, full paths included in context, or other biases introduced by the agent harness.

A trajectory isn't merely a debugging log. It's part of the evidence produced by your eval. Store it alongside the result and task cost. Something you didn't think was important at the time might later become exactly what you need to understand a behavior.

I prefer to store trajectories in Agent Trajectory Interchange Format (ATIF) and explore them using the ATIF Preview extension for VS Code.

ATIF gives different agents a common format, so I can compare their trajectories without first translating each agent's raw logs or events. It also makes trajectories easier to read. ATIF can link a parent trajectory to its subagent trajectories and distinguish utility calls, such as progress messages or generated session titles, from the work itself.

You don't necessarily need to inspect every trajectory forever. When you're running mature incremental evals and results continue to follow established patterns, there might be little reason to pore over every run. But when you're starting an evaluation, investigating a failure, changing your Agent Experience, or looking for ways to improve it, replay the run step by step.

Why did the agent choose this tool, and were the arguments sensible? Did it need that file or search? Could it have reached the same result with less information and fewer tokens? Answering those questions reveals both waste and missed opportunities. And don't look only for mistakes. Surprisingly good decisions show where the model already has enough knowledge and where adding more guidance might make the experience worse. A score compresses an entire run into a number. That's useful, but if you're trying to understand how agents experience your product, don't throw away everything that happened before that number. Look at the trajectory.

Is your eval lying to you?

Sat, 19 Sep 2026 15:34:24 GMT

When building evals for AI coding agents, string contains is incredibly tempting. It's fast and deterministic. And it costs practically nothing, so you can run it on every change in CI without worrying about inference cost or latency. But is using it making your eval lie to you?

Want to know whether the agent used Azure? Look for Azure, right? Want to know whether it used a particular API? Look for the API name. Want to know whether it created the right configuration? Check whether the expected string is in the file. Pass. Ship it. But you haven't established any of those things.

What did your grader prove?

Suppose your eval says that the generated solution must use Azure, and your grader checks whether the generated files contain the string Azure. The grader passes. What do you now know? Only that the string Azure appears somewhere in the generated files. That's it.

The string could appear in code that uses Azure, but it could also appear in a comment. After all, a single line is enough to satisfy your grader. Here is all it needs to see:

// Don't use Azure here.

That comment passes the check. So would dead code, an unused dependency, documentation, or a test fixture. A configuration file could contain the string even though the application never loads that file. Your grader finds what it's looking for, but the behavior you care about is still absent.

The same problem works in reverse. If the string isn't present, can you conclude that the solution doesn't use Azure? No. The agent could use an SDK without mentioning Azure explicitly, refer only to a package or class name, or express the same intent using words you didn't anticipate. Natural language makes this worse because the difference between did and didn't can be one word while the keyword remains exactly the same. A deterministic grader gives you a deterministic answer, but that answer only means what the check establishes.

Finish the sentence

There's a simple test I like for eval criteria. Complete one sentence for a passing result, then another for a failure. Be literal:

If this grader passes, I now know that ______.

If this grader fails, I now know that ______.

For a string contains "Azure" grader, the answers are narrow. They describe only what the grader observed. Anything more is an inference:

If this grader passes, I know the string Azure occurs in the inspected content.

If this grader fails, I know the string Azure doesn't occur in the inspected content.

If that's what you wanted to measure, great. But when you complete the first sentence with the agent correctly used Azure, you've made a leap your evidence doesn't support. The grader isn't necessarily inaccurate. It might be perfectly accurate at what it does, while we ask it to prove something it can't.

Easy evidence can give you false assurance

It's easy to see why people use simple deterministic graders. Agentic evals cost money and can take a long time to run. LLM judges add inference cost and latency. Building, running, or deploying generated applications adds infrastructure and complexity. In comparison, string contains takes milliseconds, which makes it ideal for CI... and potentially terrible evidence.

So you start optimizing an eval around what's convenient to measure instead of what you need to know. The result can be a fast, repeatable evaluation pipeline that gives you false assurance on every run. I'd rather run fewer evals that produce meaningful evidence than continuously run evals that confidently tell me very little.

Ask the system that can answer the question

If you want to know whether generated code builds, build it. We've seen solutions receive perfect scores from an LLM judge only to fail compilation. The judge could assess whether the implementation looked correct, but only the compiler could verify it.

If you want to establish whether dependencies can be restored, restore them. If you want to know whether a JSON file follows a schema, validate it. If you want to establish whether tests pass, run them. And where reasonably possible, if you want to establish whether the application works, run it.

This is where deterministic graders work well because the check directly establishes the property you care about. Using an LLM to predict whether a project will compile makes little sense when the compiler can answer authoritatively. You'd be asking an LLM to approximate an answer that an existing tool can give you.

But running generated applications is harder than building them. Real applications need credentials and configuration. They also need data, infrastructure, and access to external services. Deploying them adds another layer of complexity. Unfortunately, coding agents often struggle at those boundaries, which makes it tempting to verify something easier instead.

So instead, you check for the expected artifacts or words in the output. An LLM judge says the implementation looks plausible. Then you treat those observations as evidence that the integration works, even though none of them exercised it. Sometimes running the complete solution genuinely isn't practical, and that's okay. An eval doesn't need to prove everything, but its report should make clear what it did and didn't establish.

LLM judges answer different questions

Semantic criteria are different. An LLM judge can distinguish use Azure for storing the data from do not use Azure for storing the data. It can inspect surrounding code, reason about how components relate, and recognize implementations that don't contain the vocabulary you anticipated when writing the eval.

But an LLM judge doesn't eliminate the evidence problem. It can't reliably replace building and running the software. And even if it could reproduce every relevant compiler, package manager, test runner, or runtime check, why would you ask it to? We already have those tools. Use LLM judges for semantic questions and deterministic tools when they can answer a question directly.

Treat agentic evals like integration tests

When you evaluate an MCP server, skill, extension, or another capability for a coding agent, you care whether the agent can complete the task successfully. Finding a particular implementation detail in its output tells you far less. That's why I find it more useful to think about these evals as integration tests than unit tests.

In our evals, building, testing, running, and deploying are separate gates. Alongside them, LLM judges evaluate the semantic properties relevant to each scenario. Keeping the checks separate shows you whether dependency restoration failed, compilation broke, or a buildable implementation missed an important requirement. One broad proxy shouldn't have to answer all of these questions.

Still, deterministic graders are excellent when the property you're measuring is deterministic. The problem starts when you choose a convenient signal and quietly expand the claim you make from it. Before adding a grader to your next agentic eval, finish these two sentences:

If this grader passes, I now know that ______.

If this grader fails, I now know that ______.

Then compare those answers with what you intend to claim about your agent, skill, MCP server, or extension. If they don't match, find stronger evidence. Your eval doesn't become trustworthy because it runs on every commit. It becomes trustworthy when its evidence supports the claims you make from it.