<?xml version="1.0" encoding="UTF-8" standalone="no"?><rss xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" version="2.0">

<channel>
	<title>Deep Fried Bytes</title>
	<atom:link href="http://deepfriedbytes.com/feed/" rel="self" type="application/rss+xml"/>
	<link>https://deepfriedbytes.com/</link>
	<description>Deep Fried Bytes is an audio talk show with a Southern flavor hosted by technologists and developers Keith Elder and Chris Woodruff. The show discusses a wide range of topics including application development, operating systems and technology in general. Anything is fair game if it plugs into the wall or takes a battery.</description>
	<lastBuildDate>Thu, 19 Nov 2026 22:01:56 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1</generator>

<image>
	<url>https://deepfriedbytes.com/wp-content/uploads/2025/07/cropped-cropped-Deep-Fried-Bytes-32x32.png</url>
	<title>Blog about a digital future</title>
	<link>https://deepfriedbytes.com/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<itunes:explicit>no</itunes:explicit><copyright>2008 by Deep Fried Bytes, All rights reserved</copyright><itunes:image href="http://deepfriedbytes.com/images/deepfried_feedimage.png"/><itunes:keywords>technology,windows,apple,linux,osx,net,c,vb,net,home,server,ipod,zune,sql,server,programmer,developer</itunes:keywords><itunes:summary>Deep Fried Bytes is an audio talk show with a Southern flavor hosted by technologists and developers Keith Elder and Chris Woodruff. The show discusses a wide range of topics including application development, operating systems and technology in general. Anything is fair game if it plugs into the wall or takes a battery.</itunes:summary><itunes:subtitle>Everything tastes better deep fried, especially technology!</itunes:subtitle><itunes:category text="Technology"/><itunes:category text="Technology"><itunes:category text="Podcasting"/></itunes:category><itunes:category text="Technology"><itunes:category text="Gadgets"/></itunes:category><itunes:category text="Technology"><itunes:category text="Tech News"/></itunes:category><itunes:author>Keith Elder &amp; Chris Woodruff</itunes:author><itunes:owner><itunes:email>comments@deepfriedbytes.com</itunes:email><itunes:name>Keith Elder &amp; Chris Woodruff</itunes:name></itunes:owner><item>
		<title>Your GenAI Vendor Demo Hides Half the Project Scope</title>
		<link>https://deepfriedbytes.com/your-genai-vendor-demo-hides-half-the-project-scope/</link>
		
		
		<pubDate>Tue, 17 Nov 2026 12:24:05 +0000</pubDate>
				<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[AI Web Solutions]]></category>
		<category><![CDATA[API]]></category>
		<category><![CDATA[Software Costs]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/your-genai-vendor-demo-hides-half-the-project-scope/</guid>

					<description><![CDATA[<p>A GenAI vendor can produce a convincing answer in a demo and still leave your team with an unscoped retrieval project. I would buy the option that makes failures cheap to detect and correct, even if its answers look less polished in a sales call. For a product manager, the evaluation is not “Which model sounds best?” It is “What work remains ours after we sign?” The best demo is the one you can reproduce without the vendor Give every candidate the same small, permissioned corpus and the same questions before discussing architecture. A demo built on a vendor’s prepared documents tells you little about your own workload, because document structure, access rules, and update frequency determine much of the retrieval work. Ask your team to supply documents with awkward tables, conflicting revisions, missing answers, and two test tenants whose files must never cross. Make the vendor ingest that corpus while your team watches. Record who cleans files, splits text, assigns metadata, configures access control, and resolves failed imports. Those tasks belong in the project estimate even when the platform advertises “managed RAG,” because managed infrastructure does not make source documents consistent or permissions self-enforcing. Require an export of the processed chunks and their source identifiers; without it, moving providers later may mean repeating the least visible part of implementation. Use a fixed evaluation set: for example, reserve 80 questions as a proposed starting size, including answerable, unanswerable, and access-denied cases. Have a product owner approve the expected source for each answer. Score retrieval with recall@10 and NDCG@10, then score the final answer separately for factual support and appropriate refusal. A high retrieval score cannot excuse a fabricated conclusion, while a good answer on a few handpicked prompts cannot establish that the right evidence is consistently retrieved. I would not choose a framework from its chatbot screenshot, because the screenshot hides the ingestion decisions and error handling that determine the delivery schedule. Ask instead for the exact failing question, retrieved chunk IDs, prompt version, model identifier, and permission decision behind any disputed result. If a vendor cannot produce those artifacts from an ordinary request, your team will have to build its own diagnostic layer or spend support cycles guessing. The architectural argument in RAG vs Fine-Tuning in Your First GenAI Backend is a useful starting point, but it should not become a purchasing rule: a vendor must prove that its chosen approach handles your update and permission requirements. Price a re-index after a changed document, and ask how quickly that change becomes visible. A model that answers yesterday’s policy beautifully can still fail the product requirement if today’s correction is unavailable. Ownership of the retrieval layer determines the real migration cost For a first release, I would compare a managed retrieval service with a stack the team can inspect. Pinecone Serverless wins when the team needs a hosted vector service and does not want to operate index capacity; it costs usage-based service spend, integration work, and a migration plan for vectors and metadata. PostgreSQL 16 with pgvector 0.8.0 wins when the corpus and query load fit a database the team already operates and when SQL-level inspection matters; it costs database tuning, backup planning, and engineering time as the index grows. Neither choice removes the need to test filtering and ranking against your documents. Pin the evaluation environment rather than accepting a comparison between unlike defaults. For a pgvector trial, record the embedding model, distance metric, index type, and hnsw.ef_search; changing the last setting can change recall and query time, so it belongs beside the results. For a hosted service, record namespace design, metadata filters, and the index configuration supplied by the vendor. Demand the same filter semantics in both trials: a permission failure is a release blocker, not a small deduction from an accuracy score. Libraries create a different kind of ownership. LlamaIndex may shorten the path from documents to a retrieval prototype, while LangChain may suit a team already using its application components. Neither is the retrieval service itself. Ask the engineer proposing either one to show where parsers, embeddings, prompts, retries, and source IDs are configured, and to lock package versions in the trial. Otherwise, a product estimate may treat connector setup as the whole job while leaving document repair, evaluations, and operational support unassigned. Include security and data handling in the proof, not in a later questionnaire. Confirm whether the service accepts your OIDC-based identity flow, where tenant identity is enforced, how deletion propagates to indexes and caches, and whether prompts or documents are retained for training. A SOC 2 Type II report can support a controls review, but it cannot demonstrate that this particular integration prevents one tenant from retrieving another tenant’s text. Run that test yourself. For each option, put the residual work in a named column: ingestion, evaluation, access control, observability, incident response, and exit. If the vendor owns a task, specify the contract or product setting that makes it theirs. “The platform handles it” is not a scope assumption you can schedule against. A latency claim is credible only when it includes retrieval and failure Ask for end-to-end measurements at the application boundary. A model provider’s token-generation figure omits query embedding, permission filtering, retrieval, reranking, network hops, and citation assembly; those steps can dominate the user’s wait. The warning in Your GenAI Feature Scales Until Retrieval and Latency Collide deserves a procurement consequence: require evidence for the whole request path before accepting a latency commitment. Set an initial p95 target of 3 seconds as a value to tune with your product team, not as a universal benchmark. Measure time to first token as well as completed response time if the interface streams; a quick first token can make a slow answer feel responsive, but it does not help a workflow that needs the complete cited answer before continuing. Track HTTP 429 responses and timeouts alongside latency, because excluding failed requests makes a strained service appear faster. Require a stable trial endpoint that returns an answer and citation identifiers. The following Python probe runs with standard-library modules after you set RAG_URL and RAG_TOKEN. Its 40 calls and 5 workers are a deliberately small smoke-test setting, not a production load model: import concurrent.futures, json, math, os, time, urllib.request url, token = os.environ["RAG_URL"], os.environ["RAG_TOKEN"] def probe(_): body = json.dumps({"question": "What is the refund window?", "tenant_id": "test-a"}).encode() req = urllib.request.Request(url, body, {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}) start = time.perf_counter() with urllib.request.urlopen(req, timeout=15) as response: data = json.load(response) return time.perf_counter() - start, bool(data.get("citations")) with concurrent.futures.ThreadPoolExecutor(max_workers=5) as pool: results = list(pool.map(probe, range(40))) times = sorted(seconds for seconds, _ in results) print("p95_seconds", times[math.ceil(0.95 * len(times)) - 1]) print("citation_presence", sum(cited for _, cited in results) / len(results)) Agree the endpoint schema with each candidate first. Citation presence is only a wiring check: an evaluator must still open the cited source and decide whether it supports the answer. The script also does not record failures as results, so a serious trial should add exception counts rather than quietly discarding them. Use k6 for a repeatable load profile, and instrument the application with OpenTelemetry spans so retrieval, reranking, and generation have separate timings. A Prometheus histogram can expose p95 over time; keep the underlying request count visible so a quiet interval is not mistaken for improved performance. Test a warm index, a fresh document, and a degraded dependency. Those cases matter because caches, indexing delays, and retries change what a user sees even when the model and prompt are unchanged. Request the vendor’s rate-limit policy and retry guidance in writing, then estimate what your application must do when the service is unavailable: show an error, fall back to search results, or queue the request. Each behavior has a different product and engineering cost. A short paid trial should end in a scope decision, not a winner’s slide Run a paid trial with an explicit stop date; two weeks is a planning allowance to adjust to your team’s document readiness, not a promise that integration finishes in two weeks. Agree on the corpus, question set, tenant tests, expected traffic shape, and acceptance thresholds before the vendors see the results. Otherwise, a candidate can optimize for whichever examples were easiest to present. Score the trial in three ledgers. The first is user outcome: supported answers, correct refusals, and failures that a reviewer can reproduce. The second is operating behavior: end-to-end latency, errors, indexing delay, and the time needed to identify a bad source. The third is ownership: tasks your engineers must build, tasks the vendor contractually covers, and tasks nobody has accepted. Keep those ledgers separate because a strong answer score does not pay for an undocumented permission system. Ask each candidate for the cost per 1,000 successful, cited answers, including embeddings, storage, retrieval, generation, retries, and any required observability tier. This is a budgeting unit to calculate from the trial’s usage, not a vendor-published price. “Per request” can conceal the cost of retries and unusable answers; “successful, cited” ties the estimate more closely to the feature you intend to ship. Request an export and deletion exercise before approving the contract, because an exit plan that has never been tested is only a clause. Finish with a decision record that names the rejected option, its advantage, and the condition under which you would reconsider it. That record prevents a later team from treating the choice as an eternal verdict and gives the product manager a defensible change trigger. The vendor that wins should leave you with fewer unknown work items, not merely the highest-scoring demo. The first purchase decision is a test fixture Before booking another demo, commission a permissioned sample corpus and an approved question set with source-level expected answers. Give the same fixture to two candidates and require request traces, failed cases, and a written ownership map. That first deliverable lets you scope ingestion, evaluation, and support work even if neither vendor passes—and keeps a persuasive sales presentation from becoming your project plan.</p>
<p>The post <a href="https://deepfriedbytes.com/your-genai-vendor-demo-hides-half-the-project-scope/">Your GenAI Vendor Demo Hides Half the Project Scope</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A GenAI vendor can produce a convincing answer in a demo and still leave your team with an unscoped retrieval project. I would buy the option that makes failures cheap to detect and correct, even if its answers look less polished in a sales call. For a product manager, the evaluation is not “Which model sounds best?” It is “What work remains ours after we sign?”</p>
<h2>The best demo is the one you can reproduce without the vendor</h2>
<p>Give every candidate the same small, permissioned corpus and the same questions before discussing architecture. A demo built on a vendor’s prepared documents tells you little about your own workload, because document structure, access rules, and update frequency determine much of the retrieval work. Ask your team to supply documents with awkward tables, conflicting revisions, missing answers, and two test tenants whose files must never cross.</p>
<p>Make the vendor ingest that corpus while your team watches. Record who cleans files, splits text, assigns metadata, configures access control, and resolves failed imports. Those tasks belong in the project estimate even when the platform advertises “managed RAG,” because managed infrastructure does not make source documents consistent or permissions self-enforcing. Require an export of the processed chunks and their source identifiers; without it, moving providers later may mean repeating the least visible part of implementation.</p>
<p>Use a fixed evaluation set: for example, reserve <strong>80 questions</strong> as a proposed starting size, including answerable, unanswerable, and access-denied cases. Have a product owner approve the expected source for each answer. Score retrieval with <strong>recall@10</strong> and <strong>NDCG@10</strong>, then score the final answer separately for factual support and appropriate refusal. A high retrieval score cannot excuse a fabricated conclusion, while a good answer on a few handpicked prompts cannot establish that the right evidence is consistently retrieved.</p>
<p>I would not choose a framework from its chatbot screenshot, because the screenshot hides the ingestion decisions and error handling that determine the delivery schedule. Ask instead for the exact failing question, retrieved chunk IDs, prompt version, model identifier, and permission decision behind any disputed result. If a vendor cannot produce those artifacts from an ordinary request, your team will have to build its own diagnostic layer or spend support cycles guessing.</p>
<p>The architectural argument in <a href=/rag-vs-fine-tuning-in-your-first-genai-backend/>RAG vs Fine-Tuning in Your First GenAI Backend</a> is a useful starting point, but it should not become a purchasing rule: a vendor must prove that its chosen approach handles your update and permission requirements. Price a re-index after a changed document, and ask how quickly that change becomes visible. A model that answers yesterday’s policy beautifully can still fail the product requirement if today’s correction is unavailable.</p>
<h2>Ownership of the retrieval layer determines the real migration cost</h2>
<p>For a first release, I would compare a managed retrieval service with a stack the team can inspect. <strong>Pinecone Serverless</strong> wins when the team needs a hosted vector service and does not want to operate index capacity; it costs usage-based service spend, integration work, and a migration plan for vectors and metadata. <strong>PostgreSQL 16 with pgvector 0.8.0</strong> wins when the corpus and query load fit a database the team already operates and when SQL-level inspection matters; it costs database tuning, backup planning, and engineering time as the index grows. Neither choice removes the need to test filtering and ranking against your documents.</p>
<p>Pin the evaluation environment rather than accepting a comparison between unlike defaults. For a pgvector trial, record the embedding model, distance metric, index type, and <strong>hnsw.ef_search</strong>; changing the last setting can change recall and query time, so it belongs beside the results. For a hosted service, record namespace design, metadata filters, and the index configuration supplied by the vendor. Demand the same filter semantics in both trials: a permission failure is a release blocker, not a small deduction from an accuracy score.</p>
<p>Libraries create a different kind of ownership. <strong>LlamaIndex</strong> may shorten the path from documents to a retrieval prototype, while <strong>LangChain</strong> may suit a team already using its application components. Neither is the retrieval service itself. Ask the engineer proposing either one to show where parsers, embeddings, prompts, retries, and source IDs are configured, and to lock package versions in the trial. Otherwise, a product estimate may treat connector setup as the whole job while leaving document repair, evaluations, and operational support unassigned.</p>
<p>Include security and data handling in the proof, not in a later questionnaire. Confirm whether the service accepts your <strong>OIDC</strong>-based identity flow, where tenant identity is enforced, how deletion propagates to indexes and caches, and whether prompts or documents are retained for training. A <strong>SOC 2 Type II</strong> report can support a controls review, but it cannot demonstrate that this particular integration prevents one tenant from retrieving another tenant’s text. Run that test yourself.</p>
<p>For each option, put the residual work in a named column: ingestion, evaluation, access control, observability, incident response, and exit. If the vendor owns a task, specify the contract or product setting that makes it theirs. “The platform handles it” is not a scope assumption you can schedule against.</p>
<h2>A latency claim is credible only when it includes retrieval and failure</h2>
<p>Ask for end-to-end measurements at the application boundary. A model provider’s token-generation figure omits query embedding, permission filtering, retrieval, reranking, network hops, and citation assembly; those steps can dominate the user’s wait. The warning in <a href=/your-genai-feature-scales-until-retrieval-and-latency-collide/>Your GenAI Feature Scales Until Retrieval and Latency Collide</a> deserves a procurement consequence: require evidence for the whole request path before accepting a latency commitment.</p>
<p>Set an initial <strong>p95 target of 3 seconds</strong> as a value to tune with your product team, not as a universal benchmark. Measure time to first token as well as completed response time if the interface streams; a quick first token can make a slow answer feel responsive, but it does not help a workflow that needs the complete cited answer before continuing. Track <strong>HTTP 429</strong> responses and timeouts alongside latency, because excluding failed requests makes a strained service appear faster.</p>
<p>Require a stable trial endpoint that returns an answer and citation identifiers. The following Python probe runs with standard-library modules after you set <strong>RAG_URL</strong> and <strong>RAG_TOKEN</strong>. Its <strong>40 calls</strong> and <strong>5 workers</strong> are a deliberately small smoke-test setting, not a production load model:</p>
<pre>import concurrent.futures, json, math, os, time, urllib.request
url, token = os.environ["RAG_URL"], os.environ["RAG_TOKEN"]
def probe(_):
    body = json.dumps({"question": "What is the refund window?", "tenant_id": "test-a"}).encode()
    req = urllib.request.Request(url, body, {"Authorization": f"Bearer {token}", "Content-Type": "application/json"})
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=15) as response:
        data = json.load(response)
    return time.perf_counter() - start, bool(data.get("citations"))
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as pool:
    results = list(pool.map(probe, range(40)))
times = sorted(seconds for seconds, _ in results)
print("p95_seconds", times[math.ceil(0.95 * len(times)) - 1])
print("citation_presence", sum(cited for _, cited in results) / len(results))</pre>
<p>Agree the endpoint schema with each candidate first. Citation presence is only a wiring check: an evaluator must still open the cited source and decide whether it supports the answer. The script also does not record failures as results, so a serious trial should add exception counts rather than quietly discarding them. Use <strong>k6</strong> for a repeatable load profile, and instrument the application with <strong>OpenTelemetry</strong> spans so retrieval, reranking, and generation have separate timings. A <strong>Prometheus</strong> histogram can expose p95 over time; keep the underlying request count visible so a quiet interval is not mistaken for improved performance.</p>
<p>Test a warm index, a fresh document, and a degraded dependency. Those cases matter because caches, indexing delays, and retries change what a user sees even when the model and prompt are unchanged. Request the vendor’s rate-limit policy and retry guidance in writing, then estimate what your application must do when the service is unavailable: show an error, fall back to search results, or queue the request. Each behavior has a different product and engineering cost.</p>
<h2>A short paid trial should end in a scope decision, not a winner’s slide</h2>
<p>Run a paid trial with an explicit stop date; <strong>two weeks</strong> is a planning allowance to adjust to your team’s document readiness, not a promise that integration finishes in two weeks. Agree on the corpus, question set, tenant tests, expected traffic shape, and acceptance thresholds before the vendors see the results. Otherwise, a candidate can optimize for whichever examples were easiest to present.</p>
<p>Score the trial in three ledgers. The first is user outcome: supported answers, correct refusals, and failures that a reviewer can reproduce. The second is operating behavior: end-to-end latency, errors, indexing delay, and the time needed to identify a bad source. The third is ownership: tasks your engineers must build, tasks the vendor contractually covers, and tasks nobody has accepted. Keep those ledgers separate because a strong answer score does not pay for an undocumented permission system.</p>
<p>Ask each candidate for the cost per <strong>1,000 successful, cited answers</strong>, including embeddings, storage, retrieval, generation, retries, and any required observability tier. This is a budgeting unit to calculate from the trial’s usage, not a vendor-published price. “Per request” can conceal the cost of retries and unusable answers; “successful, cited” ties the estimate more closely to the feature you intend to ship. Request an export and deletion exercise before approving the contract, because an exit plan that has never been tested is only a clause.</p>
<p>Finish with a decision record that names the rejected option, its advantage, and the condition under which you would reconsider it. That record prevents a later team from treating the choice as an eternal verdict and gives the product manager a defensible change trigger. The vendor that wins should leave you with fewer unknown work items, not merely the highest-scoring demo.</p>
<h2>The first purchase decision is a test fixture</h2>
<p>Before booking another demo, commission a permissioned sample corpus and an approved question set with source-level expected answers. Give the same fixture to two candidates and require request traces, failed cases, and a written ownership map. That first deliverable lets you scope ingestion, evaluation, and support work even if neither vendor passes—and keeps a persuasive sales presentation from becoming your project plan.</p>
<p>The post <a href="https://deepfriedbytes.com/your-genai-vendor-demo-hides-half-the-project-scope/">Your GenAI Vendor Demo Hides Half the Project Scope</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>At 20 Engineers, Vision Accuracy Alone Is the Wrong KPI</title>
		<link>https://deepfriedbytes.com/at-20-engineers-vision-accuracy-alone-is-the-wrong-kpi/</link>
		
		
		<pubDate>Tue, 17 Nov 2026 10:30:34 +0000</pubDate>
				<category><![CDATA[Uncategorized]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/at-20-engineers-vision-accuracy-alone-is-the-wrong-kpi/</guid>

					<description><![CDATA[<p>A vision model can improve on a test set while making the product worse: it may automate easy cases and send the expensive mistakes straight to users. For a tech lead with a small engineering team, I would make the release decision on audited outcomes per eligible request, not model accuracy. If you cannot connect a prediction to an action, a later label, and its cost, you cannot tell whether the deployment works. The release gate should measure harmful actions, not correct predictions Start with the decision your application makes. Suppose a camera check either accepts an image automatically or sends it to a reviewer. “The model was correct” is too broad: a wrong automatic acceptance may require a costly correction, while a wrong referral to review mainly consumes staff time. I would define the primary metric as harmful automatic acceptances per 1,000 eligible requests, with review minutes and p95 response latency as guardrails. This makes the denominator include requests the system declines to automate, so the team cannot improve its apparent error rate simply by reviewing everything. Write down what counts as harm before looking at model scores. For example, an automatic acceptance might be harmful if a trained reviewer would reject it under the current policy. Record the action taken, the policy version, and the reviewer’s reason. If a downstream complaint can overturn that judgment, retain both labels and the date of the correction. Otherwise, a delayed complaint will appear to be a new incident rather than evidence that the original decision was wrong. A starting constraint could be fewer than 2 harmful acceptances per 1,000 eligible requests; that is a value to negotiate with the product owner, not a universal safety threshold. Pair it with a review-capacity limit—for example, an initial ceiling of 25% of eligible requests—because a model that avoids errors by transferring all work to people has not improved the service. Set a latency budget separately: a provisional 600 ms p95 may be appropriate only if the calling application can tolerate it. These figures should be changed when the cost of a wrong action, reviewer capacity, or request deadline changes. I would not gate this release on a single validation-set precision score, because its class balance and image quality need not match the requests that reach production. Nor would I call a lower review rate a win without auditing accepted cases: review avoidance is exactly where an overconfident model can hide damage. Read the adjudication log first, then read How to prove your vision model works in production metrics. The log, rather than any general metric recipe, has to settle what this team means by success. An auditable decision stream is more useful than another model dashboard Give every eligible request a stable request ID and emit an event whether inference succeeds, times out, falls back, or never starts. An OpenTelemetry trace should connect image intake, model call, policy decision, and response; attributes such as service.name and deployment.environment.name distinguish deployments. Store the model identifier, policy version, confidence or score, decision, review reason, and eventual label against that ID. Record an image reference or restricted hash rather than copying raw images into general-purpose telemetry, because broad access to diagnostic data creates a second, unnecessary exposure path. Prometheus histograms can track inference and end-to-end latency using the le bucket label; Grafana can show p95 latency beside timeout and fallback counts. Those tools cannot tell you whether accepted images were correct. Keep adjudications in a queryable ledger—PostgreSQL is sufficient at this scale—and export joined events to Parquet when an audit needs cohort analysis in DuckDB. Use CVAT to manage reviewer annotations and export COCO-format labels where the task fits that format. Put annotation-guideline versions alongside labels, because a changed guideline can look like model drift even when the images have not changed. The following Python snippet runs as written. Its rows are illustrative, not measured production results; replace the embedded CSV with an export of adjudicated decisions. It deliberately reports each cohort separately rather than hiding a bad cohort in an average. import csv from io import StringIO data = """decision,correct,cohort accept,1,night accept,0,night review,1,day accept,1,day review,0,night""" rows = list(csv.DictReader(StringIO(data))) for cohort in sorted({r["cohort"] for r in rows}): s = [r for r in rows if r["cohort"] == cohort] a = [r for r in s if r["decision"] == "accept"] print(cohort, "false_accept", sum(r["correct"] == "0" for r in a) / len(a) if a else None, "review", sum(r["decision"] == "review" for r in s) / len(s)) The snippet exposes a trap rather than solving sampling. If reviewers label only referred cases, accepted cases have no trustworthy “correct” field. Randomly sample automatic acceptances for blinded adjudication, and keep the sampling probability so an analyst can reconstruct production rates. Reserve a planning budget of 400 independently sampled accepted cases per major cohort before treating a zero-error result as reassuring; even zero errors in 400 observations gives only an approximate two-sided 95% upper bound of 0.75% under independent sampling. That bound does not repair correlated images, inconsistent reviewers, or a cohort defined after inspecting failures. Use scikit-learn’s precision_recall_curve for threshold exploration, but calculate the release metric from the sampled decision ledger, because threshold curves alone do not include operational fallbacks or review cost. A pretrained API wins only if it wins under the same decision policy Compare a named pretrained vision API with a named custom deployment on the same incoming images, but do not compare their raw confidence numbers: the scores need not share a calibration scale. For a concrete choice, put Google Cloud Vision API against a PyTorch model served through ONNX Runtime. The API can win when labeled data is scarce and its audited decisions meet the harm and latency limits; its costs include per-request charges, network dependence, and the work of checking vendor behavior after changes. The custom deployment can win when the team has representative labels and can materially reduce harmful decisions at its required volume; its costs include training, serving, annotation, and ownership of every regression. Run both candidates in shadow mode on identical requests while the existing policy still makes the user-visible decision. Give each candidate its own threshold, selected on an earlier labeled set, but enforce the same maximum review fraction and action definitions during evaluation. Otherwise, a candidate can appear safer merely because it sends more work to reviewers. Log API timeouts, malformed responses, and custom-serving failures as outcomes, not missing rows. Use MLflow Model Registry aliases such as @champion to identify the custom artifact under test, and pin the API request configuration and observed provider response metadata so a later comparison does not silently mix versions. A proposed trial might shadow 10% of eligible traffic for seven days; treat that as a tunable coverage and duration choice, not evidence that seven days captures every rare condition. Check whether that interval includes the lighting, device, and input-quality cohorts that drive failures. Compare candidates on paired requests, then inspect cases where their proposed actions disagree. A difference in average accuracy matters less than which candidate automatically accepts the cases a reviewer would reject. Test both candidates on the same inputs before you read Custom Vision Model or Pretrained API Which Fits Your First App; a first-app preference does not establish a production winner without paired outcomes. Price the comparison in the unit the organisation can act on: expected correction cost plus reviewer minutes and serving cost per 1,000 eligible requests. Include engineering time for maintaining the custom model, but report it separately from request-variable cost, because combining them into one opaque number prevents a useful volume forecast. If the API and custom model are statistically indistinguishable on harmful actions, I would keep the simpler deployment until a larger audit can justify switching; a claimed improvement smaller than the label uncertainty is a poor reason to add operational ownership. A rollout works only if it improves on the policy it replaces Shadow results test predictions, not the full intervention. Once the team has an audit stream and a safe fallback, randomly assign a small eligible slice to the new decision policy and leave a comparable slice on the existing one. Keep request assignment stable for repeated images or users where feasible, because switching policies between related requests can contaminate both groups. Measure harmful actions per eligible request, review minutes, latency, fallbacks, and corrections in both groups. Include requests that timed out or bypassed inference in the assigned group’s denominator; excluding them would reward an unreliable deployment. Set the label-lag rule in advance. An example operating rule is to wait two business days for routine adjudications before closing a daily comparison, while continuing to update it for later reversals. That interval is an assumption to check against the team’s actual review queue, not a reason to discard slow-arriving complaints. Report counts and uncertainty by cohort as well as overall results. A dashboard that shows a favorable aggregate while night images deteriorate should block expansion if night images are part of the promised service. Give the on-call engineer a reversible control: route decisions back to review or the old policy without redeploying the model. Alert on a breached harm estimate only when enough audited labels exist to support it; alert immediately on missing events, fallback spikes, or exhausted review capacity, because those failures can be observed without waiting for labels. Evidently can flag input-distribution changes for investigation, but drift is a diagnostic rather than a verdict: a changed input distribution matters to this release when it changes audited decisions or makes the audit sample unrepresentative. Precommit the stop conditions with the people who own corrections. If harmful acceptances exceed the agreed limit, if review demand exceeds staffed capacity, or if adjudication coverage falls below the level needed to judge either, pause expansion. “We have not seen errors” is not evidence of success when accepted cases are not being checked. The rollout has worked when the new policy produces a defensible improvement against the old one at an acceptable operating cost—not when its model score looks better in isolation. The first task is to price one wrong action Before selecting another model, take one recent batch of eligible requests and have the product owner and a reviewer agree on what an incorrect automatic action costs. Write that definition into a decision-event schema, then sample accepted cases for independent adjudication. If those two steps cannot produce a credible baseline harm rate, keep the current policy: there is no sound release comparison yet.</p>
<p>The post <a href="https://deepfriedbytes.com/at-20-engineers-vision-accuracy-alone-is-the-wrong-kpi/">At 20 Engineers, Vision Accuracy Alone Is the Wrong KPI</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A vision model can improve on a test set while making the product worse: it may automate easy cases and send the expensive mistakes straight to users. For a tech lead with a small engineering team, I would make the release decision on audited outcomes per eligible request, not model accuracy. If you cannot connect a prediction to an action, a later label, and its cost, you cannot tell whether the deployment works.</p>
<h2>The release gate should measure harmful actions, not correct predictions</h2>
<p>Start with the decision your application makes. Suppose a camera check either accepts an image automatically or sends it to a reviewer. “The model was correct” is too broad: a wrong automatic acceptance may require a costly correction, while a wrong referral to review mainly consumes staff time. I would define the primary metric as harmful automatic acceptances per 1,000 eligible requests, with review minutes and p95 response latency as guardrails. This makes the denominator include requests the system declines to automate, so the team cannot improve its apparent error rate simply by reviewing everything.</p>
<p>Write down what counts as harm before looking at model scores. For example, an automatic acceptance might be harmful if a trained reviewer would reject it under the current policy. Record the action taken, the policy version, and the reviewer’s reason. If a downstream complaint can overturn that judgment, retain both labels and the date of the correction. Otherwise, a delayed complaint will appear to be a new incident rather than evidence that the original decision was wrong.</p>
<p>A starting constraint could be fewer than <strong>2 harmful acceptances per 1,000 eligible requests</strong>; that is a value to negotiate with the product owner, not a universal safety threshold. Pair it with a review-capacity limit—for example, an initial ceiling of <strong>25%</strong> of eligible requests—because a model that avoids errors by transferring all work to people has not improved the service. Set a latency budget separately: a provisional <strong>600 ms p95</strong> may be appropriate only if the calling application can tolerate it. These figures should be changed when the cost of a wrong action, reviewer capacity, or request deadline changes.</p>
<p>I would <strong>not</strong> gate this release on a single validation-set precision score, because its class balance and image quality need not match the requests that reach production. Nor would I call a lower review rate a win without auditing accepted cases: review avoidance is exactly where an overconfident model can hide damage. Read the adjudication log first, then read <a href=/how-to-prove-your-vision-model-works-in-production-metrics/>How to prove your vision model works in production metrics</a>. The log, rather than any general metric recipe, has to settle what this team means by success.</p>
<h2>An auditable decision stream is more useful than another model dashboard</h2>
<p>Give every eligible request a stable request ID and emit an event whether inference succeeds, times out, falls back, or never starts. An OpenTelemetry trace should connect image intake, model call, policy decision, and response; attributes such as <strong>service.name</strong> and <strong>deployment.environment.name</strong> distinguish deployments. Store the model identifier, policy version, confidence or score, decision, review reason, and eventual label against that ID. Record an image reference or restricted hash rather than copying raw images into general-purpose telemetry, because broad access to diagnostic data creates a second, unnecessary exposure path.</p>
<p>Prometheus histograms can track inference and end-to-end latency using the <strong>le</strong> bucket label; Grafana can show p95 latency beside timeout and fallback counts. Those tools cannot tell you whether accepted images were correct. Keep adjudications in a queryable ledger—PostgreSQL is sufficient at this scale—and export joined events to Parquet when an audit needs cohort analysis in DuckDB. Use CVAT to manage reviewer annotations and export COCO-format labels where the task fits that format. Put annotation-guideline versions alongside labels, because a changed guideline can look like model drift even when the images have not changed.</p>
<p>The following Python snippet runs as written. Its rows are illustrative, not measured production results; replace the embedded CSV with an export of adjudicated decisions. It deliberately reports each cohort separately rather than hiding a bad cohort in an average.</p>
<pre>import csv
from io import StringIO
data = """decision,correct,cohort
accept,1,night
accept,0,night
review,1,day
accept,1,day
review,0,night"""
rows = list(csv.DictReader(StringIO(data)))
for cohort in sorted({r["cohort"] for r in rows}):
    s = [r for r in rows if r["cohort"] == cohort]
    a = [r for r in s if r["decision"] == "accept"]
    print(cohort, "false_accept", sum(r["correct"] == "0" for r in a) / len(a) if a else None,
          "review", sum(r["decision"] == "review" for r in s) / len(s))</pre>
<p>The snippet exposes a trap rather than solving sampling. If reviewers label only referred cases, accepted cases have no trustworthy “correct” field. Randomly sample automatic acceptances for blinded adjudication, and keep the sampling probability so an analyst can reconstruct production rates. Reserve a planning budget of <strong>400 independently sampled accepted cases per major cohort</strong> before treating a zero-error result as reassuring; even zero errors in 400 observations gives only an approximate two-sided 95% upper bound of 0.75% under independent sampling. That bound does not repair correlated images, inconsistent reviewers, or a cohort defined after inspecting failures. Use scikit-learn’s <strong>precision_recall_curve</strong> for threshold exploration, but calculate the release metric from the sampled decision ledger, because threshold curves alone do not include operational fallbacks or review cost.</p>
<h2>A pretrained API wins only if it wins under the same decision policy</h2>
<p>Compare a named pretrained vision API with a named custom deployment on the same incoming images, but do not compare their raw confidence numbers: the scores need not share a calibration scale. For a concrete choice, put <strong>Google Cloud Vision API</strong> against a <strong>PyTorch</strong> model served through <strong>ONNX Runtime</strong>. The API can win when labeled data is scarce and its audited decisions meet the harm and latency limits; its costs include per-request charges, network dependence, and the work of checking vendor behavior after changes. The custom deployment can win when the team has representative labels and can materially reduce harmful decisions at its required volume; its costs include training, serving, annotation, and ownership of every regression.</p>
<p>Run both candidates in shadow mode on identical requests while the existing policy still makes the user-visible decision. Give each candidate its own threshold, selected on an earlier labeled set, but enforce the same maximum review fraction and action definitions during evaluation. Otherwise, a candidate can appear safer merely because it sends more work to reviewers. Log API timeouts, malformed responses, and custom-serving failures as outcomes, not missing rows. Use MLflow Model Registry aliases such as <strong>@champion</strong> to identify the custom artifact under test, and pin the API request configuration and observed provider response metadata so a later comparison does not silently mix versions.</p>
<p>A proposed trial might shadow <strong>10% of eligible traffic for seven days</strong>; treat that as a tunable coverage and duration choice, not evidence that seven days captures every rare condition. Check whether that interval includes the lighting, device, and input-quality cohorts that drive failures. Compare candidates on paired requests, then inspect cases where their proposed actions disagree. A difference in average accuracy matters less than which candidate automatically accepts the cases a reviewer would reject. Test both candidates on the same inputs before you read <a href=/custom-vision-model-or-pretrained-api-which-fits-your-first-app/>Custom Vision Model or Pretrained API Which Fits Your First App</a>; a first-app preference does not establish a production winner without paired outcomes.</p>
<p>Price the comparison in the unit the organisation can act on: expected correction cost plus reviewer minutes and serving cost per 1,000 eligible requests. Include engineering time for maintaining the custom model, but report it separately from request-variable cost, because combining them into one opaque number prevents a useful volume forecast. If the API and custom model are statistically indistinguishable on harmful actions, I would keep the simpler deployment until a larger audit can justify switching; a claimed improvement smaller than the label uncertainty is a poor reason to add operational ownership.</p>
<h2>A rollout works only if it improves on the policy it replaces</h2>
<p>Shadow results test predictions, not the full intervention. Once the team has an audit stream and a safe fallback, randomly assign a small eligible slice to the new decision policy and leave a comparable slice on the existing one. Keep request assignment stable for repeated images or users where feasible, because switching policies between related requests can contaminate both groups. Measure harmful actions per eligible request, review minutes, latency, fallbacks, and corrections in both groups. Include requests that timed out or bypassed inference in the assigned group’s denominator; excluding them would reward an unreliable deployment.</p>
<p>Set the label-lag rule in advance. An example operating rule is to wait <strong>two business days</strong> for routine adjudications before closing a daily comparison, while continuing to update it for later reversals. That interval is an assumption to check against the team’s actual review queue, not a reason to discard slow-arriving complaints. Report counts and uncertainty by cohort as well as overall results. A dashboard that shows a favorable aggregate while night images deteriorate should block expansion if night images are part of the promised service.</p>
<p>Give the on-call engineer a reversible control: route decisions back to review or the old policy without redeploying the model. Alert on a breached harm estimate only when enough audited labels exist to support it; alert immediately on missing events, fallback spikes, or exhausted review capacity, because those failures can be observed without waiting for labels. Evidently can flag input-distribution changes for investigation, but drift is a diagnostic rather than a verdict: a changed input distribution matters to this release when it changes audited decisions or makes the audit sample unrepresentative.</p>
<p>Precommit the stop conditions with the people who own corrections. If harmful acceptances exceed the agreed limit, if review demand exceeds staffed capacity, or if adjudication coverage falls below the level needed to judge either, pause expansion. “We have not seen errors” is not evidence of success when accepted cases are not being checked. The rollout has worked when the new policy produces a defensible improvement against the old one at an acceptable operating cost—not when its model score looks better in isolation.</p>
<h2>The first task is to price one wrong action</h2>
<p>Before selecting another model, take one recent batch of eligible requests and have the product owner and a reviewer agree on what an incorrect automatic action costs. Write that definition into a decision-event schema, then sample accepted cases for independent adjudication. If those two steps cannot produce a credible baseline harm rate, keep the current policy: there is no sound release comparison yet.</p>
<p>The post <a href="https://deepfriedbytes.com/at-20-engineers-vision-accuracy-alone-is-the-wrong-kpi/">At 20 Engineers, Vision Accuracy Alone Is the Wrong KPI</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>The AI Vision Maintenance Tax Solo Freelancers Miss</title>
		<link>https://deepfriedbytes.com/the-ai-vision-maintenance-tax-solo-freelancers-miss/</link>
		
		
		<pubDate>Wed, 11 Nov 2026 07:17:36 +0000</pubDate>
				<category><![CDATA[AI Computer Vision]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[API]]></category>
		<category><![CDATA[Computer Vision]]></category>
		<category><![CDATA[Machine Learning Apps]]></category>
		<category><![CDATA[Software Engineering]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/the-ai-vision-maintenance-tax-solo-freelancers-miss/</guid>

					<description><![CDATA[<p>A vision feature can be cheap to launch and expensive to keep honest. For a solo freelancer, the recurring work is not usually retraining; it is finding out that yesterday’s camera, lighting, API response, or customer expectations changed. I would choose a vision system whose failures I can inspect and reverse myself, even if a more sophisticated model wins the initial demo. The first model choice matters less than the exit path I treat the first-app comparison, Custom Vision Model or Pretrained API Which Fits Your First App, as a launch checklist rather than a maintenance plan. The choice between a pretrained API and a custom model matters, but neither choice tells you who will diagnose a bad result on a Tuesday night. If the client cannot show you the input, the version that processed it, and the decision that followed, you have inherited an incident without evidence. A pretrained vision API wins when labels are scarce and its existing categories closely match the job, because you can test a useful workflow before collecting a training set. It costs per request, exposes you to provider changes, and may impose retention or data-transfer terms you need to check. A custom model run with ONNX Runtime wins when the task is narrow, you have permission to retain representative examples, and an API’s outputs repeatedly miss the same requirement. It costs annotation time, evaluation time, and responsibility for serving a pinned model file. Those are different maintenance bills, not a free-versus-paid choice. My default would be a pretrained API behind one small adapter, with a manual-review path that remains usable if the API fails. I would not train a custom model merely to avoid an API bill, because a small monthly invoice is easier for one person to manage than an unmeasured labeling and retraining obligation. I would change that position only after saved examples show a persistent error the API cannot be configured to avoid. Make the adapter return your own stable fields, such as decision, confidence, provider, and model_version, rather than passing the provider’s full JSON response into the app. That boundary limits a provider migration to one place because the rest of the code depends on your contract. Pin the provider endpoint or model identifier where the service permits it, and keep a small set of approved test images; a green HTTP response proves transport worked, not that the answer stayed the same. There is also an exit cost in the images themselves. If your examples contain client data, confirm who may keep them before building a training archive, because a model you cannot legally or contractually retest is difficult to maintain. Record only what you need, set a deletion date, and keep a SHA-256 hash when you need to tell whether two files are identical without keeping duplicate copies. Every accepted prediction creates a small support obligation The tempting launch metric is “the model was correct on our test set.” The maintenance question is narrower: which mistakes reach the customer without review? A false positive that merely opens a review screen has a different cost from a false positive that sends an irreversible notification. Set the review threshold around the action’s cost, because the same model score can be acceptable for one action and unacceptable for another. Start with an explicit record for each decision: a timestamp, a pseudonymous input ID, the model or API version, the score, the threshold in force, the action taken, and whether a human later corrected it. SQLite with PRAGMA journal_mode=WAL is sufficient for a single-process pilot because it makes local writes and later inspection straightforward. If multiple workers will write to the same database across machines, use a managed database instead, because a shared SQLite file is the wrong coordination mechanism. Do not confuse a confidence score with a probability of being right. If a service returns 0.90, that number may be a ranking signal rather than a calibrated 90% chance, because providers and model architectures define scores differently. The threshold must be chosen against labeled examples from the actual workflow. A provisional threshold of 0.85 is a value to tune, not a recommendation to ship: compare errors above and below it before allowing automatic acceptance. A workable solo routine is to review a fixed sample of accepted decisions and all customer-reported mistakes each week. A sample of 30 reviewed accepted decisions is an operational starting count, not a statistically sufficient guarantee; small or changing traffic can make its error rate swing sharply. Log the denominator alongside the rate so “zero errors” cannot disguise “zero labels.” Include some low-confidence decisions in the review queue as well, because examining only accepted outputs will hide cases the system repeatedly sends to manual work. Keep the review record deliberately plain. A CSV export can carry IDs, scores, outcomes, and model versions; a separate, access-controlled folder can hold the few images you have permission to retain. Strip unnecessary EXIF metadata before sharing examples, because camera locations and timestamps can reveal information unrelated to the task. This is less elegant than an annotation platform, but it is easier to audit when you are the only operator. Monitoring must tell you when to stop trusting automation I use the production-metrics guide, How to prove your vision model works in production metrics, as a starting point for a stronger rule: a metric earns its place only if it changes what I do next. A dashboard of request counts cannot tell you whether accepted predictions are wrong, because correctness usually arrives later through review or complaint. Write the pause rule before launch so you are not inventing one while a client waits for an answer. The following Python 3.11 smoke test calculates the error rate among reviewed, automatically accepted examples. It runs as written and intentionally refuses to call a four-example sample healthy. Replace the inline rows with a reviewed CSV export before using it for an operational decision. python3 - &#60;&#60;'PY' import csv, io rows = csv.DictReader(io.StringIO("""score,correct 0.93,1 0.88,0 0.94,0 0.89,0""")) accepted = [r for r in rows if float(r["score"]) &#62;= 0.85] n = len(accepted) rate = sum(r["correct"] == "0" for r in accepted) / n if n else 0 print(f"sample={n} error={rate:.1%}") print("REVIEW" if n &#62;= 30 and rate &#62; 0.10 else "INSUFFICIENT LABELS" if n &#60; 30 else "OK") PY The 10% pause boundary is an example policy to set with the client, not evidence that a 9% error rate is safe. The script also needs an important production companion: report how many requests bypassed review, because an apparently good reviewed sample may not represent the requests the model accepted. If corrections arrive days late, calculate outcomes by decision date and mark recent cohorts incomplete; otherwise, late labels can make a new version appear better than it is. Track availability separately from correctness. Log HTTP status codes and latency at the adapter, and honor HTTP 429 and the Retry-After header rather than retrying immediately, because a burst of retries can prolong a provider rate limit. In an illustrative pilot, a measured p95 latency of 420 ms across 200 requests would be a baseline for that pilot only; it would not predict a future client’s traffic. If requests start timing out, fall back to manual review rather than silently treating missing predictions as negative results. I would not deploy Prometheus and OpenTelemetry just to watch one low-traffic freelancer project, because maintaining collectors and dashboards can consume more time than reading a daily export. A scheduled check, an emailed failure notice, and a weekly review appointment are enough until missed incidents show that the simpler arrangement is failing. A rollback plan saves more time than a retraining plan Many vision regressions start outside the model. A phone update can change image dimensions; an upload path can rotate pictures differently; a provider can change response fields. Keep a compact fixture set with expected decisions and image properties, then run it with pytest against the adapter before every release. Store the version of the adapter alongside the model identifier, because “same model” does not imply “same preprocessing.” For a self-hosted model, pin the ONNX file and its checksum, the ONNX Runtime version, and any preprocessing constants such as resize dimensions and color order. A Docker image pinned by digest helps reproduce the software environment because a mutable tag can point to different contents later. It does not preserve the customer’s input distribution, so keep separately approved examples that exercise glare, rotation, blur, and the kinds of mistakes reviewers have actually seen. For an API, save redacted request and response examples and test your adapter against them. Disable automatic acceptance with a configuration switch that does not require retraining or redeployment, because a freelancer needs a safe response even when the provider is unavailable. Tell the client what the manual path costs in time. An honest delay is preferable to an automatic answer you cannot validate. Keep the rollback small enough to rehearse: restore the previous adapter release, restore its threshold, process one known fixture, and confirm that new requests enter the intended review path. Run that rehearsal after a change rather than assuming a saved model file is a rollback; configuration, preprocessing, and routing can all change the result while the weights stay identical. The first maintenance task is to write the pause rule Before signing off on a demo, take ten recent inputs you are allowed to retain and record the output, version, threshold, and human verdict for each. Then write down who switches off automatic acceptance, what evidence triggers that switch, and how work continues manually. If that procedure cannot fit on one page, simplify the feature before promising the client it will run unattended.</p>
<p>The post <a href="https://deepfriedbytes.com/the-ai-vision-maintenance-tax-solo-freelancers-miss/">The AI Vision Maintenance Tax Solo Freelancers Miss</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A vision feature can be cheap to launch and expensive to keep honest. For a solo freelancer, the recurring work is not usually retraining; it is finding out that yesterday’s camera, lighting, API response, or customer expectations changed. I would choose a vision system whose failures I can inspect and reverse myself, even if a more sophisticated model wins the initial demo.</p>
<h2>The first model choice matters less than the exit path</h2>
<p>I treat the first-app comparison, <a href=/custom-vision-model-or-pretrained-api-which-fits-your-first-app/>Custom Vision Model or Pretrained API Which Fits Your First App</a>, as a launch checklist rather than a maintenance plan. The choice between a pretrained API and a custom model matters, but neither choice tells you who will diagnose a bad result on a Tuesday night. If the client cannot show you the input, the version that processed it, and the decision that followed, you have inherited an incident without evidence.</p>
<p><strong>A pretrained vision API wins</strong> when labels are scarce and its existing categories closely match the job, because you can test a useful workflow before collecting a training set. It costs per request, exposes you to provider changes, and may impose retention or data-transfer terms you need to check. <strong>A custom model run with ONNX Runtime wins</strong> when the task is narrow, you have permission to retain representative examples, and an API’s outputs repeatedly miss the same requirement. It costs annotation time, evaluation time, and responsibility for serving a pinned model file. Those are different maintenance bills, not a free-versus-paid choice.</p>
<p>My default would be a pretrained API behind one small adapter, with a manual-review path that remains usable if the API fails. I would <strong>not</strong> train a custom model merely to avoid an API bill, because a small monthly invoice is easier for one person to manage than an unmeasured labeling and retraining obligation. I would change that position only after saved examples show a persistent error the API cannot be configured to avoid.</p>
<p>Make the adapter return your own stable fields, such as <em>decision</em>, <em>confidence</em>, <em>provider</em>, and <em>model_version</em>, rather than passing the provider’s full JSON response into the app. That boundary limits a provider migration to one place because the rest of the code depends on your contract. Pin the provider endpoint or model identifier where the service permits it, and keep a small set of approved test images; a green HTTP response proves transport worked, not that the answer stayed the same.</p>
<p>There is also an exit cost in the images themselves. If your examples contain client data, confirm who may keep them before building a training archive, because a model you cannot legally or contractually retest is difficult to maintain. Record only what you need, set a deletion date, and keep a SHA-256 hash when you need to tell whether two files are identical without keeping duplicate copies.</p>
<h2>Every accepted prediction creates a small support obligation</h2>
<p>The tempting launch metric is “the model was correct on our test set.” The maintenance question is narrower: which mistakes reach the customer without review? A false positive that merely opens a review screen has a different cost from a false positive that sends an irreversible notification. Set the review threshold around the action’s cost, because the same model score can be acceptable for one action and unacceptable for another.</p>
<p>Start with an explicit record for each decision: a timestamp, a pseudonymous input ID, the model or API version, the score, the threshold in force, the action taken, and whether a human later corrected it. SQLite with <code>PRAGMA journal_mode=WAL</code> is sufficient for a single-process pilot because it makes local writes and later inspection straightforward. If multiple workers will write to the same database across machines, use a managed database instead, because a shared SQLite file is the wrong coordination mechanism.</p>
<p>Do not confuse a confidence score with a probability of being right. If a service returns <em>0.90</em>, that number may be a ranking signal rather than a calibrated 90% chance, because providers and model architectures define scores differently. The threshold must be chosen against labeled examples from the actual workflow. A provisional threshold of <em>0.85</em> is a value to tune, not a recommendation to ship: compare errors above and below it before allowing automatic acceptance.</p>
<p>A workable solo routine is to review a fixed sample of accepted decisions and all customer-reported mistakes each week. A sample of <em>30</em> reviewed accepted decisions is an operational starting count, not a statistically sufficient guarantee; small or changing traffic can make its error rate swing sharply. Log the denominator alongside the rate so “zero errors” cannot disguise “zero labels.” Include some low-confidence decisions in the review queue as well, because examining only accepted outputs will hide cases the system repeatedly sends to manual work.</p>
<p>Keep the review record deliberately plain. A CSV export can carry IDs, scores, outcomes, and model versions; a separate, access-controlled folder can hold the few images you have permission to retain. Strip unnecessary EXIF metadata before sharing examples, because camera locations and timestamps can reveal information unrelated to the task. This is less elegant than an annotation platform, but it is easier to audit when you are the only operator.</p>
<h2>Monitoring must tell you when to stop trusting automation</h2>
<p>I use the production-metrics guide, <a href=/how-to-prove-your-vision-model-works-in-production-metrics/>How to prove your vision model works in production metrics</a>, as a starting point for a stronger rule: a metric earns its place only if it changes what I do next. A dashboard of request counts cannot tell you whether accepted predictions are wrong, because correctness usually arrives later through review or complaint. Write the pause rule before launch so you are not inventing one while a client waits for an answer.</p>
<p>The following Python 3.11 smoke test calculates the error rate among reviewed, automatically accepted examples. It runs as written and intentionally refuses to call a four-example sample healthy. Replace the inline rows with a reviewed CSV export before using it for an operational decision.</p>
<pre>python3 - &lt;&lt;'PY'
import csv, io
rows = csv.DictReader(io.StringIO("""score,correct
0.93,1
0.88,0
0.94,0
0.89,0"""))
accepted = [r for r in rows if float(r["score"]) &gt;= 0.85]
n = len(accepted)
rate = sum(r["correct"] == "0" for r in accepted) / n if n else 0
print(f"sample={n} error={rate:.1%}")
print("REVIEW" if n &gt;= 30 and rate &gt; 0.10 else "INSUFFICIENT LABELS" if n &lt; 30 else "OK")
PY</pre>
<p>The <em>10%</em> pause boundary is an example policy to set with the client, not evidence that a 9% error rate is safe. The script also needs an important production companion: report how many requests bypassed review, because an apparently good reviewed sample may not represent the requests the model accepted. If corrections arrive days late, calculate outcomes by decision date and mark recent cohorts incomplete; otherwise, late labels can make a new version appear better than it is.</p>
<p>Track availability separately from correctness. Log HTTP status codes and latency at the adapter, and honor HTTP 429 and the <code>Retry-After</code> header rather than retrying immediately, because a burst of retries can prolong a provider rate limit. In an illustrative pilot, a measured p95 latency of <em>420 ms across 200 requests</em> would be a baseline for that pilot only; it would not predict a future client’s traffic. If requests start timing out, fall back to manual review rather than silently treating missing predictions as negative results.</p>
<p>I would not deploy Prometheus and OpenTelemetry just to watch one low-traffic freelancer project, because maintaining collectors and dashboards can consume more time than reading a daily export. A scheduled check, an emailed failure notice, and a weekly review appointment are enough until missed incidents show that the simpler arrangement is failing.</p>
<h2>A rollback plan saves more time than a retraining plan</h2>
<p>Many vision regressions start outside the model. A phone update can change image dimensions; an upload path can rotate pictures differently; a provider can change response fields. Keep a compact fixture set with expected decisions and image properties, then run it with pytest against the adapter before every release. Store the version of the adapter alongside the model identifier, because “same model” does not imply “same preprocessing.”</p>
<p>For a self-hosted model, pin the ONNX file and its checksum, the ONNX Runtime version, and any preprocessing constants such as resize dimensions and color order. A Docker image pinned by digest helps reproduce the software environment because a mutable tag can point to different contents later. It does not preserve the customer’s input distribution, so keep separately approved examples that exercise glare, rotation, blur, and the kinds of mistakes reviewers have actually seen.</p>
<p>For an API, save redacted request and response examples and test your adapter against them. Disable automatic acceptance with a configuration switch that does not require retraining or redeployment, because a freelancer needs a safe response even when the provider is unavailable. Tell the client what the manual path costs in time. An honest delay is preferable to an automatic answer you cannot validate.</p>
<p>Keep the rollback small enough to rehearse: restore the previous adapter release, restore its threshold, process one known fixture, and confirm that new requests enter the intended review path. Run that rehearsal after a change rather than assuming a saved model file is a rollback; configuration, preprocessing, and routing can all change the result while the weights stay identical.</p>
<h2>The first maintenance task is to write the pause rule</h2>
<p>Before signing off on a demo, take ten recent inputs you are allowed to retain and record the output, version, threshold, and human verdict for each. Then write down who switches off automatic acceptance, what evidence triggers that switch, and how work continues manually. If that procedure cannot fit on one page, simplify the feature before promising the client it will run unattended.</p>
<p>The post <a href="https://deepfriedbytes.com/the-ai-vision-maintenance-tax-solo-freelancers-miss/">The AI Vision Maintenance Tax Solo Freelancers Miss</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>From Empty Cluster to a Production-Ready GenAI Service</title>
		<link>https://deepfriedbytes.com/from-empty-cluster-to-a-production-ready-genai-service/</link>
		
		
		<pubDate>Tue, 03 Nov 2026 07:12:55 +0000</pubDate>
				<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[AI Web Solutions]]></category>
		<category><![CDATA[IT architecture]]></category>
		<category><![CDATA[Software Engineering]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/from-empty-cluster-to-a-production-ready-genai-service/</guid>

					<description><![CDATA[<p>A GenAI backend can pass a demo and still be unfit to operate: the answer looks plausible while its source document belongs to another tenant, its index is stale, or its retrieval call consumes the entire request budget. I would build the first production setup around an enforceable retrieval boundary, not model optimization, because access and failure behavior are harder to retrofit than prompts. The first deployable unit is an authorized document pipeline Begin with one source of truth: a document store whose records have a stable document ID, tenant ID, source version, and access policy. Give the ingestion worker read access to that source and write access to the retrieval database; do not give the request-serving API permission to rewrite source documents. That separation makes a bad reindex recoverable without allowing a compromised API process to alter the evidence it retrieves. For a first deployment, I would use PostgreSQL 16 with pgvector, an object store for original documents, and a queue such as Amazon SQS for change events. PostgreSQL wins here because the team can inspect document metadata, transaction state, and retrieval candidates with ordinary SQL. A dedicated vector service such as Qdrant wins when vector volume or search load has outgrown that arrangement; it costs another service to secure, back up, and reconcile with document permissions. Neither option removes the need to authorize each result. Make ingestion idempotent before generating an embedding. An event should identify a document and its source version, not contain an untrusted copy of its text. The worker fetches the current document, checks its deletion state, splits it into chunks, embeds those chunks, and replaces the previous version in one database transaction. A uniqueness constraint on document ID, source version, and chunk position prevents an SQS redelivery from creating duplicates. On deletion, remove the chunks and record the processed version; otherwise, a delayed older event can restore text that should no longer be searchable. For an initial tokenizer setting, try chunks of 400 tokens with 60 tokens of overlap; both are values to tune against your documents and evaluation queries, not universal limits. Record the splitter version alongside the embedding model ID, since changing either can make old and new vectors incomparable. OpenAI publishes a default dimension of 1,536 for text-embedding-3-small; choose the table dimension from the embedding configuration actually deployed, and reject vectors with the wrong dimension at ingestion. The model-choice framing (RAG vs Fine-Tuning in Your First GenAI Backend) is premature as a launch gate because neither choice specifies who may retrieve a chunk. I would not put all tenants into a shared vector search and ask the model to ignore unauthorized results: once the text has reached the prompt, a prompt instruction is not an access control. A small database you can recreate beats an elaborate search cluster you cannot audit Bring up a local retrieval database first. The following Bash snippet starts a PostgreSQL 16 image with pgvector and creates the minimum chunk table. It binds the port to loopback for local development; it is not a production networking or password configuration. docker run -d --name rag-db -e POSTGRES_PASSWORD=localonly \ -p 127.0.0.1:5432:5432 pgvector/pgvector:pg16 until docker exec rag-db pg_isready -U postgres &#62;/dev/null 2&#62;&#38;1; do sleep 1; done docker exec -i rag-db psql -U postgres -v ON_ERROR_STOP=1 &#60;&#60;'SQL' CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE chunks ( tenant_id text NOT NULL, document_id text NOT NULL, chunk_id text PRIMARY KEY, source_version text NOT NULL, body text NOT NULL, embedding vector(1536) NOT NULL, indexed_at timestamptz NOT NULL DEFAULT now() ); CREATE INDEX chunks_tenant ON chunks (tenant_id, document_id); SQL Pin the container image by digest in deployment so a rebuild cannot silently change database binaries. Put PostgreSQL on a private network, use a secret manager instead of the local password, enable TLS for database connections, and run migrations with a separate role. Back up both the database and original documents: vectors can be regenerated, but rebuilding them takes time and requires the exact source versions and embedding configuration. Next, implement a query that joins chunks to authorized documents before selecting the nearest matches. Use the authenticated principal from the API gateway or OIDC token to resolve allowed document IDs on the server; never accept a tenant ID from the request body as proof of access. A tenant predicate is a useful first filter, but document-level permissions still matter when people within one tenant have different access. Consider PostgreSQL row-level security as a second guard, with tests run under the restricted application role; table owners can bypass policies unless configured appropriately. Start with exact pgvector cosine-distance search using the &#60;=&#62; operator over the authorized candidate set. It may be fast enough when that set is small, and its behavior is easier to verify than an approximate index. Add an HNSW index only after query plans and load tests show the exact scan is the bottleneck: approximate search can return fewer useful matches after a selective permission filter, so test result quality as well as latency. Keep the retrieved chunk IDs and source versions with each response for audit and reproducible debugging. A working setup is more than a successful SQL query. The request path should authenticate the caller, embed the question, fetch authorized chunks, construct a prompt with source identifiers, call the generation model, and return an answer with citations that map to those identifiers. If retrieval returns no authorized evidence, return an explicit “no supporting source found” response rather than letting the model answer from general knowledge; that behavior keeps a document-grounded feature from silently changing its contract. Timeouts and traces are part of the feature, not later hardening Set one end-to-end deadline and divide it among embedding, retrieval, and generation. As an initial service-level objective, 900 ms at p95 might be a target to test for the retrieval portion, not a claim about pgvector performance. Leave room for the model call and network transit in the overall deadline. Use cancellation across HTTP and database calls so a disconnected request does not keep consuming connections and model quota. Instrument the path with OpenTelemetry spans for authentication, query embedding, authorized-candidate lookup, vector search, prompt assembly, and model response. Export duration histograms and error counters to Prometheus; calculate p95 with histogram_quantile() over an appropriate rate window. Track candidate count, returned chunk count, PostgreSQL pool wait time, and the age of the newest indexed source version. These measurements distinguish a slow search from a queue backlog or a model-provider delay, which require different fixes. The scaling warning (Your GenAI Feature Scales Until Retrieval and Latency Collide) deserves a narrower operational test: retrieval and latency collide only when traces show which resource is saturated. Run a staging workload with, for example, 30 concurrent virtual users as an initial test setting, then vary concurrency while holding the query set and authorization distribution fixed. Include common queries, low-result queries, and tenants with very different document counts. A single aggregate p95 can hide the tenant whose authorized search space is much larger. Configure separate timeouts and concurrency limits for ingestion and serving. An embedding provider outage should stop new indexing without exhausting API workers, while an SQS dead-letter queue should retain failed events for inspection. During a generation-provider outage, do not automatically retry every live request: retries can multiply traffic precisely when capacity is scarce. Return a bounded error, preserve the trace ID, and let the client retry according to its own backoff policy. A release is safe only when access, freshness, and recovery have tests Build a small evaluation set before opening traffic. Include questions with a known supporting chunk, questions with no supporting chunk, and questions where a tempting matching chunk belongs to another principal. Test the retrieved IDs separately from the generated prose; a fluent answer cannot prove that retrieval honored permissions. For each deployment, run these cases against the same source versions and record whether the expected chunk appears in the authorized top results. Add three failure drills to the deployment checklist. First, revoke document access after it has been indexed and verify that the next request cannot retrieve it; this tests authorization at query time rather than assuming index updates are instantaneous. Second, delete a document and replay an older ingestion event to confirm that version checks prevent resurrection. Third, stop the embedding provider and verify that serving continues from existing indexed data while ingestion failures go to a visible queue. Define freshness as an observable delay between a committed source version and its searchable version. A five-minute maximum lag is one possible acceptance threshold to adjust to the update rate and user expectation; alert on sustained breaches rather than every short queue spike. Also alert when the dead-letter queue grows, database connections approach their configured limit, or authorized-result counts drop abruptly after a deployment. Each alert should point to a runbook action: pause ingestion, roll back a splitter change, drain a bad event, or restore a database snapshot. Roll out to a small group with citations and trace IDs visible to support staff. Keep the previous embedding model and index available until the new version passes access tests and query evaluation, because changing embedding versions in place makes regressions hard to isolate. If costs or latency rise, inspect span data and query plans before buying a faster vector service; moving the index will not repair an authorization join, an unbounded connection pool, or a slow provider call. The first production action is to prove a forbidden document stays forbidden Create two test principals with overlapping source material but different document permissions. Index one document each, then issue the same question under both identities and inspect the retrieved chunk IDs, not merely the answers. Run that test after an access revocation as well. If it fails, stop there: tuning prompts, indexes, and latency against a retrieval path that leaks evidence only makes the wrong setup faster.</p>
<p>The post <a href="https://deepfriedbytes.com/from-empty-cluster-to-a-production-ready-genai-service/">From Empty Cluster to a Production-Ready GenAI Service</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A GenAI backend can pass a demo and still be unfit to operate: the answer looks plausible while its source document belongs to another tenant, its index is stale, or its retrieval call consumes the entire request budget. I would build the first production setup around an enforceable retrieval boundary, not model optimization, because access and failure behavior are harder to retrofit than prompts.</p>
<h2>The first deployable unit is an authorized document pipeline</h2>
<p>Begin with one source of truth: a document store whose records have a stable document ID, tenant ID, source version, and access policy. Give the ingestion worker read access to that source and write access to the retrieval database; do not give the request-serving API permission to rewrite source documents. That separation makes a bad reindex recoverable without allowing a compromised API process to alter the evidence it retrieves.</p>
<p>For a first deployment, I would use PostgreSQL 16 with pgvector, an object store for original documents, and a queue such as Amazon SQS for change events. PostgreSQL wins here because the team can inspect document metadata, transaction state, and retrieval candidates with ordinary SQL. A dedicated vector service such as Qdrant wins when vector volume or search load has outgrown that arrangement; it costs another service to secure, back up, and reconcile with document permissions. Neither option removes the need to authorize each result.</p>
<p>Make ingestion idempotent before generating an embedding. An event should identify a document and its source version, not contain an untrusted copy of its text. The worker fetches the current document, checks its deletion state, splits it into chunks, embeds those chunks, and replaces the previous version in one database transaction. A uniqueness constraint on document ID, source version, and chunk position prevents an SQS redelivery from creating duplicates. On deletion, remove the chunks and record the processed version; otherwise, a delayed older event can restore text that should no longer be searchable.</p>
<p>For an initial tokenizer setting, try chunks of <strong>400 tokens</strong> with <strong>60 tokens</strong> of overlap; both are values to tune against your documents and evaluation queries, not universal limits. Record the splitter version alongside the embedding model ID, since changing either can make old and new vectors incomparable. OpenAI publishes a default dimension of <strong>1,536</strong> for <code>text-embedding-3-small</code>; choose the table dimension from the embedding configuration actually deployed, and reject vectors with the wrong dimension at ingestion.</p>
<p>The model-choice framing (<a href=/rag-vs-fine-tuning-in-your-first-genai-backend/>RAG vs Fine-Tuning in Your First GenAI Backend</a>) is premature as a launch gate because neither choice specifies who may retrieve a chunk. I would <strong>not</strong> put all tenants into a shared vector search and ask the model to ignore unauthorized results: once the text has reached the prompt, a prompt instruction is not an access control.</p>
<h2>A small database you can recreate beats an elaborate search cluster you cannot audit</h2>
<p>Bring up a local retrieval database first. The following Bash snippet starts a PostgreSQL 16 image with pgvector and creates the minimum chunk table. It binds the port to loopback for local development; it is not a production networking or password configuration.</p>
<pre>docker run -d --name rag-db -e POSTGRES_PASSWORD=localonly \
  -p 127.0.0.1:5432:5432 pgvector/pgvector:pg16
until docker exec rag-db pg_isready -U postgres &gt;/dev/null 2&gt;&amp;1; do sleep 1; done
docker exec -i rag-db psql -U postgres -v ON_ERROR_STOP=1 &lt;&lt;'SQL'
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (
  tenant_id text NOT NULL,
  document_id text NOT NULL,
  chunk_id text PRIMARY KEY,
  source_version text NOT NULL,
  body text NOT NULL,
  embedding vector(1536) NOT NULL,
  indexed_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX chunks_tenant ON chunks (tenant_id, document_id);
SQL</pre>
<p>Pin the container image by digest in deployment so a rebuild cannot silently change database binaries. Put PostgreSQL on a private network, use a secret manager instead of the local password, enable TLS for database connections, and run migrations with a separate role. Back up both the database and original documents: vectors can be regenerated, but rebuilding them takes time and requires the exact source versions and embedding configuration.</p>
<p>Next, implement a query that joins chunks to authorized documents <em>before</em> selecting the nearest matches. Use the authenticated principal from the API gateway or OIDC token to resolve allowed document IDs on the server; never accept a tenant ID from the request body as proof of access. A tenant predicate is a useful first filter, but document-level permissions still matter when people within one tenant have different access. Consider PostgreSQL row-level security as a second guard, with tests run under the restricted application role; table owners can bypass policies unless configured appropriately.</p>
<p>Start with exact pgvector cosine-distance search using the <code>&lt;=&gt;</code> operator over the authorized candidate set. It may be fast enough when that set is small, and its behavior is easier to verify than an approximate index. Add an HNSW index only after query plans and load tests show the exact scan is the bottleneck: approximate search can return fewer useful matches after a selective permission filter, so test result quality as well as latency. Keep the retrieved chunk IDs and source versions with each response for audit and reproducible debugging.</p>
<p>A working setup is more than a successful SQL query. The request path should authenticate the caller, embed the question, fetch authorized chunks, construct a prompt with source identifiers, call the generation model, and return an answer with citations that map to those identifiers. If retrieval returns no authorized evidence, return an explicit “no supporting source found” response rather than letting the model answer from general knowledge; that behavior keeps a document-grounded feature from silently changing its contract.</p>
<h2>Timeouts and traces are part of the feature, not later hardening</h2>
<p>Set one end-to-end deadline and divide it among embedding, retrieval, and generation. As an initial service-level objective, <strong>900 ms at p95</strong> might be a target to test for the retrieval portion, not a claim about pgvector performance. Leave room for the model call and network transit in the overall deadline. Use cancellation across HTTP and database calls so a disconnected request does not keep consuming connections and model quota.</p>
<p>Instrument the path with OpenTelemetry spans for authentication, query embedding, authorized-candidate lookup, vector search, prompt assembly, and model response. Export duration histograms and error counters to Prometheus; calculate p95 with <code>histogram_quantile()</code> over an appropriate rate window. Track candidate count, returned chunk count, PostgreSQL pool wait time, and the age of the newest indexed source version. These measurements distinguish a slow search from a queue backlog or a model-provider delay, which require different fixes.</p>
<p>The scaling warning (<a href=/your-genai-feature-scales-until-retrieval-and-latency-collide/>Your GenAI Feature Scales Until Retrieval and Latency Collide</a>) deserves a narrower operational test: retrieval and latency collide only when traces show which resource is saturated. Run a staging workload with, for example, <strong>30 concurrent virtual users</strong> as an initial test setting, then vary concurrency while holding the query set and authorization distribution fixed. Include common queries, low-result queries, and tenants with very different document counts. A single aggregate p95 can hide the tenant whose authorized search space is much larger.</p>
<p>Configure separate timeouts and concurrency limits for ingestion and serving. An embedding provider outage should stop new indexing without exhausting API workers, while an SQS dead-letter queue should retain failed events for inspection. During a generation-provider outage, do not automatically retry every live request: retries can multiply traffic precisely when capacity is scarce. Return a bounded error, preserve the trace ID, and let the client retry according to its own backoff policy.</p>
<h2>A release is safe only when access, freshness, and recovery have tests</h2>
<p>Build a small evaluation set before opening traffic. Include questions with a known supporting chunk, questions with no supporting chunk, and questions where a tempting matching chunk belongs to another principal. Test the retrieved IDs separately from the generated prose; a fluent answer cannot prove that retrieval honored permissions. For each deployment, run these cases against the same source versions and record whether the expected chunk appears in the authorized top results.</p>
<p>Add three failure drills to the deployment checklist. First, revoke document access after it has been indexed and verify that the next request cannot retrieve it; this tests authorization at query time rather than assuming index updates are instantaneous. Second, delete a document and replay an older ingestion event to confirm that version checks prevent resurrection. Third, stop the embedding provider and verify that serving continues from existing indexed data while ingestion failures go to a visible queue.</p>
<p>Define freshness as an observable delay between a committed source version and its searchable version. A <strong>five-minute</strong> maximum lag is one possible acceptance threshold to adjust to the update rate and user expectation; alert on sustained breaches rather than every short queue spike. Also alert when the dead-letter queue grows, database connections approach their configured limit, or authorized-result counts drop abruptly after a deployment. Each alert should point to a runbook action: pause ingestion, roll back a splitter change, drain a bad event, or restore a database snapshot.</p>
<p>Roll out to a small group with citations and trace IDs visible to support staff. Keep the previous embedding model and index available until the new version passes access tests and query evaluation, because changing embedding versions in place makes regressions hard to isolate. If costs or latency rise, inspect span data and query plans before buying a faster vector service; moving the index will not repair an authorization join, an unbounded connection pool, or a slow provider call.</p>
<h2>The first production action is to prove a forbidden document stays forbidden</h2>
<p>Create two test principals with overlapping source material but different document permissions. Index one document each, then issue the same question under both identities and inspect the retrieved chunk IDs, not merely the answers. Run that test after an access revocation as well. If it fails, stop there: tuning prompts, indexes, and latency against a retrieval path that leaks evidence only makes the wrong setup faster.</p>
<p>The post <a href="https://deepfriedbytes.com/from-empty-cluster-to-a-production-ready-genai-service/">From Empty Cluster to a Production-Ready GenAI Service</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>5 Robotics Team Mistakes That Cost Junior Devs Weeks</title>
		<link>https://deepfriedbytes.com/5-robotics-team-mistakes-that-cost-junior-devs-weeks/</link>
		
		
		<pubDate>Thu, 29 Oct 2026 10:52:49 +0000</pubDate>
				<category><![CDATA[Robotics]]></category>
		<category><![CDATA[Development Trends]]></category>
		<category><![CDATA[IT architecture]]></category>
		<category><![CDATA[Software Engineering]]></category>
		<category><![CDATA[technology]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/5-robotics-team-mistakes-that-cost-junior-devs-weeks/</guid>

					<description><![CDATA[<p>A robot that moves on a developer’s laptop has passed an onboarding exercise, not a release test. I would not start an automation project by adding a planner or splitting nodes across machines, because neither fixes the failures a short demo rarely exposes: stale sensor data, inconsistent clocks, and commands that outlive the connection that sent them. Teams pay for that confusion in debugging time first and unsafe motion later. A moving bot is a poor release criterion because the demo hides failure modes Read ROS 2 on a laptop in 60 minutes from install to moving bot, then replay the resulting demo with missing odometry before you call it ready. The quickstart’s moving-bot milestone is useful for confirming that ROS 2, a launch file, and a command path work together. It does not establish what happens when one of those pieces stops working during motion. A common team mistake is to turn that first successful run into an implicit interface contract. A developer sees a cmd_vel message produce movement and assumes the robot will stop when the publisher exits. That assumption depends on the motor controller’s behavior, the command timeout, and whether another publisher is still active. If nobody has specified those conditions, the cost is a late discovery during integration: someone must trace which process owns motion while the robot is already behaving unexpectedly. Write down the contract before changing the architecture. For each motion command, identify its owner, the conditions under which it expires, and the component that enforces expiration. For each input, identify how old it may be before the controller rejects it. These are testable statements; “the robot responds smoothly” is not, because two observers can watch the same run and disagree about whether it passed. ROS 2 Jazzy Jalisco is a practical version to pin for a new bench exercise, but the version alone does not make the exercise reproducible because middleware, dependencies, and launch parameters also affect behavior. Record the ROS distribution, the DDS implementation, and the configuration committed with the test. A teammate should be able to replay the same fault without reconstructing your shell history. Make one failure test as routine as the happy-path demo: start motion at a deliberately capped speed, stop publishing odometry, and record the time until the controller inhibits motion. Treat 0.5 m/s as a trial cap to set for your own platform, not as a universally safe speed. If the test has no expected outcome, the team has found a missing requirement rather than a flaky robot. QoS defaults can turn a live sensor into a silent data gap ROS 2 uses DDS quality-of-service policies, and a publisher being visible on the graph does not prove a subscriber can receive its messages. A reliable subscriber, for example, is incompatible with a publisher offering only best-effort delivery. A junior developer may spend hours inspecting callback code when the real fault is a policy mismatch at discovery. Start with what the running graph actually offers. The following bench probe publishes a string and prints the endpoint details on a machine with ROS 2 Jazzy installed and sourced from the standard path: source /opt/ros/jazzy/setup.bash ros2 topic pub -r 10 /probe std_msgs/msg/String '{data: alive}' &#62;/tmp/ros-probe.log 2&#62;&#38;1 &#38; pub_pid=$! trap 'kill "$pub_pid" 2&#62;/dev/null &#124;&#124; true' EXIT sleep 2 ros2 topic info --verbose /probe ros2 topic echo --once /probe std_msgs/msg/String kill "$pub_pid" wait "$pub_pid" 2&#62;/dev/null &#124;&#124; true The 10 Hz rate here is a bench setting to make the probe easy to observe, not a recommendation for a sensor. On an actual sensor_msgs/msg/LaserScan topic, inspect reliability, durability, and history on both endpoints with ros2 topic info &#8211;verbose. Then check whether the subscriber is receiving timely scans; a matching QoS policy cannot guarantee fresh data if the publisher itself is delayed. Best effort versus reliable is a choice with different costs. Best effort often wins for frequent sensor samples when a newer sample can replace a lost one, because retransmitting an old sample may add little value. It costs you occasional missing samples, which the consumer must tolerate and measure. Reliable delivery wins when losing an individual message would break the application’s intended behavior; it costs buffering and potentially more delay under a poor connection. Neither setting is a substitute for rejecting stale input, because a delivered message can still arrive too late to use. Define freshness at the consumer, not just frequency at the publisher. An initial 100 ms maximum scan age can be a value to tune during bench tests; choose it against the robot’s speed, sensor timing, and stopping behavior. Log message header time, receipt time, and whether the controller accepted the sample. Without those fields, a team can “fix” a missing-data incident by changing QoS while leaving the late-data problem intact. Clock and frame mistakes often masquerade as planner defects When a navigation run fails, teams often tune the planner first because the visible symptom is a bad path. That can waste days if the underlying error is a transform at the wrong time or a frame with the wrong meaning. A planner cannot make a sound decision from a pose that was assembled using inconsistent timestamps. Use the ROS coordinate-frame conventions in REP-105 as a check on meaning: map, odom, and base_link are not interchangeable labels. The odom frame supports locally continuous motion estimates, while map can change as localization corrects its estimate. If a team silently treats a map correction as physical movement, a controller may react to a jump the robot never made. Inspect the transform tree with tf2 tools before changing planner parameters. Time needs the same discipline. A simulation using Gazebo and use_sim_time:=true may publish /clock, while a hardware node using its system clock may stamp nav_msgs/msg/Odometry differently. The resulting transform lookup failures can appear intermittent because they depend on which messages arrived and when. Check the use_sim_time parameter for every node involved, then compare header stamps with the clock each node uses. Pick a provisional 50 ms transform-age budget for a bench test and adjust it from observed timing and the platform’s motion constraints. Record both the measured age distribution and the count of rejected lookups; reporting only an average hides the late samples most likely to disrupt a run. rosbag2 helps here because a recorded failure gives another developer the same message timing to inspect, although playback still needs the clock and parameters set deliberately. The cost of skipping these checks is a misleading fix. Increasing a planner’s tolerance might make one run look better because it accepts older data, while reducing the time available to detect a genuinely stale pose. Change a tolerance only after you can show which timestamp or transform caused the rejection. A watchdog that only logs failures does not stop a robot ROS 2 DDS deadline and liveliness policies can reveal that an endpoint missed an expected update or disappeared. They do not, by themselves, guarantee that a motor stops, because detection still has to trigger an action through a functioning path. Treat a warning in a terminal as diagnostic evidence, not as a safety mechanism. Trace the entire stop path with the team: sensor or command expires, a supervisor detects it, the controller rejects further motion, and an actuator-level timeout or independent stop path enforces the result. If the same computer and network connection are required both to command motion and to stop it, losing that connection can defeat the planned response. A separate hardware emergency stop or motor-controller timeout may be warranted, but claiming compliance with ISO 13849 or IEC 61508 requires a safety process and evidence, not the presence of a ROS node. A provisional 200 ms command timeout is a test setting, not a universal stopping limit. Tune it against command frequency, network behavior, and the time the platform needs to stop. Then interrupt the publisher, disconnect the network, and kill the supervisor in separate trials. Those are different faults: a robot that stops after one may keep moving after another. Measure elapsed time from the last accepted command to inhibited motion, and record whether the motor controller or the ROS supervisor caused the stop. Run 30 induced dropouts as an initial experiment size rather than treating a clean run as proof of safety. Even zero observed late stops in that set would establish only what happened under those tested conditions. Put the fault procedure in launch_testing or a hardware test script so the next developer can repeat it after a controller change. Keep security separate from this stop claim. ROS_DOMAIN_ID can help separate discovery domains, and SROS2 can configure DDS security, but neither enforces a physical stop when a command path fails. Mixing those concerns creates a costly review error: the team sees an access-control measure and assumes it has also addressed motion safety. Trend-led rewrites charge interest before they remove a fault Read Robotics Software Development Trends for Smart Automation, then ask which proposed change would eliminate a failure your tests have reproduced. Trend discussions can suggest useful approaches, but adopting an architecture because it is current gives a junior developer more interfaces to debug before the team knows which interface is failing. Consider ROS 2 composable nodes versus separate processes. Composition can win when nodes exchange large messages on one computer and measured transfer overhead is a bottleneck; its cost is shared process fate and harder isolation when one component crashes. Separate processes can win when fault isolation and independent restarts matter more; their cost is extra interprocess communication and deployment configuration. Choose after a trace identifies the bottleneck or a fault test identifies the needed isolation. Otherwise, the rewrite risks changing failure behavior without improving it. The same discipline applies before switching between Cyclone DDS and Fast DDS. Middleware differences can matter on a particular network, but changing implementations while also changing QoS and node layout makes the result difficult to attribute. Hold the launch configuration constant, run the same rosbag2 input or repeatable hardware scenario, and compare the metric that prompted the change. Use colcon test and pytest for ordinary regressions, while keeping hardware fault results distinct from simulator results because a simulator cannot establish how a real actuator stops. On your next task, take one motion-related topic and write down its publisher, subscriber, QoS policies, timestamp source, expiry condition, and stop owner. Run one dropout and save the log before proposing a new node or middleware setting. If you cannot identify who rejects an expired command, that is the first defect to fix.</p>
<p>The post <a href="https://deepfriedbytes.com/5-robotics-team-mistakes-that-cost-junior-devs-weeks/">5 Robotics Team Mistakes That Cost Junior Devs Weeks</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A robot that moves on a developer’s laptop has passed an onboarding exercise, not a release test. I would not start an automation project by adding a planner or splitting nodes across machines, because neither fixes the failures a short demo rarely exposes: stale sensor data, inconsistent clocks, and commands that outlive the connection that sent them. Teams pay for that confusion in debugging time first and unsafe motion later.</p>
<h2>A moving bot is a poor release criterion because the demo hides failure modes</h2>
<p>Read <a href=/ros-2-on-a-laptop-in-60-minutes-from-install-to-moving-bot/>ROS 2 on a laptop in 60 minutes from install to moving bot</a>, then replay the resulting demo with missing odometry before you call it ready. The quickstart’s moving-bot milestone is useful for confirming that ROS 2, a launch file, and a command path work together. It does not establish what happens when one of those pieces stops working during motion.</p>
<p>A common team mistake is to turn that first successful run into an implicit interface contract. A developer sees a <em>cmd_vel</em> message produce movement and assumes the robot will stop when the publisher exits. That assumption depends on the motor controller’s behavior, the command timeout, and whether another publisher is still active. If nobody has specified those conditions, the cost is a late discovery during integration: someone must trace which process owns motion while the robot is already behaving unexpectedly.</p>
<p>Write down the contract before changing the architecture. For each motion command, identify its owner, the conditions under which it expires, and the component that enforces expiration. For each input, identify how old it may be before the controller rejects it. These are testable statements; “the robot responds smoothly” is not, because two observers can watch the same run and disagree about whether it passed.</p>
<p>ROS 2 Jazzy Jalisco is a practical version to pin for a new bench exercise, but the version alone does not make the exercise reproducible because middleware, dependencies, and launch parameters also affect behavior. Record the ROS distribution, the DDS implementation, and the configuration committed with the test. A teammate should be able to replay the same fault without reconstructing your shell history.</p>
<p>Make one failure test as routine as the happy-path demo: start motion at a deliberately capped speed, stop publishing odometry, and record the time until the controller inhibits motion. Treat <strong>0.5 m/s</strong> as a trial cap to set for your own platform, not as a universally safe speed. If the test has no expected outcome, the team has found a missing requirement rather than a flaky robot.</p>
<h2>QoS defaults can turn a live sensor into a silent data gap</h2>
<p>ROS 2 uses DDS quality-of-service policies, and a publisher being visible on the graph does not prove a subscriber can receive its messages. A reliable subscriber, for example, is incompatible with a publisher offering only best-effort delivery. A junior developer may spend hours inspecting callback code when the real fault is a policy mismatch at discovery.</p>
<p>Start with what the running graph actually offers. The following bench probe publishes a string and prints the endpoint details on a machine with ROS 2 Jazzy installed and sourced from the standard path:</p>
<pre>source /opt/ros/jazzy/setup.bash
ros2 topic pub -r 10 /probe std_msgs/msg/String '{data: alive}' &gt;/tmp/ros-probe.log 2&gt;&amp;1 &amp;
pub_pid=$!
trap 'kill "$pub_pid" 2&gt;/dev/null || true' EXIT
sleep 2
ros2 topic info --verbose /probe
ros2 topic echo --once /probe std_msgs/msg/String
kill "$pub_pid"
wait "$pub_pid" 2&gt;/dev/null || true</pre>
<p>The <strong>10 Hz</strong> rate here is a bench setting to make the probe easy to observe, not a recommendation for a sensor. On an actual <em>sensor_msgs/msg/LaserScan</em> topic, inspect reliability, durability, and history on both endpoints with <em>ros2 topic info &#8211;verbose</em>. Then check whether the subscriber is receiving timely scans; a matching QoS policy cannot guarantee fresh data if the publisher itself is delayed.</p>
<p><strong>Best effort versus reliable</strong> is a choice with different costs. Best effort often wins for frequent sensor samples when a newer sample can replace a lost one, because retransmitting an old sample may add little value. It costs you occasional missing samples, which the consumer must tolerate and measure. Reliable delivery wins when losing an individual message would break the application’s intended behavior; it costs buffering and potentially more delay under a poor connection. Neither setting is a substitute for rejecting stale input, because a delivered message can still arrive too late to use.</p>
<p>Define freshness at the consumer, not just frequency at the publisher. An initial <strong>100 ms</strong> maximum scan age can be a value to tune during bench tests; choose it against the robot’s speed, sensor timing, and stopping behavior. Log message header time, receipt time, and whether the controller accepted the sample. Without those fields, a team can “fix” a missing-data incident by changing QoS while leaving the late-data problem intact.</p>
<h2>Clock and frame mistakes often masquerade as planner defects</h2>
<p>When a navigation run fails, teams often tune the planner first because the visible symptom is a bad path. That can waste days if the underlying error is a transform at the wrong time or a frame with the wrong meaning. A planner cannot make a sound decision from a pose that was assembled using inconsistent timestamps.</p>
<p>Use the ROS coordinate-frame conventions in REP-105 as a check on meaning: <em>map</em>, <em>odom</em>, and <em>base_link</em> are not interchangeable labels. The <em>odom</em> frame supports locally continuous motion estimates, while <em>map</em> can change as localization corrects its estimate. If a team silently treats a map correction as physical movement, a controller may react to a jump the robot never made. Inspect the transform tree with <em>tf2</em> tools before changing planner parameters.</p>
<p>Time needs the same discipline. A simulation using Gazebo and <em>use_sim_time:=true</em> may publish <em>/clock</em>, while a hardware node using its system clock may stamp <em>nav_msgs/msg/Odometry</em> differently. The resulting transform lookup failures can appear intermittent because they depend on which messages arrived and when. Check the <em>use_sim_time</em> parameter for every node involved, then compare header stamps with the clock each node uses.</p>
<p>Pick a provisional <strong>50 ms</strong> transform-age budget for a bench test and adjust it from observed timing and the platform’s motion constraints. Record both the measured age distribution and the count of rejected lookups; reporting only an average hides the late samples most likely to disrupt a run. <em>rosbag2</em> helps here because a recorded failure gives another developer the same message timing to inspect, although playback still needs the clock and parameters set deliberately.</p>
<p>The cost of skipping these checks is a misleading fix. Increasing a planner’s tolerance might make one run look better because it accepts older data, while reducing the time available to detect a genuinely stale pose. Change a tolerance only after you can show which timestamp or transform caused the rejection.</p>
<h2>A watchdog that only logs failures does not stop a robot</h2>
<p>ROS 2 DDS deadline and liveliness policies can reveal that an endpoint missed an expected update or disappeared. They do not, by themselves, guarantee that a motor stops, because detection still has to trigger an action through a functioning path. Treat a warning in a terminal as diagnostic evidence, not as a safety mechanism.</p>
<p>Trace the entire stop path with the team: sensor or command expires, a supervisor detects it, the controller rejects further motion, and an actuator-level timeout or independent stop path enforces the result. If the same computer and network connection are required both to command motion and to stop it, losing that connection can defeat the planned response. A separate hardware emergency stop or motor-controller timeout may be warranted, but claiming compliance with ISO 13849 or IEC 61508 requires a safety process and evidence, not the presence of a ROS node.</p>
<p>A provisional <strong>200 ms</strong> command timeout is a test setting, not a universal stopping limit. Tune it against command frequency, network behavior, and the time the platform needs to stop. Then interrupt the publisher, disconnect the network, and kill the supervisor in separate trials. Those are different faults: a robot that stops after one may keep moving after another.</p>
<p>Measure elapsed time from the last accepted command to inhibited motion, and record whether the motor controller or the ROS supervisor caused the stop. Run <strong>30 induced dropouts</strong> as an initial experiment size rather than treating a clean run as proof of safety. Even zero observed late stops in that set would establish only what happened under those tested conditions. Put the fault procedure in <em>launch_testing</em> or a hardware test script so the next developer can repeat it after a controller change.</p>
<p>Keep security separate from this stop claim. <em>ROS_DOMAIN_ID</em> can help separate discovery domains, and SROS2 can configure DDS security, but neither enforces a physical stop when a command path fails. Mixing those concerns creates a costly review error: the team sees an access-control measure and assumes it has also addressed motion safety.</p>
<h2>Trend-led rewrites charge interest before they remove a fault</h2>
<p>Read <a href=/robotics-software-development-trends-for-smart-automation/>Robotics Software Development Trends for Smart Automation</a>, then ask which proposed change would eliminate a failure your tests have reproduced. Trend discussions can suggest useful approaches, but adopting an architecture because it is current gives a junior developer more interfaces to debug before the team knows which interface is failing.</p>
<p>Consider <strong>ROS 2 composable nodes versus separate processes</strong>. Composition can win when nodes exchange large messages on one computer and measured transfer overhead is a bottleneck; its cost is shared process fate and harder isolation when one component crashes. Separate processes can win when fault isolation and independent restarts matter more; their cost is extra interprocess communication and deployment configuration. Choose after a trace identifies the bottleneck or a fault test identifies the needed isolation. Otherwise, the rewrite risks changing failure behavior without improving it.</p>
<p>The same discipline applies before switching between Cyclone DDS and Fast DDS. Middleware differences can matter on a particular network, but changing implementations while also changing QoS and node layout makes the result difficult to attribute. Hold the launch configuration constant, run the same <em>rosbag2</em> input or repeatable hardware scenario, and compare the metric that prompted the change. Use <em>colcon test</em> and <em>pytest</em> for ordinary regressions, while keeping hardware fault results distinct from simulator results because a simulator cannot establish how a real actuator stops.</p>
<p>On your next task, take one motion-related topic and write down its publisher, subscriber, QoS policies, timestamp source, expiry condition, and stop owner. Run one dropout and save the log before proposing a new node or middleware setting. If you cannot identify who rejects an expired command, that is the first defect to fix.</p>
<p>The post <a href="https://deepfriedbytes.com/5-robotics-team-mistakes-that-cost-junior-devs-weeks/">5 Robotics Team Mistakes That Cost Junior Devs Weeks</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>2024 Crypto Runbooks Fail Production in 2026</title>
		<link>https://deepfriedbytes.com/2024-crypto-runbooks-fail-production-in-2026/</link>
		
		
		<pubDate>Thu, 22 Oct 2026 07:47:32 +0000</pubDate>
				<category><![CDATA[Cryptocurrencies]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[AI Web Solutions]]></category>
		<category><![CDATA[Blockchain]]></category>
		<category><![CDATA[Digital ecosystems]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/2024-crypto-runbooks-fail-production-in-2026/</guid>

					<description><![CDATA[<p>A wallet request can time out after it has been broadcast, leaving the caller unable to tell whether retrying will create a second transfer. Since 2024, delegated accounts and more varied transaction paths have made that uncertainty harder to contain. My position: a production wallet needs a durable intent record before it needs another signing integration. If no component owns recovery, scaling workers only scales the ambiguity. The old build-or-buy boundary no longer matches the recovery boundary I disagree with Your 2024 Crypto Build-vs-Buy Playbook Is Obsolete if its sourcing question is treated as the production decision: buying signing, RPC, or account-abstraction services does not buy an answer to “Did this customer request execute?” Your team still has to join the HTTP request, the signed operation, the broadcast attempt, and the eventual chain result into one recoverable record. That distinction has become sharper over the last two years. Ethereum’s Pectra upgrade activated on May 7, 2025—a network date, not a performance claim—and introduced EIP-7702, which lets an externally owned account delegate execution to code. An EIP-7702-capable wallet can therefore have behavior that an older “EOA means one simple transaction” runbook does not describe. ERC-4337 adds another operational path: a UserOperation goes through a bundler and an EntryPoint rather than through the ordinary EOA transaction submission path. EntryPoint v0.7 matters as a version boundary because tooling that serializes a UserOperation for a different EntryPoint version can produce an operation the intended path will not accept. These mechanisms do not make every wallet request interchangeable. EIP-5792’s wallet_sendCalls describes a wallet-facing call-batching interface, while eth_sendRawTransaction broadcasts an already signed EVM transaction; a successful response from either interface is not proof of final execution because inclusion and chain history can still change. A paymaster may also be involved in an ERC-4337 operation, so rebuilding an apparently identical request after a timeout can change its signed contents. Store the exact bytes or canonical operation fields that were submitted, not merely the API payload that started the process. I would not make a vendor webhook the system of record, because a delayed, duplicated, or missing webhook cannot establish what your API already promised the caller. Use webhooks as evidence to reconcile against receipts and your durable intent record. Keep the vendor’s request ID, the chain ID, the signer or smart-account address, and the transaction or UserOperation hash together; without that mapping, an on-call engineer must guess which customer action a chain event represents. Idempotency now has to survive a signer and a broadcaster At 10x users, your crypto wallet breaks at idempotency names a real failure mode, but I would not use traffic growth as the trigger for fixing it: one timeout between signing and persistence is enough to make a retry unsafe. The operational contract should be narrower and testable: the same authenticated request key and the same request body identify one wallet intent, while a reused key with a different body is rejected. That contract prevents your API from deliberately creating a second intent; it does not claim that a blockchain provides exactly-once execution. Put a uniqueness constraint around the intent before dispatching work. This PostgreSQL example runs in psql; its sample hash is a placeholder, so production code must compute the digest from a canonical request representation before insertion. CREATE TABLE wallet_intents ( tenant text NOT NULL, chain_id bigint NOT NULL, idem_key text NOT NULL, body_hash text NOT NULL, state text NOT NULL DEFAULT 'accepted', tx_hash text, PRIMARY KEY (tenant, chain_id, idem_key) ); INSERT INTO wallet_intents (tenant, chain_id, idem_key, body_hash) VALUES ('demo', 1, 'request-7', 'sha256:sample') ON CONFLICT (tenant, chain_id, idem_key) DO NOTHING; SELECT body_hash, state, tx_hash FROM wallet_intents WHERE tenant = 'demo' AND chain_id = 1 AND idem_key = 'request-7'; The insert makes concurrent acceptance of the same key converge on one row because PostgreSQL enforces the primary key. It does not finish the API contract by itself: after a conflict, compare the stored body_hash with the incoming hash and return a conflict if they differ. Treat that check and any state claim as database transactions, because two workers that merely read “accepted” can both decide to sign. A worker should claim a dispatch state with a conditional update, record the resulting signed payload or hash, and only then proceed through the broadcast state machine. Nonce management needs its own boundary. For an EOA, reserve a nonce per chain ID plus signer under database coordination; asking two RPC endpoints for eth_getTransactionCount with pending is not a lock, because their mempools need not contain the same transactions. Never allocate a fresh nonce just because eth_sendRawTransaction timed out: rebroadcasting the same signed bytes preserves the transaction hash, whereas a newly signed transaction can create a second payment. Replacement under EIP-1559 is a deliberate state transition with the same nonce and changed fee fields, not an automatic retry of the customer intent. For ERC-4337, track the UserOperation hash, EntryPoint address, smart account, and bundler submission separately from the business intent. Calling eth_sendUserOperation again with altered operation fields is not equivalent to replaying the same submission, because the operation identifier depends on what was signed. A receipt lookup, followed by a check of the relevant chain state, is the recovery path when the submitter loses its response. An accepted API response should mean “durably recorded,” not “paid,” because a signer or bundler can still reject work after acceptance. PostgreSQL beats a Redis lock for the source of truth There are two plausible defaults for coordinating wallet workers: a PostgreSQL intent row with a conditional claim, or a Redis SET key value NX PX lock. PostgreSQL wins when the claim and intent transition must remain auditable together, because its transaction can make the state change and ownership decision atomic. Its cost is write latency and contention on a busy signer or intent partition. Redis wins for short-lived, replaceable coordination where another durable record already defines the truth, because acquiring an expiring lock is simple and avoids a database write on that fast path. Its cost is that lock expiry or loss does not tell you whether the former holder signed or broadcast anything. For money-moving work, I would choose PostgreSQL as the authority and use Redis only as an optional scheduling optimization, because a worker crash must not turn an expired lock into permission to create a new transaction. Start with a 5-minute lease as a tuning input for a worker claim, but make expiry return the intent to reconciliation, not directly to signing. That rule matters when a process broadcasts successfully and dies before writing the response: the next worker must inspect the stored signed payload, receipt, nonce, and account state before deciding what action remains. The state machine should distinguish accepted, claimed, signed, broadcast, included, final, and needs_review. Those names are useful because they separate your own durable actions from observations about the chain. A transaction receipt showing inclusion is not the same thing as your application’s finality policy; a reorganization can remove a recently observed receipt. Configure that policy per chain rather than copying an Ethereum mainnet confirmation count to an L2, because the chains can expose different settlement and finality signals. Resist collapsing a stuck item into “failed” merely because the provider returned an error. An RPC timeout says that your client did not receive an answer, not that a node did not receive the bytes. Conversely, a nonce-too-low response does not by itself prove this intent succeeded, because another transaction from the signer may have consumed the nonce. The useful on-call question is whether the exact operation is known to be included, still eligible for broadcast, deliberately replaced, or genuinely unaccounted for. A recovery drill is more valuable than a throughput benchmark Put an end-to-end trace around intent acceptance, worker claim, signing, submission, and reconciliation with OpenTelemetry, but keep raw signing material out of spans because tracing backends are not key stores. Propagate an internal intent ID through Kubernetes jobs or queue messages; do not use the transaction hash as the only correlation ID, because it may not exist when the request first fails. With Prometheus, count state transitions and expose the age of the oldest non-final intent by chain and signer. Avoid a metric label containing every idempotency key, because unbounded label cardinality can overwhelm the monitoring system. As a proposed SLO rather than an observed result, target 99.9% availability for durably accepting valid wallet intents; measure chain completion separately because network fees and inclusion are outside the API acceptance boundary. Set an initial alert threshold when the oldest claimed or broadcast intent has made no verifiable progress for 15 minutes, then tune it against the chains and fee policies you operate. Alert on an aging state, not merely a low transaction count, because a quiet wallet can have no transactions while a single stranded transfer still needs intervention. Run a fault drill before raising Kubernetes Horizontal Pod Autoscaler worker limits. Inject a 30-second test timeout after submission but before the broadcaster records its response, then retry the same API key while two workers are active. The expected result is one intent row, no second nonce allocation for that intent, and a reconciliation path that either finds the submitted operation or marks the evidence insufficient for automated resubmission. That test catches a more consequential defect than a requests-per-second benchmark because it exercises the moment at which the caller, database, and chain can disagree. Budget two independently operated RPC providers as an operational starting point for investigation, not as a voting oracle. Comparing their receipt and pending-nonce views helps expose provider-local gaps, but disagreement should pause an irreversible decision rather than cause a third broadcast. Record which provider saw each submission and which block and receipt informed each transition, so the next on-call engineer can reconstruct the decision without relying on a dashboard’s current view. Prove one timeout cannot become two payments Start by selecting one production transfer path and finding the exact point where signed bytes can leave your process before their identity is durable. Add a fault-injection test at that boundary and retry the customer request with the same key. If the test creates a second nonce or operation, stop increasing worker concurrency: fix the intent record and reconciliation transition first. That is a release gate an on-call engineer can verify, not a promise that the network will never fail.</p>
<p>The post <a href="https://deepfriedbytes.com/2024-crypto-runbooks-fail-production-in-2026/">2024 Crypto Runbooks Fail Production in 2026</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A wallet request can time out after it has been broadcast, leaving the caller unable to tell whether retrying will create a second transfer. Since 2024, delegated accounts and more varied transaction paths have made that uncertainty harder to contain. My position: a production wallet needs a durable intent record before it needs another signing integration. If no component owns recovery, scaling workers only scales the ambiguity.</p>
<h2>The old build-or-buy boundary no longer matches the recovery boundary</h2>
<p>I disagree with <a href=/your-2024-crypto-build-vs-buy-playbook-is-obsolete/>Your 2024 Crypto Build-vs-Buy Playbook Is Obsolete</a> if its sourcing question is treated as the production decision: buying signing, RPC, or account-abstraction services does not buy an answer to “Did this customer request execute?” Your team still has to join the HTTP request, the signed operation, the broadcast attempt, and the eventual chain result into one recoverable record.</p>
<p>That distinction has become sharper over the last two years. Ethereum’s Pectra upgrade activated on May 7, 2025—a network date, not a performance claim—and introduced EIP-7702, which lets an externally owned account delegate execution to code. An EIP-7702-capable wallet can therefore have behavior that an older “EOA means one simple transaction” runbook does not describe. ERC-4337 adds another operational path: a UserOperation goes through a bundler and an EntryPoint rather than through the ordinary EOA transaction submission path. EntryPoint v0.7 matters as a version boundary because tooling that serializes a UserOperation for a different EntryPoint version can produce an operation the intended path will not accept.</p>
<p>These mechanisms do not make every wallet request interchangeable. EIP-5792’s <em>wallet_sendCalls</em> describes a wallet-facing call-batching interface, while <em>eth_sendRawTransaction</em> broadcasts an already signed EVM transaction; a successful response from either interface is not proof of final execution because inclusion and chain history can still change. A paymaster may also be involved in an ERC-4337 operation, so rebuilding an apparently identical request after a timeout can change its signed contents. Store the exact bytes or canonical operation fields that were submitted, not merely the API payload that started the process.</p>
<p>I would <strong>not</strong> make a vendor webhook the system of record, because a delayed, duplicated, or missing webhook cannot establish what your API already promised the caller. Use webhooks as evidence to reconcile against receipts and your durable intent record. Keep the vendor’s request ID, the chain ID, the signer or smart-account address, and the transaction or UserOperation hash together; without that mapping, an on-call engineer must guess which customer action a chain event represents.</p>
<h2>Idempotency now has to survive a signer and a broadcaster</h2>
<p><a href=/at-10x-users-your-crypto-wallet-breaks-at-idempotency/>At 10x users, your crypto wallet breaks at idempotency</a> names a real failure mode, but I would not use traffic growth as the trigger for fixing it: one timeout between signing and persistence is enough to make a retry unsafe. The operational contract should be narrower and testable: the same authenticated request key and the same request body identify one wallet intent, while a reused key with a different body is rejected. That contract prevents your API from deliberately creating a second intent; it does not claim that a blockchain provides exactly-once execution.</p>
<p>Put a uniqueness constraint around the intent before dispatching work. This PostgreSQL example runs in <em>psql</em>; its sample hash is a placeholder, so production code must compute the digest from a canonical request representation before insertion.</p>
<pre>CREATE TABLE wallet_intents (
  tenant text NOT NULL, chain_id bigint NOT NULL,
  idem_key text NOT NULL, body_hash text NOT NULL,
  state text NOT NULL DEFAULT 'accepted',
  tx_hash text, PRIMARY KEY (tenant, chain_id, idem_key)
);
INSERT INTO wallet_intents (tenant, chain_id, idem_key, body_hash)
VALUES ('demo', 1, 'request-7', 'sha256:sample')
ON CONFLICT (tenant, chain_id, idem_key) DO NOTHING;
SELECT body_hash, state, tx_hash FROM wallet_intents
WHERE tenant = 'demo' AND chain_id = 1 AND idem_key = 'request-7';</pre>
<p>The insert makes concurrent acceptance of the same key converge on one row because PostgreSQL enforces the primary key. It does <strong>not</strong> finish the API contract by itself: after a conflict, compare the stored <em>body_hash</em> with the incoming hash and return a conflict if they differ. Treat that check and any state claim as database transactions, because two workers that merely read “accepted” can both decide to sign. A worker should claim a dispatch state with a conditional update, record the resulting signed payload or hash, and only then proceed through the broadcast state machine.</p>
<p>Nonce management needs its own boundary. For an EOA, reserve a nonce per <em>chain ID plus signer</em> under database coordination; asking two RPC endpoints for <em>eth_getTransactionCount</em> with <em>pending</em> is not a lock, because their mempools need not contain the same transactions. Never allocate a fresh nonce just because <em>eth_sendRawTransaction</em> timed out: rebroadcasting the same signed bytes preserves the transaction hash, whereas a newly signed transaction can create a second payment. Replacement under EIP-1559 is a deliberate state transition with the same nonce and changed fee fields, not an automatic retry of the customer intent.</p>
<p>For ERC-4337, track the UserOperation hash, EntryPoint address, smart account, and bundler submission separately from the business intent. Calling <em>eth_sendUserOperation</em> again with altered operation fields is not equivalent to replaying the same submission, because the operation identifier depends on what was signed. A receipt lookup, followed by a check of the relevant chain state, is the recovery path when the submitter loses its response. An accepted API response should mean “durably recorded,” not “paid,” because a signer or bundler can still reject work after acceptance.</p>
<h2>PostgreSQL beats a Redis lock for the source of truth</h2>
<p>There are two plausible defaults for coordinating wallet workers: a PostgreSQL intent row with a conditional claim, or a Redis <em>SET key value NX PX</em> lock. PostgreSQL wins when the claim and intent transition must remain auditable together, because its transaction can make the state change and ownership decision atomic. Its cost is write latency and contention on a busy signer or intent partition. Redis wins for short-lived, replaceable coordination where another durable record already defines the truth, because acquiring an expiring lock is simple and avoids a database write on that fast path. Its cost is that lock expiry or loss does not tell you whether the former holder signed or broadcast anything.</p>
<p>For money-moving work, I would choose PostgreSQL as the authority and use Redis only as an optional scheduling optimization, because a worker crash must not turn an expired lock into permission to create a new transaction. Start with a <strong>5-minute lease as a tuning input</strong> for a worker claim, but make expiry return the intent to <em>reconciliation</em>, not directly to <em>signing</em>. That rule matters when a process broadcasts successfully and dies before writing the response: the next worker must inspect the stored signed payload, receipt, nonce, and account state before deciding what action remains.</p>
<p>The state machine should distinguish <em>accepted</em>, <em>claimed</em>, <em>signed</em>, <em>broadcast</em>, <em>included</em>, <em>final</em>, and <em>needs_review</em>. Those names are useful because they separate your own durable actions from observations about the chain. A transaction receipt showing inclusion is not the same thing as your application’s finality policy; a reorganization can remove a recently observed receipt. Configure that policy per chain rather than copying an Ethereum mainnet confirmation count to an L2, because the chains can expose different settlement and finality signals.</p>
<p>Resist collapsing a stuck item into “failed” merely because the provider returned an error. An RPC timeout says that your client did not receive an answer, not that a node did not receive the bytes. Conversely, a nonce-too-low response does not by itself prove this intent succeeded, because another transaction from the signer may have consumed the nonce. The useful on-call question is whether the exact operation is known to be included, still eligible for broadcast, deliberately replaced, or genuinely unaccounted for.</p>
<h2>A recovery drill is more valuable than a throughput benchmark</h2>
<p>Put an end-to-end trace around intent acceptance, worker claim, signing, submission, and reconciliation with OpenTelemetry, but keep raw signing material out of spans because tracing backends are not key stores. Propagate an internal intent ID through Kubernetes jobs or queue messages; do not use the transaction hash as the only correlation ID, because it may not exist when the request first fails. With Prometheus, count state transitions and expose the age of the oldest non-final intent by chain and signer. Avoid a metric label containing every idempotency key, because unbounded label cardinality can overwhelm the monitoring system.</p>
<p>As a <strong>proposed SLO rather than an observed result</strong>, target 99.9% availability for durably accepting valid wallet intents; measure chain completion separately because network fees and inclusion are outside the API acceptance boundary. Set an <strong>initial alert threshold</strong> when the oldest claimed or broadcast intent has made no verifiable progress for 15 minutes, then tune it against the chains and fee policies you operate. Alert on an aging state, not merely a low transaction count, because a quiet wallet can have no transactions while a single stranded transfer still needs intervention.</p>
<p>Run a fault drill before raising Kubernetes Horizontal Pod Autoscaler worker limits. Inject a <strong>30-second test timeout</strong> after submission but before the broadcaster records its response, then retry the same API key while two workers are active. The expected result is one intent row, no second nonce allocation for that intent, and a reconciliation path that either finds the submitted operation or marks the evidence insufficient for automated resubmission. That test catches a more consequential defect than a requests-per-second benchmark because it exercises the moment at which the caller, database, and chain can disagree.</p>
<p>Budget <strong>two independently operated RPC providers as an operational starting point</strong> for investigation, not as a voting oracle. Comparing their receipt and pending-nonce views helps expose provider-local gaps, but disagreement should pause an irreversible decision rather than cause a third broadcast. Record which provider saw each submission and which block and receipt informed each transition, so the next on-call engineer can reconstruct the decision without relying on a dashboard’s current view.</p>
<h2>Prove one timeout cannot become two payments</h2>
<p>Start by selecting one production transfer path and finding the exact point where signed bytes can leave your process before their identity is durable. Add a fault-injection test at that boundary and retry the customer request with the same key. If the test creates a second nonce or operation, stop increasing worker concurrency: fix the intent record and reconciliation transition first. That is a release gate an on-call engineer can verify, not a promise that the network will never fail.</p>
<p>The post <a href="https://deepfriedbytes.com/2024-crypto-runbooks-fail-production-in-2026/">2024 Crypto Runbooks Fail Production in 2026</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>CTO, Score Robotics Stacks Before You Build or Buy</title>
		<link>https://deepfriedbytes.com/cto-score-robotics-stacks-before-you-build-or-buy/</link>
		
		
		<pubDate>Tue, 13 Oct 2026 09:41:24 +0000</pubDate>
				<category><![CDATA[Robotics]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[Machine Learning Apps]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/cto-score-robotics-stacks-before-you-build-or-buy/</guid>

					<description><![CDATA[<p>A vision API can win a demo and lose the procurement decision: the invoice may be predictable while data retention, camera drift, and migration remain untested. For a first production app, I would buy inference only from a vendor that passes a priced exit test. If the contract or interface prevents that test, I would start with a licensed model running in-house, even if its initial accuracy is lower. A vendor trial should start with an exit test, not a leaderboard Before asking vendors for accuracy claims, write down what you would need to replace them. That means access to the original images where your rights permit it, human-reviewed labels, model outputs, confidence values, timestamps, and enough request metadata to replay a decision. A supplier that returns only a category name makes replacement harder because you cannot reconstruct thresholds or diagnose disagreements after the fact. Ask each candidate to process the same held-out images through the interface you would actually deploy. A useful trial set includes ordinary images, low-light captures, motion blur, empty frames, and examples from every camera type you expect to support. Set 2,000 labeled images as an initial trial size to tune, then increase it if important slices contain too few examples to estimate error rates. Keep near-duplicate frames from the same event together when splitting data; otherwise, a test can reward recognition of a scene rather than performance on a new one. Record results by slice, not just in one aggregate score. A measured 95% overall agreement in your trial would still be unacceptable if one camera’s false-accept rate exceeded the limit you set for that camera. For each vendor, report false accepts and false rejects at a threshold selected on a separate development set. Do not move the threshold after inspecting held-out results, because doing so turns the test set into another tuning set. The simplest comparison artifact is a JSON Lines file with one reviewed truth value and one prediction per candidate. The following script runs with Python’s standard library; give it an eval.jsonl file whose rows contain slice, truth, api, and onnx fields. Use the same categorical output vocabulary for both candidates. python3 - &#60;&#60;'PY' import json from collections import defaultdict rows = [json.loads(s) for s in open("eval.jsonl") if s.strip()] by = defaultdict(lambda: [0, 0]) for r in rows: for name in ("api", "onnx"): key = (name, r["slice"]) by[key][0] += int(r[name] != r["truth"]) by[key][1] += 1 for (name, sl), (errors, n) in sorted(by.items()): print(name, sl, f"errors={errors}/{n}", f"rate={errors/n:.3f}") PY That script counts disagreements, not detection quality; use intersection over union and class-specific precision and recall when the app depends on bounding boxes. Make a human review procedure part of the trial, since “ground truth” produced by one of the competing services would favor that service. I would not accept a vendor’s public benchmark as the procurement test, because its image distribution, annotation rules, and decision threshold are unlikely to match yours. Amazon Rekognition wins on setup; ONNX Runtime wins on control Compare two concrete options: Amazon Rekognition DetectLabels as the bought service and ONNX Runtime 1.20 running a properly licensed vision model as the in-house path. Rekognition wins when its labels match your decision, request volume is uncertain, and you need an operational endpoint quickly; you pay per applicable API request, plus engineering time for mapping labels, handling failures, and reviewing AWS service terms. ONNX Runtime wins when you need a fixed preprocessing path, local inference, or tighter control over output formats; you pay for model selection, compute, deployment, monitoring, and an owner for every regression. Neither option should receive credit for work it does not perform. Rekognition’s MinConfidence parameter can filter returned labels, but a setting of 80 is a threshold to tune against your error costs, not proof that returned results are correct 80% of the time. ONNX Runtime executes a model efficiently, but it does not supply an appropriate model, usable training data, or a license to ship the weights. If you consider an Ultralytics YOLO11 export, review its AGPL-3.0 and enterprise licensing paths with counsel before treating an ONNX file as unrestricted. Use OpenCV 4.10 to make resizing, color conversion, and image orientation explicit in the trial harness. Record the exported ONNX opset, such as 17 where compatible, and the ONNX Runtime execution provider, such as CPUExecutionProvider or CUDAExecutionProvider; these settings matter because a benchmark from one preprocessing and hardware path does not establish production latency on another. If deployment needs NVIDIA TensorRT 10 or Triton Inference Server 2.50, test the exported graph there before approving a model, since a successful notebook inference does not prove the target runtime accepts every operation. Set a 200 ms p95 server-side inference budget as a provisional design target if that fits your app, and measure it under the concurrency you intend to support. Separate decode, preprocessing, inference, and network time using OpenTelemetry traces; a vendor may have fast model execution while upload and response transfer dominate the user-visible wait. Track a Prometheus latency histogram rather than an average, because the p95 estimated with histogram_quantile() will expose slow requests that an average conceals. Read Custom Vision Model or Pretrained API Which Fits Your First App before soliciting model proposals. I would make reversibility the earlier decision, because an excellent model is a poor purchase if you cannot reproduce its inputs and outputs elsewhere. The contract can erase an apparent accuracy advantage Run the technical trial in parallel with a contract review. Ask whether uploaded images are retained, whether they can be used to improve the supplier’s services, where processing occurs, how deletion is verified, and what changes require notice. Get the answers in the governing agreement or service terms, not only in a sales email, because your team will need enforceable terms if policies change. Price three years of use with a volume model you can explain. For example, treat 1 million images per month as a planning scenario to revise, and include retries, duplicate submissions, minimum commitments, storage, outbound transfer, and human review. Price the in-house alternative with compute, a spare-capacity allowance, model maintenance, and on-call time. Compare both at low and high volume: per-request pricing can be attractive while demand is uncertain, whereas owned capacity becomes easier to justify when utilization is steady. Then ask the vendor to perform an exit drill. Give its team a written request for your available images, annotations, outputs, and configuration, and time how long it takes to receive a usable export. Require stable identifiers so exported decisions can be joined to your records. A promise that “data is portable” is weak evidence because portability depends on file formats, identifiers, rights, and the effort needed to replay decisions. Read CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated after pricing that exit. A revised build-versus-buy argument is useful, but I would reject any shortcut from “buying is faster” to “this supplier is replaceable”; the second claim requires an export and replay test. Open-source choices need a contract review too. Check model-weight terms separately from framework terms, and retain a software bill of materials using SPDX 3.0 or CycloneDX 1.6 if your organization supports those formats. ONNX Runtime’s license does not grant rights to every model loaded into it, so record the provenance of weights, training datasets where known, and any commercial-use restrictions before counting self-hosting as the safer option. A purchase decision should have a failure gate and an owner Turn the trial into a short approval memo with gates that can fail. One gate should cover error by image slice; another should cover latency under load; a third should cover deletion, export, and contract terms. Assign an owner to each gate, since an unowned exception is likely to become an implicit approval. Set a 30-day shadow period as an initial operating window to adjust, during which the candidate produces results without controlling the final decision. During shadow operation, log the model or API version, request identifier, preprocessing version, threshold, and human override. Sample disagreements for review rather than sending every image back to an annotator, because review cost can overwhelm a low per-request inference price. Give reviewers a documented rule for ambiguous images; inconsistent labels can make either option appear unstable even when its predictions have not changed. Write the rollback trigger before launch. For example, choose a 1% false-accept ceiling for a high-cost decision only if your product owner agrees that it reflects the real cost of errors; otherwise, set a different value and document why. Trigger rollback on a sustained breach in the affected slice, not merely on an overall average. Keep the previous inference path callable until you have tested rollback, because a feature flag that points to a retired endpoint is not a recovery plan. Finally, decide who can change the system. A managed API still needs an owner for changed label taxonomies, quotas, authentication, and vendor notices. A self-hosted model needs an owner for patches, capacity, model updates, and drift investigations. Approve the option whose ongoing duties have named people and budget, since an apparently cheaper path can become expensive when those duties surface during an incident. The first approval artifact should be a paired test set Tomorrow, select one real decision your app makes and collect its first reviewed, consent-appropriate image sample. Define the output schema and error cost before inviting either vendor demos or model proposals. Then send both candidates the same test and ask for a priced export. The winner is the option your team can operate and replace, not the one with the most persuasive demo.</p>
<p>The post <a href="https://deepfriedbytes.com/cto-score-robotics-stacks-before-you-build-or-buy/">CTO, Score Robotics Stacks Before You Build or Buy</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A vision API can win a demo and lose the procurement decision: the invoice may be predictable while data retention, camera drift, and migration remain untested. For a first production app, I would buy inference only from a vendor that passes a priced exit test. If the contract or interface prevents that test, I would start with a licensed model running in-house, even if its initial accuracy is lower.</p>
<h2>A vendor trial should start with an exit test, not a leaderboard</h2>
<p>Before asking vendors for accuracy claims, write down what you would need to replace them. That means access to the original images where your rights permit it, human-reviewed labels, model outputs, confidence values, timestamps, and enough request metadata to replay a decision. A supplier that returns only a category name makes replacement harder because you cannot reconstruct thresholds or diagnose disagreements after the fact.</p>
<p>Ask each candidate to process the same held-out images through the interface you would actually deploy. A useful trial set includes ordinary images, low-light captures, motion blur, empty frames, and examples from every camera type you expect to support. Set <strong>2,000 labeled images</strong> as an initial trial size to tune, then increase it if important slices contain too few examples to estimate error rates. Keep near-duplicate frames from the same event together when splitting data; otherwise, a test can reward recognition of a scene rather than performance on a new one.</p>
<p>Record results by slice, not just in one aggregate score. A measured <strong>95% overall agreement</strong> in your trial would still be unacceptable if one camera’s false-accept rate exceeded the limit you set for that camera. For each vendor, report false accepts and false rejects at a threshold selected on a separate development set. Do not move the threshold after inspecting held-out results, because doing so turns the test set into another tuning set.</p>
<p>The simplest comparison artifact is a JSON Lines file with one reviewed truth value and one prediction per candidate. The following script runs with Python’s standard library; give it an <code>eval.jsonl</code> file whose rows contain <code>slice</code>, <code>truth</code>, <code>api</code>, and <code>onnx</code> fields. Use the same categorical output vocabulary for both candidates.</p>
<pre>python3 - &lt;&lt;'PY'
import json
from collections import defaultdict
rows = [json.loads(s) for s in open("eval.jsonl") if s.strip()]
by = defaultdict(lambda: [0, 0])
for r in rows:
    for name in ("api", "onnx"):
        key = (name, r["slice"])
        by[key][0] += int(r[name] != r["truth"])
        by[key][1] += 1
for (name, sl), (errors, n) in sorted(by.items()):
    print(name, sl, f"errors={errors}/{n}", f"rate={errors/n:.3f}")
PY</pre>
<p>That script counts disagreements, not detection quality; use intersection over union and class-specific precision and recall when the app depends on bounding boxes. Make a human review procedure part of the trial, since “ground truth” produced by one of the competing services would favor that service. I would <strong>not</strong> accept a vendor’s public benchmark as the procurement test, because its image distribution, annotation rules, and decision threshold are unlikely to match yours.</p>
<h2>Amazon Rekognition wins on setup; ONNX Runtime wins on control</h2>
<p>Compare two concrete options: <strong>Amazon Rekognition DetectLabels</strong> as the bought service and <strong>ONNX Runtime 1.20</strong> running a properly licensed vision model as the in-house path. Rekognition wins when its labels match your decision, request volume is uncertain, and you need an operational endpoint quickly; you pay per applicable API request, plus engineering time for mapping labels, handling failures, and reviewing AWS service terms. ONNX Runtime wins when you need a fixed preprocessing path, local inference, or tighter control over output formats; you pay for model selection, compute, deployment, monitoring, and an owner for every regression.</p>
<p>Neither option should receive credit for work it does not perform. Rekognition’s <code>MinConfidence</code> parameter can filter returned labels, but a setting of <strong>80</strong> is a threshold to tune against your error costs, not proof that returned results are correct 80% of the time. ONNX Runtime executes a model efficiently, but it does not supply an appropriate model, usable training data, or a license to ship the weights. If you consider an Ultralytics YOLO11 export, review its AGPL-3.0 and enterprise licensing paths with counsel before treating an ONNX file as unrestricted.</p>
<p>Use OpenCV 4.10 to make resizing, color conversion, and image orientation explicit in the trial harness. Record the exported ONNX opset, such as <strong>17</strong> where compatible, and the ONNX Runtime execution provider, such as <code>CPUExecutionProvider</code> or <code>CUDAExecutionProvider</code>; these settings matter because a benchmark from one preprocessing and hardware path does not establish production latency on another. If deployment needs NVIDIA TensorRT 10 or Triton Inference Server 2.50, test the exported graph there before approving a model, since a successful notebook inference does not prove the target runtime accepts every operation.</p>
<p>Set a <strong>200 ms p95</strong> server-side inference budget as a provisional design target if that fits your app, and measure it under the concurrency you intend to support. Separate decode, preprocessing, inference, and network time using OpenTelemetry traces; a vendor may have fast model execution while upload and response transfer dominate the user-visible wait. Track a Prometheus latency histogram rather than an average, because the p95 estimated with <code>histogram_quantile()</code> will expose slow requests that an average conceals.</p>
<p>Read <a href=/custom-vision-model-or-pretrained-api-which-fits-your-first-app/>Custom Vision Model or Pretrained API Which Fits Your First App</a> before soliciting model proposals. I would make reversibility the earlier decision, because an excellent model is a poor purchase if you cannot reproduce its inputs and outputs elsewhere.</p>
<h2>The contract can erase an apparent accuracy advantage</h2>
<p>Run the technical trial in parallel with a contract review. Ask whether uploaded images are retained, whether they can be used to improve the supplier’s services, where processing occurs, how deletion is verified, and what changes require notice. Get the answers in the governing agreement or service terms, not only in a sales email, because your team will need enforceable terms if policies change.</p>
<p>Price three years of use with a volume model you can explain. For example, treat <strong>1 million images per month</strong> as a planning scenario to revise, and include retries, duplicate submissions, minimum commitments, storage, outbound transfer, and human review. Price the in-house alternative with compute, a spare-capacity allowance, model maintenance, and on-call time. Compare both at low and high volume: per-request pricing can be attractive while demand is uncertain, whereas owned capacity becomes easier to justify when utilization is steady.</p>
<p>Then ask the vendor to perform an exit drill. Give its team a written request for your available images, annotations, outputs, and configuration, and time how long it takes to receive a usable export. Require stable identifiers so exported decisions can be joined to your records. A promise that “data is portable” is weak evidence because portability depends on file formats, identifiers, rights, and the effort needed to replay decisions.</p>
<p>Read <a href=/ctos-your-2024-genai-build-vs-buy-playbook-is-outdated/>CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated</a> after pricing that exit. A revised build-versus-buy argument is useful, but I would reject any shortcut from “buying is faster” to “this supplier is replaceable”; the second claim requires an export and replay test.</p>
<p>Open-source choices need a contract review too. Check model-weight terms separately from framework terms, and retain a software bill of materials using SPDX 3.0 or CycloneDX 1.6 if your organization supports those formats. ONNX Runtime’s license does not grant rights to every model loaded into it, so record the provenance of weights, training datasets where known, and any commercial-use restrictions before counting self-hosting as the safer option.</p>
<h2>A purchase decision should have a failure gate and an owner</h2>
<p>Turn the trial into a short approval memo with gates that can fail. One gate should cover error by image slice; another should cover latency under load; a third should cover deletion, export, and contract terms. Assign an owner to each gate, since an unowned exception is likely to become an implicit approval. Set a <strong>30-day</strong> shadow period as an initial operating window to adjust, during which the candidate produces results without controlling the final decision.</p>
<p>During shadow operation, log the model or API version, request identifier, preprocessing version, threshold, and human override. Sample disagreements for review rather than sending every image back to an annotator, because review cost can overwhelm a low per-request inference price. Give reviewers a documented rule for ambiguous images; inconsistent labels can make either option appear unstable even when its predictions have not changed.</p>
<p>Write the rollback trigger before launch. For example, choose a <strong>1% false-accept ceiling</strong> for a high-cost decision only if your product owner agrees that it reflects the real cost of errors; otherwise, set a different value and document why. Trigger rollback on a sustained breach in the affected slice, not merely on an overall average. Keep the previous inference path callable until you have tested rollback, because a feature flag that points to a retired endpoint is not a recovery plan.</p>
<p>Finally, decide who can change the system. A managed API still needs an owner for changed label taxonomies, quotas, authentication, and vendor notices. A self-hosted model needs an owner for patches, capacity, model updates, and drift investigations. Approve the option whose ongoing duties have named people and budget, since an apparently cheaper path can become expensive when those duties surface during an incident.</p>
<h2>The first approval artifact should be a paired test set</h2>
<p>Tomorrow, select one real decision your app makes and collect its first reviewed, consent-appropriate image sample. Define the output schema and error cost before inviting either vendor demos or model proposals. Then send both candidates the same test and ask for a priced export. The winner is the option your team can operate and replace, not the one with the most persuasive demo.</p>
<p>The post <a href="https://deepfriedbytes.com/cto-score-robotics-stacks-before-you-build-or-buy/">CTO, Score Robotics Stacks Before You Build or Buy</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>RAG vs Fine-Tuning in Your First GenAI Backend</title>
		<link>https://deepfriedbytes.com/rag-vs-fine-tuning-in-your-first-genai-backend/</link>
		
		
		<pubDate>Tue, 06 Oct 2026 09:58:34 +0000</pubDate>
				<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[AI Web Solutions]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/rag-vs-fine-tuning-in-your-first-genai-backend/</guid>

					<description><![CDATA[<p>A coding assistant does not need to understand an entire repository before it can answer a developer’s question. It needs to find the right code, show where it found it, and avoid presenting stale guesses as facts. For a first implementation, I would choose live repository search over a vector index. Embeddings become worthwhile when real questions repeatedly defeat symbol and text search—not simply because the feature uses generative AI. Live repository search is the better default for code questions Consider a developer asking, “Where do we reject an expired session token?” The likely answer has a function name, an error string, a test, or a call site that ripgrep can find in the current checkout. A coding assistant can search those candidates, read bounded excerpts, and ask the model to explain the path from request handling to rejection. This approach wins when questions mention identifiers, configuration keys, endpoints, stack traces, or behavior with recognizable vocabulary, because those clues already exist as searchable text. Live search also gives a backend developer a familiar consistency model: the assistant reads the same revision the developer is inspecting. git supplies the commit SHA; rg supplies file paths and line numbers; Tree-sitter can expand a hit to its containing function when a few matching lines are insufficient. No background indexing job has to catch up after a merge. Treat Generative AI for Software Development: Practical Use Cases as a menu of candidate tasks rather than evidence that every coding task needs an embeddings index. The first version can be deliberately small. This Python 3 script searches a checked-out repository for a literal phrase, excludes two common dependency directories, and prints a bounded set of matches. Run it from the repository root after installing ripgrep: #!/usr/bin/env python3 import subprocess import sys if len(sys.argv) != 2: raise SystemExit("usage: python3 search.py PHRASE") cmd = ["rg", "-n", "-F", "-m", "2", "--glob", "!vendor/**", "--glob", "!node_modules/**", "--", sys.argv[1], "."] result = subprocess.run(cmd, text=True, capture_output=True) if result.returncode not in (0, 1): raise SystemExit(result.stderr) print("\n".join(result.stdout.splitlines()[:40])) Here, -F would normally be written as an inline flag, but the important behavior is that ripgrep treats the query as a literal phrase rather than a regular expression. The 40-line output cap is a starting limit to tune against answer quality and prompt size, not a measured optimum. In a service, avoid collecting every match before applying that cap: search within authorized paths, stream results, and stop once enough distinct files have been inspected. Add a second pass for call sites or tests only when the first pass leaves an unresolved question. I would not embed the entire repository on day one, because an index introduces update, deletion, and access-control work before there is evidence that lexical search misses important answers. Live search has costs too: repeated disk reads, poor matches for questions that use different words from the code, and the need to rank noisy hits. Those costs are visible in a request trace, which makes them easier to justify or fix than an index built on assumption. Vector retrieval wins when developers ask about concepts the code never names Now change the question to, “Why can a cancelled job still produce an invoice?” The relevant implementation may be spread across a queue consumer, a state transition, and an event handler without using the words “cancelled” and “invoice” together. A preindexed vector search can surface semantically related passages that literal matching misses. It is more likely to win across several repositories, long design documents, or renamed subsystems, because users cannot reliably guess the identifiers that tie those sources together. The competing approaches have distinct bills. Live search with ripgrep wins for fresh, identifier-rich questions and costs request-time I/O plus ranking logic. Preindexed retrieval with pgvector wins when conceptual discovery repeatedly matters and costs embedding generation, storage, reindexing, permission filtering, and investigation of stale results. A vector hit is only a candidate, because similarity does not establish that a passage describes the current behavior. If you add an index, make its unit of retrieval useful to a programmer. Chunking at arbitrary character boundaries can separate a condition from the branch it guards, so prefer a function or similarly coherent unit identified with Tree-sitter. Store repository, path, revision, symbol, and access scope beside each chunk. PostgreSQL with pgvector can hold embeddings and metadata together; an HNSW index can accelerate approximate nearest-neighbor search, although its recall and query cost still need testing. The hnsw.ef_search setting is a search-effort knob to tune rather than a value to copy from another workload. Do not force a choice between semantic and exact matches inside the indexed approach. A developer who pastes SessionExpiredError expects the exact symbol to rank highly, so combine vector candidates with BM25 results from a search engine such as Elasticsearch, then rerank the merged set. An initial top-k of 8 is an experiment setting: increase it if relevant files are missed, or decrease it if excerpts crowd out the question and answer. Keep the final context tied to a revision so the model can cite code the developer can still open. I agree with Your GenAI Feature Scales Until Retrieval and Latency Collide that retrieval latency becomes a production constraint, but index freshness deserves equal attention because a fast answer from yesterday’s code can be worse than a slower answer from the current commit. The vector approach earns its extra machinery only when it improves grounded answers under both constraints. Answer quality, not search relevance alone, should decide the winner A search result can look relevant and still lead the model to the wrong conclusion. Build an evaluation set from questions developers actually ask in code review, incident follow-up, and onboarding. For each question, record the revision, acceptable source locations, and what a correct answer must say. A 30-question pilot is a planning size for the first comparison, not a statistically persuasive benchmark; its purpose is to expose failure modes before anyone optimizes a leaderboard number. Run both retrieval approaches against the same repository revisions and the same answer model. Count whether a necessary file appears in the retrieved set using Recall@k, and use MRR to notice when it appears only after several distractions. Neither metric says whether the final explanation is sound, so also check whether the answer cites lines that support its claims, distinguishes tests from production behavior, and says “I cannot establish that” when the evidence is incomplete. Reviewers should mark a correct conclusion with an unsupported citation as a failure, because the assistant has given the developer no reliable way to verify it. Include adversarially ordinary questions: a symbol renamed last week, documentation that contradicts implementation, and two functions with similar names in different services. Re-run the set after a merge to see whether indexed results still point at the intended revision. If vector retrieval improves conceptual questions but regresses exact-symbol questions, add hybrid ranking before declaring it the winner; that fix targets the observed failure rather than replacing a useful search method wholesale. Measure the request path with OpenTelemetry spans for search, excerpt loading, model input preparation, and generation. Use p95 latency rather than only the mean, because slow repository reads or index queries affect the developer waiting on an answer even when most requests are quick. An 800 ms retrieval budget is an example service objective to adjust after measuring your own environment, not a claim about either approach’s speed. Record input tokens and retrieval calls as well, since an apparently cheap search can become expensive when it sends large, repetitive excerpts to the model. The decision rule should be hard to game: adopt the index for a question class only if it raises supported-answer quality without breaching that class’s latency and freshness requirements. Keep live search available for exact identifiers even after adding vectors, because a semantic ranking model has no reason to treat an exact error code as more important than a nearby conceptual match unless the system explicitly tells it to. Permissions and freshness can overturn a retrieval win A repository assistant is also an authorization system. Live search should run only over paths the requester may read; an indexed service must apply the same restriction before returning candidates. Filtering forbidden excerpts after a model has seen them is too late, because the answer can disclose their content. Store access metadata with indexed chunks, invalidate it when permissions change, and test that an unauthorized user cannot retrieve a restricted file through either exact or semantic queries. Freshness needs an explicit contract. Suppose an index is allowed to lag the main branch by 15 minutes; treat that as a proposed policy ceiling, not a property of the indexing software. Attach the indexed commit SHA to every result and compare it with the requested revision. If they differ beyond the permitted lag, fall back to live search or state plainly that the answer uses an older revision. Silent fallback to stale context makes a code explanation difficult to debug, because the developer and assistant may be reasoning about different programs. Retrieved repository text is untrusted input, even when it comes from a file the developer can read. A comment can tell an assistant to ignore instructions or reveal credentials; it remains source material, not an instruction from the user. Keep tool permissions separate from the language model’s retrieved context, and avoid passing secrets or whole environment files merely because they matched a search. These controls apply to both approaches, while a persistent index adds the obligation to remove deleted or newly restricted content from stored copies. Start by collecting real developer questions and implementing bounded, permission-aware live search at a pinned commit. Ask reviewers to mark the files and lines each answer actually needed. Once that baseline exists, test an indexed retriever on the questions it misses. If vectors cannot produce more supported answers at an acceptable freshness and latency cost, keep the simpler system and improve its ranking instead.</p>
<p>The post <a href="https://deepfriedbytes.com/rag-vs-fine-tuning-in-your-first-genai-backend/">RAG vs Fine-Tuning in Your First GenAI Backend</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A coding assistant does not need to understand an entire repository before it can answer a developer’s question. It needs to find the right code, show where it found it, and avoid presenting stale guesses as facts. For a first implementation, I would choose live repository search over a vector index. Embeddings become worthwhile when real questions repeatedly defeat symbol and text search—not simply because the feature uses generative AI.</p>
<h2>Live repository search is the better default for code questions</h2>
<p>Consider a developer asking, “Where do we reject an expired session token?” The likely answer has a function name, an error string, a test, or a call site that <strong>ripgrep</strong> can find in the current checkout. A coding assistant can search those candidates, read bounded excerpts, and ask the model to explain the path from request handling to rejection. This approach wins when questions mention identifiers, configuration keys, endpoints, stack traces, or behavior with recognizable vocabulary, because those clues already exist as searchable text.</p>
<p>Live search also gives a backend developer a familiar consistency model: the assistant reads the same revision the developer is inspecting. <strong>git</strong> supplies the commit SHA; <strong>rg</strong> supplies file paths and line numbers; <strong>Tree-sitter</strong> can expand a hit to its containing function when a few matching lines are insufficient. No background indexing job has to catch up after a merge. Treat <a href=/generative-ai-for-software-development-practical-use-cases/>Generative AI for Software Development: Practical Use Cases</a> as a menu of candidate tasks rather than evidence that every coding task needs an embeddings index.</p>
<p>The first version can be deliberately small. This Python 3 script searches a checked-out repository for a literal phrase, excludes two common dependency directories, and prints a bounded set of matches. Run it from the repository root after installing <strong>ripgrep</strong>:</p>
<pre>#!/usr/bin/env python3
import subprocess
import sys
if len(sys.argv) != 2:
    raise SystemExit("usage: python3 search.py PHRASE")
cmd = ["rg", "-n", "-F", "-m", "2", "--glob", "!vendor/**",
       "--glob", "!node_modules/**", "--", sys.argv[1], "."]
result = subprocess.run(cmd, text=True, capture_output=True)
if result.returncode not in (0, 1):
    raise SystemExit(result.stderr)
print("\n".join(result.stdout.splitlines()[:40]))</pre>
<p>Here, <code>-F</code> would normally be written as an inline flag, but the important behavior is that ripgrep treats the query as a literal phrase rather than a regular expression. The <strong>40-line output cap</strong> is a starting limit to tune against answer quality and prompt size, not a measured optimum. In a service, avoid collecting every match before applying that cap: search within authorized paths, stream results, and stop once enough distinct files have been inspected. Add a second pass for call sites or tests only when the first pass leaves an unresolved question.</p>
<p>I would <strong>not</strong> embed the entire repository on day one, because an index introduces update, deletion, and access-control work before there is evidence that lexical search misses important answers. Live search has costs too: repeated disk reads, poor matches for questions that use different words from the code, and the need to rank noisy hits. Those costs are visible in a request trace, which makes them easier to justify or fix than an index built on assumption.</p>
<h2>Vector retrieval wins when developers ask about concepts the code never names</h2>
<p>Now change the question to, “Why can a cancelled job still produce an invoice?” The relevant implementation may be spread across a queue consumer, a state transition, and an event handler without using the words “cancelled” and “invoice” together. A preindexed vector search can surface semantically related passages that literal matching misses. It is more likely to win across several repositories, long design documents, or renamed subsystems, because users cannot reliably guess the identifiers that tie those sources together.</p>
<p>The competing approaches have distinct bills. <strong>Live search with ripgrep</strong> wins for fresh, identifier-rich questions and costs request-time I/O plus ranking logic. <strong>Preindexed retrieval with pgvector</strong> wins when conceptual discovery repeatedly matters and costs embedding generation, storage, reindexing, permission filtering, and investigation of stale results. A vector hit is only a candidate, because similarity does not establish that a passage describes the current behavior.</p>
<p>If you add an index, make its unit of retrieval useful to a programmer. Chunking at arbitrary character boundaries can separate a condition from the branch it guards, so prefer a function or similarly coherent unit identified with Tree-sitter. Store repository, path, revision, symbol, and access scope beside each chunk. <strong>PostgreSQL</strong> with <strong>pgvector</strong> can hold embeddings and metadata together; an HNSW index can accelerate approximate nearest-neighbor search, although its recall and query cost still need testing. The <strong>hnsw.ef_search</strong> setting is a search-effort knob to tune rather than a value to copy from another workload.</p>
<p>Do not force a choice between semantic and exact matches inside the indexed approach. A developer who pastes <em>SessionExpiredError</em> expects the exact symbol to rank highly, so combine vector candidates with <strong>BM25</strong> results from a search engine such as <strong>Elasticsearch</strong>, then rerank the merged set. An initial <strong>top-k of 8</strong> is an experiment setting: increase it if relevant files are missed, or decrease it if excerpts crowd out the question and answer. Keep the final context tied to a revision so the model can cite code the developer can still open.</p>
<p>I agree with <a href=/your-genai-feature-scales-until-retrieval-and-latency-collide/>Your GenAI Feature Scales Until Retrieval and Latency Collide</a> that retrieval latency becomes a production constraint, but index freshness deserves equal attention because a fast answer from yesterday’s code can be worse than a slower answer from the current commit. The vector approach earns its extra machinery only when it improves grounded answers under both constraints.</p>
<h2>Answer quality, not search relevance alone, should decide the winner</h2>
<p>A search result can look relevant and still lead the model to the wrong conclusion. Build an evaluation set from questions developers actually ask in code review, incident follow-up, and onboarding. For each question, record the revision, acceptable source locations, and what a correct answer must say. A <strong>30-question pilot</strong> is a planning size for the first comparison, not a statistically persuasive benchmark; its purpose is to expose failure modes before anyone optimizes a leaderboard number.</p>
<p>Run both retrieval approaches against the same repository revisions and the same answer model. Count whether a necessary file appears in the retrieved set using <strong>Recall@k</strong>, and use <strong>MRR</strong> to notice when it appears only after several distractions. Neither metric says whether the final explanation is sound, so also check whether the answer cites lines that support its claims, distinguishes tests from production behavior, and says “I cannot establish that” when the evidence is incomplete. Reviewers should mark a correct conclusion with an unsupported citation as a failure, because the assistant has given the developer no reliable way to verify it.</p>
<p>Include adversarially ordinary questions: a symbol renamed last week, documentation that contradicts implementation, and two functions with similar names in different services. Re-run the set after a merge to see whether indexed results still point at the intended revision. If vector retrieval improves conceptual questions but regresses exact-symbol questions, add hybrid ranking before declaring it the winner; that fix targets the observed failure rather than replacing a useful search method wholesale.</p>
<p>Measure the request path with <strong>OpenTelemetry</strong> spans for search, excerpt loading, model input preparation, and generation. Use <strong>p95 latency</strong> rather than only the mean, because slow repository reads or index queries affect the developer waiting on an answer even when most requests are quick. An <strong>800 ms retrieval budget</strong> is an example service objective to adjust after measuring your own environment, not a claim about either approach’s speed. Record input tokens and retrieval calls as well, since an apparently cheap search can become expensive when it sends large, repetitive excerpts to the model.</p>
<p>The decision rule should be hard to game: adopt the index for a question class only if it raises supported-answer quality without breaching that class’s latency and freshness requirements. Keep live search available for exact identifiers even after adding vectors, because a semantic ranking model has no reason to treat an exact error code as more important than a nearby conceptual match unless the system explicitly tells it to.</p>
<h2>Permissions and freshness can overturn a retrieval win</h2>
<p>A repository assistant is also an authorization system. Live search should run only over paths the requester may read; an indexed service must apply the same restriction <em>before</em> returning candidates. Filtering forbidden excerpts after a model has seen them is too late, because the answer can disclose their content. Store access metadata with indexed chunks, invalidate it when permissions change, and test that an unauthorized user cannot retrieve a restricted file through either exact or semantic queries.</p>
<p>Freshness needs an explicit contract. Suppose an index is allowed to lag the main branch by <strong>15 minutes</strong>; treat that as a proposed policy ceiling, not a property of the indexing software. Attach the indexed commit SHA to every result and compare it with the requested revision. If they differ beyond the permitted lag, fall back to live search or state plainly that the answer uses an older revision. Silent fallback to stale context makes a code explanation difficult to debug, because the developer and assistant may be reasoning about different programs.</p>
<p>Retrieved repository text is untrusted input, even when it comes from a file the developer can read. A comment can tell an assistant to ignore instructions or reveal credentials; it remains source material, not an instruction from the user. Keep tool permissions separate from the language model’s retrieved context, and avoid passing secrets or whole environment files merely because they matched a search. These controls apply to both approaches, while a persistent index adds the obligation to remove deleted or newly restricted content from stored copies.</p>
<p>Start by collecting real developer questions and implementing bounded, permission-aware live search at a pinned commit. Ask reviewers to mark the files and lines each answer actually needed. Once that baseline exists, test an indexed retriever on the questions it misses. If vectors cannot produce more supported answers at an acceptable freshness and latency cost, keep the simpler system and improve its ranking instead.</p>
<p>The post <a href="https://deepfriedbytes.com/rag-vs-fine-tuning-in-your-first-genai-backend/">RAG vs Fine-Tuning in Your First GenAI Backend</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated</title>
		<link>https://deepfriedbytes.com/ctos-your-2024-genai-build-vs-buy-playbook-is-outdated/</link>
		
		
		<pubDate>Wed, 30 Sep 2026 11:05:16 +0000</pubDate>
				<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[AI Web Solutions]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/ctos-your-2024-genai-build-vs-buy-playbook-is-outdated/</guid>

					<description><![CDATA[<p>“Build or buy the AI” is the wrong question for a CTO in 2026. The useful boundary is narrower: buy model capability until your own evidence shows that owning inference or training improves the economics. Build the evaluation, data controls, and fallback paths yourself. Those assets remain valuable when an API price changes, an open-weight model improves, or a supplier’s terms stop fitting your product. The default has shifted from owning a model to owning the decision boundary Two years ago, advice to prototype with an API and then train a custom model often sounded like a natural progression. I would no longer treat training as the next stage. Better multimodal APIs have made a purchased baseline more useful, while serving software such as vLLM has made self-hosting a credible alternative without making training necessary. These changes separate three decisions that older build-versus-buy advice bundled together: whose weights to use, where inference runs, and who controls the product’s acceptance criteria. My default is to buy access to a capable model, own the evaluation set, and keep the inference boundary replaceable. I would not train a proprietary vision model or fine-tune a language model merely because a pilot works, because pilot traffic rarely reveals enough about failure distribution or operating cost to justify an irreversible investment. Training becomes defensible when a labeled dataset represents production conditions, a purchased baseline misses a quantified requirement, and the expected gain exceeds annotation, retraining, and serving costs. This is not an argument for letting a supplier define success. A CTO should own the ground-truth examples, review rules, data-retention requirements, and release gate. For generated answers, that gate may require an answer to cite an authorized source and refuse when evidence is absent. For images, it may require a specified precision at a tolerable false-rejection rate. Those are product decisions; a vendor’s aggregate benchmark cannot substitute for them because its test distribution and error costs differ from yours. Custom Vision Model or Pretrained API Which Fits Your First App frames the first application as a model choice; I would make the first choice a reversible contract and test harness instead, because a promising first model says little about how cheaply the second one can be substituted. Put API calls behind an internal interface that records model identifier, prompt or preprocessing version, request cost, latency, and adjudicated outcome. That interface is worth building even if the supplier is never replaced. Recent platform changes have made old procurement shortcuts unreliable OpenAI’s Structured Outputs, introduced in 2024 with JSON Schema enforcement through strict: true on supported models, weakened the old claim that teams must fine-tune simply to get parseable fields. Schema conformity does not establish factual correctness, however, because a well-formed JSON object can still contain an invented value. Similarly, multimodal APIs can establish an image baseline quickly, but their availability does not prove that their terms permit every image to be sent off-site. The current procurement question is therefore less “Can an API do this?” and more “Which obligations survive a provider change?” Self-hosting has also changed. vLLM’s OpenAI-compatible server and Hugging Face Transformers reduce the work needed to try open-weight models, while ONNX Runtime can simplify deployment of some vision pipelines. None removes the cost of operating them: somebody still owns GPU capacity, upgrades, security patches, load testing, and incident response. An open-weight license must also be checked model by model, because “downloadable” does not establish permission for a particular commercial use. Older advice to compare only token or per-image prices is now especially weak. A quote omits retries, image resizing, prompt caching rules, minimum capacity, and human review. It can also hide a costly exit. Ask vendors for retention and deletion terms, region controls, model-version notification, rate limits, and an export path for logs and annotations. If personal data is involved, check the processor agreement against GDPR Article 28 rather than treating an API’s security page as a contract, because operational assurances and legal obligations are different artifacts. Do not infer that a standards badge settles model risk, either. ISO/IEC 42001:2023 describes an AI management system, and the NIST AI RMF 1.0 offers a risk-management framework; neither certifies that a specific response is correct for your application. Use those frameworks to organize ownership and evidence. Use your own test set to decide whether to ship. Pretrained APIs win the first test; self-hosting wins only under a proven constraint The explicit comparison is between a managed pretrained API and self-hosted open-weight inference. The managed API wins when demand is uncertain, model quality changes quickly, and a small team needs to test a product assumption: its cost is variable request spend, vendor dependence, and whatever data-handling restrictions the contract imposes. Self-hosting wins when sustained utilization, residency requirements, or a necessary model modification outweigh operations overhead: its cost is GPU capacity, engineering time, on-call responsibility, and often idle headroom. Neither option wins merely because its demo is faster. Set thresholds before running the comparison. For example, a p95 end-to-end latency target of 800 ms is a value to tune to the user interaction, not a universal benchmark. A 30-day shadow run is a proposed observation window, long enough to include at least some traffic variation but not a substitute for seasonal testing. A 95% precision floor might be an application’s chosen acceptance criterion when false positives trigger expensive review; another product may rationally favor recall. Make the assumptions explicit so a team cannot move the goalposts after seeing a preferred model’s results. Measure cost per accepted result, not cost per request. A cheap call that fails and is retried twice may cost more than one expensive successful call; a response requiring manual correction has another cost again. Record p50 and p95 latency, error rate, retry rate, abstention rate, and reviewer time by model version. OpenTelemetry can connect application traces to inference spans, but pin the semantic-convention version used by your instrumentation because GenAI attribute names have evolved. Export the underlying events as well as a dashboard so a provider switch does not erase your comparison. The following standard-library Python example runs without an SDK. Its 100 generated rows and dollar amounts are illustrative inputs, not measured performance; replace the rows with request logs to calculate a meaningful decision metric. python3 - &#60;&#60;'PY' import math rows = [(110 + (i * 37) % 390, 0.002 + (i % 4) * 0.001, i % 9 != 0) for i in range(100)] latencies = sorted(ms for ms, cost, accepted in rows) accepted = sum(ok for ms, cost, ok in rows) spend = sum(cost for ms, cost, ok in rows) p95 = latencies[math.ceil(0.95 * len(latencies)) - 1] print(f"p95_ms={p95}") print(f"accepted={accepted}/{len(rows)}") print(f"cost_per_accepted_usd={spend / accepted:.4f}") PY Run the same scoring code on both options’ logs, with reviewer decisions attached to identical requests. If a self-hosted model misses the acceptance floor, lower per-request spend is not a saving, because the missing quality must be paid for through retries, review, or lost users. Separate the evidence that changes a model decision from the evidence that changes a system decision Not every bad AI result calls for a different model. In a retrieval-backed feature, a correct model cannot cite a document that retrieval failed to supply. In a vision pipeline, a stronger classifier cannot recover detail discarded by an aggressive image resize. Before approving training or a hosting migration, require an error review that assigns each failure to input quality, retrieval or preprocessing, model behavior, policy, or application integration. The categories make spending accountable because each has a different plausible remedy. Your GenAI Feature Scales Until Retrieval and Latency Collide correctly warns that inference is not the whole response path; I would resist treating that collision as a reason to own model serving, because a slow search index remains slow after an API is replaced with a GPU. Instrument request ingress, retrieval, reranking, inference, and post-processing separately. For image requests, record decode and preprocessing time as well as inference. Compare p95 traces at the same concurrency, since a single-request benchmark conceals queueing. Keep model and system scorecards separate. A model scorecard can show precision, recall, calibration, and abstention on a versioned test set. A system scorecard should show end-to-end latency, cost per accepted result, review load, availability, and the proportion of requests excluded by data policy. If both improve after a change, trace the mechanism before attributing the gain to the model; otherwise, the next procurement decision may reward the wrong supplier. This separation also makes an exit plan credible. Store evaluation examples in a portable format, version transformations, and preserve enough request metadata to replay cases without retaining prohibited content. Test a second provider or a locally served model against the same gate before negotiating a long commitment. A fallback that has never passed that gate is a negotiating story, not an operational option. Run a shadow test before signing an annual contract Take one representative week of requests, remove data that cannot be used for testing, and have reviewers label a failure sample before comparing suppliers. Then run a managed API and a self-hosted candidate in shadow mode against the same acceptance rules. Ask finance for fully loaded costs and legal for the actual data terms. Sign for capacity only after the traces show which constraint ownership would remove.</p>
<p>The post <a href="https://deepfriedbytes.com/ctos-your-2024-genai-build-vs-buy-playbook-is-outdated/">CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>“Build or buy the AI” is the wrong question for a CTO in 2026. The useful boundary is narrower: buy model capability until your own evidence shows that owning inference or training improves the economics. Build the evaluation, data controls, and fallback paths yourself. Those assets remain valuable when an API price changes, an open-weight model improves, or a supplier’s terms stop fitting your product.</p>
<h2>The default has shifted from owning a model to owning the decision boundary</h2>
<p>Two years ago, advice to prototype with an API and then train a custom model often sounded like a natural progression. I would no longer treat training as the next stage. Better multimodal APIs have made a purchased baseline more useful, while serving software such as vLLM has made self-hosting a credible alternative without making training necessary. These changes separate three decisions that older build-versus-buy advice bundled together: whose weights to use, where inference runs, and who controls the product’s acceptance criteria.</p>
<p>My default is to buy access to a capable model, own the evaluation set, and keep the inference boundary replaceable. I would <strong>not</strong> train a proprietary vision model or fine-tune a language model merely because a pilot works, because pilot traffic rarely reveals enough about failure distribution or operating cost to justify an irreversible investment. Training becomes defensible when a labeled dataset represents production conditions, a purchased baseline misses a quantified requirement, and the expected gain exceeds annotation, retraining, and serving costs.</p>
<p>This is not an argument for letting a supplier define success. A CTO should own the ground-truth examples, review rules, data-retention requirements, and release gate. For generated answers, that gate may require an answer to cite an authorized source and refuse when evidence is absent. For images, it may require a specified precision at a tolerable false-rejection rate. Those are product decisions; a vendor’s aggregate benchmark cannot substitute for them because its test distribution and error costs differ from yours.</p>
<p><a href=/custom-vision-model-or-pretrained-api-which-fits-your-first-app/>Custom Vision Model or Pretrained API Which Fits Your First App</a> frames the first application as a model choice; I would make the first choice a reversible contract and test harness instead, because a promising first model says little about how cheaply the second one can be substituted. Put API calls behind an internal interface that records model identifier, prompt or preprocessing version, request cost, latency, and adjudicated outcome. That interface is worth building even if the supplier is never replaced.</p>
<h2>Recent platform changes have made old procurement shortcuts unreliable</h2>
<p>OpenAI’s Structured Outputs, introduced in 2024 with JSON Schema enforcement through <strong>strict: true</strong> on supported models, weakened the old claim that teams must fine-tune simply to get parseable fields. Schema conformity does not establish factual correctness, however, because a well-formed JSON object can still contain an invented value. Similarly, multimodal APIs can establish an image baseline quickly, but their availability does not prove that their terms permit every image to be sent off-site. The current procurement question is therefore less “Can an API do this?” and more “Which obligations survive a provider change?”</p>
<p>Self-hosting has also changed. vLLM’s OpenAI-compatible server and Hugging Face Transformers reduce the work needed to try open-weight models, while ONNX Runtime can simplify deployment of some vision pipelines. None removes the cost of operating them: somebody still owns GPU capacity, upgrades, security patches, load testing, and incident response. An open-weight license must also be checked model by model, because “downloadable” does not establish permission for a particular commercial use.</p>
<p>Older advice to compare only token or per-image prices is now especially weak. A quote omits retries, image resizing, prompt caching rules, minimum capacity, and human review. It can also hide a costly exit. Ask vendors for retention and deletion terms, region controls, model-version notification, rate limits, and an export path for logs and annotations. If personal data is involved, check the processor agreement against GDPR Article 28 rather than treating an API’s security page as a contract, because operational assurances and legal obligations are different artifacts.</p>
<p>Do not infer that a standards badge settles model risk, either. ISO/IEC 42001:2023 describes an AI management system, and the NIST AI RMF 1.0 offers a risk-management framework; neither certifies that a specific response is correct for your application. Use those frameworks to organize ownership and evidence. Use your own test set to decide whether to ship.</p>
<h2>Pretrained APIs win the first test; self-hosting wins only under a proven constraint</h2>
<p>The explicit comparison is between a <strong>managed pretrained API</strong> and <strong>self-hosted open-weight inference</strong>. The managed API wins when demand is uncertain, model quality changes quickly, and a small team needs to test a product assumption: its cost is variable request spend, vendor dependence, and whatever data-handling restrictions the contract imposes. Self-hosting wins when sustained utilization, residency requirements, or a necessary model modification outweigh operations overhead: its cost is GPU capacity, engineering time, on-call responsibility, and often idle headroom. Neither option wins merely because its demo is faster.</p>
<p>Set thresholds before running the comparison. For example, a <strong>p95 end-to-end latency target of 800 ms</strong> is a value to tune to the user interaction, not a universal benchmark. A <strong>30-day</strong> shadow run is a proposed observation window, long enough to include at least some traffic variation but not a substitute for seasonal testing. A <strong>95% precision floor</strong> might be an application’s chosen acceptance criterion when false positives trigger expensive review; another product may rationally favor recall. Make the assumptions explicit so a team cannot move the goalposts after seeing a preferred model’s results.</p>
<p>Measure cost per <em>accepted result</em>, not cost per request. A cheap call that fails and is retried twice may cost more than one expensive successful call; a response requiring manual correction has another cost again. Record p50 and p95 latency, error rate, retry rate, abstention rate, and reviewer time by model version. OpenTelemetry can connect application traces to inference spans, but pin the semantic-convention version used by your instrumentation because GenAI attribute names have evolved. Export the underlying events as well as a dashboard so a provider switch does not erase your comparison.</p>
<p>The following standard-library Python example runs without an SDK. Its 100 generated rows and dollar amounts are <em>illustrative inputs</em>, not measured performance; replace the rows with request logs to calculate a meaningful decision metric.</p>
<pre>python3 - &lt;&lt;'PY'
import math
rows = [(110 + (i * 37) % 390, 0.002 + (i % 4) * 0.001,
         i % 9 != 0) for i in range(100)]
latencies = sorted(ms for ms, cost, accepted in rows)
accepted = sum(ok for ms, cost, ok in rows)
spend = sum(cost for ms, cost, ok in rows)
p95 = latencies[math.ceil(0.95 * len(latencies)) - 1]
print(f"p95_ms={p95}")
print(f"accepted={accepted}/{len(rows)}")
print(f"cost_per_accepted_usd={spend / accepted:.4f}")
PY</pre>
<p>Run the same scoring code on both options’ logs, with reviewer decisions attached to identical requests. If a self-hosted model misses the acceptance floor, lower per-request spend is not a saving, because the missing quality must be paid for through retries, review, or lost users.</p>
<h2>Separate the evidence that changes a model decision from the evidence that changes a system decision</h2>
<p>Not every bad AI result calls for a different model. In a retrieval-backed feature, a correct model cannot cite a document that retrieval failed to supply. In a vision pipeline, a stronger classifier cannot recover detail discarded by an aggressive image resize. Before approving training or a hosting migration, require an error review that assigns each failure to input quality, retrieval or preprocessing, model behavior, policy, or application integration. The categories make spending accountable because each has a different plausible remedy.</p>
<p><a href=/your-genai-feature-scales-until-retrieval-and-latency-collide/>Your GenAI Feature Scales Until Retrieval and Latency Collide</a> correctly warns that inference is not the whole response path; I would resist treating that collision as a reason to own model serving, because a slow search index remains slow after an API is replaced with a GPU. Instrument request ingress, retrieval, reranking, inference, and post-processing separately. For image requests, record decode and preprocessing time as well as inference. Compare p95 traces at the same concurrency, since a single-request benchmark conceals queueing.</p>
<p>Keep model and system scorecards separate. A model scorecard can show precision, recall, calibration, and abstention on a versioned test set. A system scorecard should show end-to-end latency, cost per accepted result, review load, availability, and the proportion of requests excluded by data policy. If both improve after a change, trace the mechanism before attributing the gain to the model; otherwise, the next procurement decision may reward the wrong supplier.</p>
<p>This separation also makes an exit plan credible. Store evaluation examples in a portable format, version transformations, and preserve enough request metadata to replay cases without retaining prohibited content. Test a second provider or a locally served model against the same gate before negotiating a long commitment. A fallback that has never passed that gate is a negotiating story, not an operational option.</p>
<h2>Run a shadow test before signing an annual contract</h2>
<p>Take one representative week of requests, remove data that cannot be used for testing, and have reviewers label a failure sample before comparing suppliers. Then run a managed API and a self-hosted candidate in shadow mode against the same acceptance rules. Ask finance for fully loaded costs and legal for the actual data terms. Sign for capacity only after the traces show which constraint ownership would remove.</p>
<p>The post <a href="https://deepfriedbytes.com/ctos-your-2024-genai-build-vs-buy-playbook-is-outdated/">CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>Your 2024 Crypto Build-vs-Buy Playbook Is Obsolete</title>
		<link>https://deepfriedbytes.com/your-2024-crypto-build-vs-buy-playbook-is-obsolete/</link>
		
		
		<pubDate>Tue, 29 Sep 2026 07:36:09 +0000</pubDate>
				<category><![CDATA[Cryptocurrencies]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[Blockchain]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/your-2024-crypto-build-vs-buy-playbook-is-obsolete/</guid>

					<description><![CDATA[<p>A wallet decision that looked sensible in 2024 can be an expensive mistake in 2026. Ethereum accounts can now delegate execution logic, while custody vendors offer increasingly complete signing and policy services. I would buy the signing and key-custody layer for most teams, then build the transaction policy and recovery process in-house. The exception is a company whose wallet infrastructure is itself the product. Ethereum’s account changes weakened the case for building a signer first Before Ethereum’s Pectra upgrade, a common argument for an in-house wallet stack was that externally owned accounts could not support enough application-specific behavior. That argument is weaker after EIP-7702, activated on Ethereum mainnet on May 7, 2025, according to the Ethereum Foundation’s published schedule. An EOA can authorize a delegation to contract code, allowing developers to design more flexible execution without first moving every user to a new smart-account address. Delegation does not remove the authority of the EOA’s underlying key, however, so it is not a substitute for protecting that key. This changes a CTO’s procurement question. “Can we build account abstraction?” is too broad to guide a decision; “Who controls the signing key, the delegated implementation, and the policy that approves a transfer?” exposes the actual boundaries. ERC-4337 remains relevant for teams using UserOperations, bundlers, and paymasters, while EIP-7702 offers another route to account behavior. Neither protocol decides who can authorize a recovery, who pays when sponsorship fails, or how an incident commander stops withdrawals. Older advice to choose one wallet architecture up front and hide it behind a generic “send transaction” API is now risky because that API obscures whether a request is an EOA transaction, an ERC-4337 UserOperation, or a delegated-account call. EIP-5792’s wallet_sendCalls can help represent batched calls where a wallet supports it, but wallet support and execution semantics must be checked rather than assumed. Put the operation type, chain identifier, expected signer, authorization method, and sponsorship decision into your internal request model. CAIP-2 chain identifiers can keep that model unambiguous when the same address appears on several networks. Cryptocurrency APIs for Developers: Build Secure Wallets is a useful starting point for wallet design, but its title’s build-first implication should not settle the custody decision. A capable team can write a viem 2.x client in days; demonstrating that its signing service will withstand a compromised application credential, a mistaken policy change, and a regional outage is a different project. I would not build a production key-signing service merely to avoid a vendor bill, because that bill is usually easier to bound than the consequences of an untested recovery path. Buy custody when it is a dependency; build it when control is the product The explicit choice is between a managed custody and signing API and an in-house signer using infrastructure such as AWS KMS or an HSM. The managed option wins when a wallet supports another product: it buys an established operational surface for key creation, signing controls, and access management. Its costs are vendor fees, integration work, exit planning, and the risk that a provider’s outage or policy model constrains your releases. The in-house option wins when you need to own signing semantics across providers or can justify dedicated cryptography and operations staff. Its costs include key ceremonies, access reviews, secure deployment, monitoring, and incident exercises, not just code. A cost comparison should price the same trust boundary on both sides. AWS KMS supports the ECC_SECG_P256K1 key specification used for Ethereum-compatible signatures, according to AWS documentation, but that feature alone is not a complete wallet policy engine. Your team still has to control who may call Sign, bind a signature request to an approved transaction, and prevent an application server from turning a broad signing permission into an unlimited withdrawal permission. Conversely, a managed provider is not automatically safe because it advertises multiparty computation: ask which actors can change policy, initiate recovery, or approve a destination, and test those answers in a sandbox. Use a comparable workload before comparing quotes. For example, model 50,000 active wallets and 10,000 signing requests per day as planning assumptions to replace with your forecast, not as claims about a typical deployment. Request a quote for that workload, including recovery events, webhook delivery, additional environments, and support. Alongside it, estimate engineering time for an in-house service, security review, on-call coverage, and an annual recovery exercise. The cheaper API price is not decisive if exporting accounts or recreating policies during an exit requires work your team has not funded. There is also a middle path worth stating precisely: buy key custody but own the approval policy outside the provider. A policy service can require a destination allowlist, a transaction simulation result, and a second approver before it requests a signature. Keep those decisions in an auditable store you control, then restrict the vendor credential so it cannot bypass them through an alternate endpoint. This arrangement wins when portability matters but key operations are not your differentiator; it costs an additional service and a careful review of every vendor permission. Do not describe it as “vendor-neutral” until you have run an exit test with another signer. A successful API call no longer proves that the intended account will execute it Integration advice used to concentrate on authenticating an API request, receiving a transaction hash, and watching for confirmation. That sequence misses an important question for delegated accounts: what code will the account use when the transaction executes? EIP-7702 delegation can change through an on-chain authorization, so an address classified once at onboarding cannot be treated as permanently unchanged. A transaction preview should record the chain, account state, target code, calldata, and block context; the approval service should recheck the facts it relies on immediately before submission. The following viem 2.x script checks whether an Ethereum mainnet address currently exposes an EIP-7702 delegation indicator. Save it as check.mjs, run npm install viem@2, and set ETH_RPC_URL and WALLET_ADDRESS before running node check.mjs: import { createPublicClient, http, isAddress } from 'viem'; import { mainnet } from 'viem/chains'; const address = process.env.WALLET_ADDRESS; if (!address &#124;&#124; !isAddress(address)) throw new Error('Set WALLET_ADDRESS'); const client = createPublicClient({ chain: mainnet, transport: http(process.env.ETH_RPC_URL) }); const code = await client.getCode({ address }); const delegated = /^0xef0100[0-9a-f]{40}$/i.test(code ?? ''); console.log({ address, delegated, implementation: delegated ? `0x${code.slice(8)}` : null }); This is an inspection aid, not an approval check: it reads current code but does not verify the implementation’s behavior, prove who authorized it, or guarantee the state at a later block. Where approval depends on a particular implementation, pin its address and reviewed code hash, simulate the proposed call against recent state, and fail closed if the delegation changes. Use a separate review for upgrades to that implementation. A passkey login under WebAuthn or a successful SIWE message under EIP-4361 proves something about authentication; neither, by itself, proves that the resulting transfer matches the user’s intent. I would use Cryptocurrency APIs for Developers Secure Wallet Integration for integration vocabulary, not as evidence that an SDK settles this trust boundary. ERC-1271 contract signatures, WalletConnect v2 sessions, provider permissions under EIP-1193, and delegated-account execution all give an application different ways to receive an apparently valid approval. The policy service must specify which of those approvals is acceptable for each operation, because treating every “signature verified” result as equivalent makes recovery and high-value transfers harder to defend. The procurement test should be an exit and incident exercise Over the last two years, the operational question has also become harder to postpone. In the EU, MiCA’s crypto-asset service provider provisions began applying on December 30, 2024; whether they apply to a particular wallet business depends on its services and jurisdiction. DORA has applied since January 17, 2025, to covered financial entities and affects their management of ICT third-party risk. Neither rule means every wallet team needs the same licence or contract. Both make “we will document our provider dependencies later” poor advice for a covered operation, because the dependency determines who can act during a failure. Before signing a custody contract, run a timed exercise rather than accepting a diagram. Give the vendor and your team a scenario in which a policy administrator is compromised while one signing region is unavailable. Set a test target of 30 minutes to disable new withdrawals; that is an internal target to tune against your risk, not a vendor performance claim. Record who has the authority to pause signing, how that authority is authenticated, and whether the action also blocks an attacker holding an existing API credential. Then restore service without quietly discarding the audit trail. Run a second exercise in which the provider relationship ends. Export what is actually exportable, identify accounts that cannot migrate without user action, and reproduce transaction policy with a replacement signer. Safe contracts, OpenZeppelin Contracts 5.x components, and standard EVM transaction formats may improve portability, but none guarantees that a vendor’s recovery roles or off-chain approval history will transfer. Ask for the data format and procedure in the contract, because an undocumented promise of “easy migration” cannot be tested during procurement. Finally, make the decision reversible where you can. Keep your application’s ledger, transaction intents, and policy decisions outside the signing provider, with identifiers that remain meaningful after a migration. Require receipts to link an approved intent to the exact signed payload and resulting on-chain transaction. This separation costs additional engineering now, but it lets the company change custody arrangements without rewriting the business rules that determine whether money may move. Start with the failure you would otherwise delegate Ask each shortlisted provider and your in-house team to demonstrate the same compromised-admin pause and account-exit exercise next week. Write down the authority, elapsed time, missing data, and manual steps for each attempt. Choose the option that leaves your team able to stop and explain a bad transfer—not the one whose SDK produces the first successful transaction hash.</p>
<p>The post <a href="https://deepfriedbytes.com/your-2024-crypto-build-vs-buy-playbook-is-obsolete/">Your 2024 Crypto Build-vs-Buy Playbook Is Obsolete</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A wallet decision that looked sensible in 2024 can be an expensive mistake in 2026. Ethereum accounts can now delegate execution logic, while custody vendors offer increasingly complete signing and policy services. I would buy the signing and key-custody layer for most teams, then build the transaction policy and recovery process in-house. The exception is a company whose wallet infrastructure is itself the product.</p>
<h2>Ethereum’s account changes weakened the case for building a signer first</h2>
<p>Before Ethereum’s Pectra upgrade, a common argument for an in-house wallet stack was that externally owned accounts could not support enough application-specific behavior. That argument is weaker after EIP-7702, activated on Ethereum mainnet on May 7, 2025, according to the Ethereum Foundation’s published schedule. An EOA can authorize a delegation to contract code, allowing developers to design more flexible execution without first moving every user to a new smart-account address. Delegation does not remove the authority of the EOA’s underlying key, however, so it is not a substitute for protecting that key.</p>
<p>This changes a CTO’s procurement question. “Can we build account abstraction?” is too broad to guide a decision; “Who controls the signing key, the delegated implementation, and the policy that approves a transfer?” exposes the actual boundaries. ERC-4337 remains relevant for teams using UserOperations, bundlers, and paymasters, while EIP-7702 offers another route to account behavior. Neither protocol decides who can authorize a recovery, who pays when sponsorship fails, or how an incident commander stops withdrawals.</p>
<p>Older advice to choose one wallet architecture up front and hide it behind a generic “send transaction” API is now risky because that API obscures whether a request is an EOA transaction, an ERC-4337 UserOperation, or a delegated-account call. EIP-5792’s <em>wallet_sendCalls</em> can help represent batched calls where a wallet supports it, but wallet support and execution semantics must be checked rather than assumed. Put the operation type, chain identifier, expected signer, authorization method, and sponsorship decision into your internal request model. CAIP-2 chain identifiers can keep that model unambiguous when the same address appears on several networks.</p>
<p><a href=/cryptocurrency-apis-for-developers-build-secure-wallets/>Cryptocurrency APIs for Developers: Build Secure Wallets</a> is a useful starting point for wallet design, but its title’s build-first implication should not settle the custody decision. A capable team can write a viem 2.x client in days; demonstrating that its signing service will withstand a compromised application credential, a mistaken policy change, and a regional outage is a different project. I would not build a production key-signing service merely to avoid a vendor bill, because that bill is usually easier to bound than the consequences of an untested recovery path.</p>
<h2>Buy custody when it is a dependency; build it when control is the product</h2>
<p>The explicit choice is between a <strong>managed custody and signing API</strong> and an <strong>in-house signer using infrastructure such as AWS KMS or an HSM</strong>. The managed option wins when a wallet supports another product: it buys an established operational surface for key creation, signing controls, and access management. Its costs are vendor fees, integration work, exit planning, and the risk that a provider’s outage or policy model constrains your releases. The in-house option wins when you need to own signing semantics across providers or can justify dedicated cryptography and operations staff. Its costs include key ceremonies, access reviews, secure deployment, monitoring, and incident exercises, not just code.</p>
<p>A cost comparison should price the same trust boundary on both sides. AWS KMS supports the <em>ECC_SECG_P256K1</em> key specification used for Ethereum-compatible signatures, according to AWS documentation, but that feature alone is not a complete wallet policy engine. Your team still has to control who may call <em>Sign</em>, bind a signature request to an approved transaction, and prevent an application server from turning a broad signing permission into an unlimited withdrawal permission. Conversely, a managed provider is not automatically safe because it advertises multiparty computation: ask which actors can change policy, initiate recovery, or approve a destination, and test those answers in a sandbox.</p>
<p>Use a comparable workload before comparing quotes. For example, model 50,000 active wallets and 10,000 signing requests per day as <em>planning assumptions to replace with your forecast</em>, not as claims about a typical deployment. Request a quote for that workload, including recovery events, webhook delivery, additional environments, and support. Alongside it, estimate engineering time for an in-house service, security review, on-call coverage, and an annual recovery exercise. The cheaper API price is not decisive if exporting accounts or recreating policies during an exit requires work your team has not funded.</p>
<p>There is also a middle path worth stating precisely: buy key custody but own the approval policy outside the provider. A policy service can require a destination allowlist, a transaction simulation result, and a second approver before it requests a signature. Keep those decisions in an auditable store you control, then restrict the vendor credential so it cannot bypass them through an alternate endpoint. This arrangement wins when portability matters but key operations are not your differentiator; it costs an additional service and a careful review of every vendor permission. Do not describe it as “vendor-neutral” until you have run an exit test with another signer.</p>
<h2>A successful API call no longer proves that the intended account will execute it</h2>
<p>Integration advice used to concentrate on authenticating an API request, receiving a transaction hash, and watching for confirmation. That sequence misses an important question for delegated accounts: what code will the account use when the transaction executes? EIP-7702 delegation can change through an on-chain authorization, so an address classified once at onboarding cannot be treated as permanently unchanged. A transaction preview should record the chain, account state, target code, calldata, and block context; the approval service should recheck the facts it relies on immediately before submission.</p>
<p>The following viem 2.x script checks whether an Ethereum mainnet address currently exposes an EIP-7702 delegation indicator. Save it as <em>check.mjs</em>, run <em>npm install viem@2</em>, and set <em>ETH_RPC_URL</em> and <em>WALLET_ADDRESS</em> before running <em>node check.mjs</em>:</p>
<pre>import { createPublicClient, http, isAddress } from 'viem';
import { mainnet } from 'viem/chains';
const address = process.env.WALLET_ADDRESS;
if (!address || !isAddress(address)) throw new Error('Set WALLET_ADDRESS');
const client = createPublicClient({
  chain: mainnet, transport: http(process.env.ETH_RPC_URL)
});
const code = await client.getCode({ address });
const delegated = /^0xef0100[0-9a-f]{40}$/i.test(code ?? '');
console.log({ address, delegated,
  implementation: delegated ? `0x${code.slice(8)}` : null });</pre>
<p>This is an inspection aid, not an approval check: it reads current code but does not verify the implementation’s behavior, prove who authorized it, or guarantee the state at a later block. Where approval depends on a particular implementation, pin its address and reviewed code hash, simulate the proposed call against recent state, and fail closed if the delegation changes. Use a separate review for upgrades to that implementation. A passkey login under WebAuthn or a successful SIWE message under EIP-4361 proves something about authentication; neither, by itself, proves that the resulting transfer matches the user’s intent.</p>
<p>I would use <a href=/cryptocurrency-apis-for-developers-secure-wallet-integration/>Cryptocurrency APIs for Developers Secure Wallet Integration</a> for integration vocabulary, not as evidence that an SDK settles this trust boundary. ERC-1271 contract signatures, WalletConnect v2 sessions, provider permissions under EIP-1193, and delegated-account execution all give an application different ways to receive an apparently valid approval. The policy service must specify which of those approvals is acceptable for each operation, because treating every “signature verified” result as equivalent makes recovery and high-value transfers harder to defend.</p>
<h2>The procurement test should be an exit and incident exercise</h2>
<p>Over the last two years, the operational question has also become harder to postpone. In the EU, MiCA’s crypto-asset service provider provisions began applying on December 30, 2024; whether they apply to a particular wallet business depends on its services and jurisdiction. DORA has applied since January 17, 2025, to covered financial entities and affects their management of ICT third-party risk. Neither rule means every wallet team needs the same licence or contract. Both make “we will document our provider dependencies later” poor advice for a covered operation, because the dependency determines who can act during a failure.</p>
<p>Before signing a custody contract, run a timed exercise rather than accepting a diagram. Give the vendor and your team a scenario in which a policy administrator is compromised while one signing region is unavailable. Set a <em>test target</em> of 30 minutes to disable new withdrawals; that is an internal target to tune against your risk, not a vendor performance claim. Record who has the authority to pause signing, how that authority is authenticated, and whether the action also blocks an attacker holding an existing API credential. Then restore service without quietly discarding the audit trail.</p>
<p>Run a second exercise in which the provider relationship ends. Export what is actually exportable, identify accounts that cannot migrate without user action, and reproduce transaction policy with a replacement signer. Safe contracts, OpenZeppelin Contracts 5.x components, and standard EVM transaction formats may improve portability, but none guarantees that a vendor’s recovery roles or off-chain approval history will transfer. Ask for the data format and procedure in the contract, because an undocumented promise of “easy migration” cannot be tested during procurement.</p>
<p>Finally, make the decision reversible where you can. Keep your application’s ledger, transaction intents, and policy decisions outside the signing provider, with identifiers that remain meaningful after a migration. Require receipts to link an approved intent to the exact signed payload and resulting on-chain transaction. This separation costs additional engineering now, but it lets the company change custody arrangements without rewriting the business rules that determine whether money may move.</p>
<h2>Start with the failure you would otherwise delegate</h2>
<p>Ask each shortlisted provider and your in-house team to demonstrate the same compromised-admin pause and account-exit exercise next week. Write down the authority, elapsed time, missing data, and manual steps for each attempt. Choose the option that leaves your team able to stop and explain a bad transfer—not the one whose SDK produces the first successful transaction hash.</p>
<p>The post <a href="https://deepfriedbytes.com/your-2024-crypto-build-vs-buy-playbook-is-obsolete/">Your 2024 Crypto Build-vs-Buy Playbook Is Obsolete</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
	</channel>
</rss>