--- title: "Agentic Modelling - What Does It Cost to Run An Agent?" url: "https://lumosbusiness.com/notebook/agentic-modelling-what-does-it-cost-to-run-an-agent/" date_published: "2026-09-01" date_modified: "2026-09-21" format: "essay" author: "Joel Finningley" thesis: "Agents are everywhere on the web, but they are not typically 'free' to run. So how do you model their expected cost and performance against certain workflows? This post explores a real agentic model using the CPCT." --- # Agentic Modelling - What Does It Cost to Run An Agent? **Thesis.** Agents are everywhere on the web, but they are not typically 'free' to run. So how do you model their expected cost and performance against certain workflows? This post explores a real agentic model using the CPCT. --- Love or hate AI, it's almost impossible to fade the potential of agents. If you haven't experimented with building an agent, you know that they can do a lot of boring, repetitive work in a short period of time. An 'agent' is not a chatbot or customer service agent. An agent is defined as a "model + harness:" - Model: one of Claude's models (Fable, Opus, etc.), OpenAI, Google, xAI's current frontier models, or open-source models. The model provides the intelligence for the tasks - Harness: A harness is everything around the model - the loop, the tools, the retries, the routing - that turns raw intelligence into a finished task; Claude Code, Codex, and Cursor are harnesses The Problem: when building an agent for the purpose of a business task, you rely on API credits in most cases, which is pay-per-tokens. Since agents run on loops until tasks are completed in non-linear ways at some points, the question is how does one 'model' a workflow or business that runs as an agent? The Challenge: most people building and experimenting with agents (myself included) will just jump into it. The goal in the early stages is typically a) see if it works b) minimize API costs. But that only works to a certain threshold because some workflows will work on the top models, which are expensive to run but get the job done. The genesis of this blog - as explained in [Rebuilding My Website for the Agentic Web](/notebook/rebuilding-my-website-for-the-agentic-web/) - is business modelling. Typically, it was analyzing a business model holistically and creating a business model canvas (see [Social Commerce Business Model](/analyses/social-commerce-business-model-canvas/) for an example). The 'agentic business model,' however, requires a deep understanding of token economics to model. And thus the genesis of the CPCT. ## The CPCT (Cost Per Completed Task) Modelling Framework Thus, Claude Code and I decided to spin up [CPCT](/offerings/) to be able to model a specific workflow agentically. First and foremost, for clarification, this type of modelling is not meant for the indie-spirited [Hermes](https://hermes-agent.nousresearch.com/) or [OpenClaw](https://openclaw.ai/) type development (which is really cool btw!), where the goal is to create a 'personal agent' that can operate on virtually any model, closed or open-source. This type of modelling is meant for the business case, enterprise-type production model where the following assumptions are made: - API Credits will be spent in large amounts - if the goal is to build an agent that can take on a mission-critical task or workflow, the assumption is that a relatively significant token spend will occur - Labor Hours will be replaced - if we are talking the typical replacement of some monotonous or repetitive task, then labor hours will be replaced in the process The delta between the two - the cost of the tokens vs. labor hours replaced - is the marginal gain for the business; and an understanding of those unit economics is of utmost importance. Thus the CPCT Framework zones in on understanding a task or workflow, from end to end, across five key layers: - Define the task unit in business terms - Trace the agent's loop anatomy (based on the task - context, retries, verification, etc.) - Estimate the end-to-end unit of work as a record, then measure; we are looking for distribution not averages to take account for the unit economics of edge cases - Run the model and trace each step of the agentic process across multiple turns (25 is the baseline) - Assess and analyze the measurement against P&L variables (including labor hours) and use it to determine pricing (or net value if internal) ## The CPCT Demo Run A tech stack was spun up using Claude Code, with the main component being [LangFuse](https://langfuse.com/), along with a database for analysis and the harness's own ledger. What is LangFuse? Langfuse is an open-source Agent Evals and Observability platform. It allows me to trace the actual costs and performance of the agent through the demo run. LangFuse taps into the world of 'telemetry' which is used for measuring evals, as we show below. I needed a relatively straightforward use case, so I chose 'professional services' and document analysis as the baseline workflow: - Documents are classified for analysis - Key fields are extracted - A summary and acknowledgement email is drafted Claude's API was used for the models - Haiku, Sonnet, Fable (Opus fallback) - for the different tasks: - Haiku for the initial run (classifier agent) - Sonnet (drafter agent) - Fable (judge agent), with Opus as the fallback Five different configurations run - one naive baseline, the lever configurations layering routing, caching, compaction, and batching, plus a fallback configuration - at 25 tasks each. The cost of each run was compared against the overall performance. ## What Happened When We Ran It? | Config | What changed | Raw cost/task (invoiced p50) | Loaded cost/task (p50, adds verification + modeled review) | Completion | Quality (rubric pass) | | --- | --- | --- | --- | --- | --- | | B0 | Baseline — Fable runs every step; no routing, caching, or batching | $0.407 | $0.558 | 72% | 72.2%† | | L1 | Routing — each step to the cheapest capable model (Haiku classify, Sonnet draft, Fable judge) | $0.052 | $0.089 | 100% | 76.0% | | L2 | + Prompt caching (a judge-only cacheable surface) | $0.052 | $0.089 | 100% | 80.0% | | L3 | + Compaction + batch — the full engineered stack | $0.027 | $0.075 | 100% | 72.0% | | B0+FB | Baseline + Opus 4.8 fallback on Fable refusals | $0.387 | $0.429 | 100% | 92.0% | *Raw figures are invoiced and reconciled to the token; loaded figures add verification cost and a modeled review-labor allowance. F2 sealed 2026-09-01, reconciliation PASS (−0.52%), token-exact (cpct-lab `9db4e73`). Both cost columns are p50; completion = 100% − task-failure rate; quality = frozen-grader rubric pass. †B0's pass rate is over its 18 non-refused survivors; its 7 refused tasks were never graded.* - B0: the opening Baseline - the frontier model does everything (Fable), every classification, every draft, every judgment, no caching, batching or routing. The baseline is built on 'the best model does everything' to establish a baseline cost judgment - L1: Lever 1 Routing - same workflow, but each step goes to the cheapest model that can do it; Haiku classifies documents, Sonnet drafts summaries and emails, Fable judges a sample of the output - L2: Levers 1+2 Routing + Prompt Caching - L1 plus Claude API's prompt caching feature turned on. - L3: Levers 1+2+3+4, the full engineered stack - routing + caching + context compaction (summarizing key elements, removing redundant bits) + batch processing (a 50% API discount but the run took ~5X longer) - B0+FB: the Baseline + Fallback - the baseline's exact configuration (all Fable, most expensive) with one change; where Fable refuses a request, the request is sent to a different frontier model (Opus 4.8). This is a completion lever. $27 in API tokens was spent across the five configurations, spread out over two lab days, which were eleven days apart. ![Langfuse tracing view for the CPCT Lab project: a table of grader (judge) traces from the L3 run — each row a grading call with start time, model type, trace name (grader:lab-###), input classification records and a JSON output at rubric_version 1.0 — above a bar chart of trace counts across Aug 25–31, with about 217 traces in total.](/assets/2026/09/CPCT_langfuse_L3_trace.webp) ## The Results - The Cost Problem was a Completion Problem - B0 (Fable only) failed 28% of tasks, not because it was incapable but because it refused 36% of calls on identifier-heavy documents. Routing to cheaper models made the workflow 8X cheaper and better because they (Haiku and Sonnet) didn't refuse the task. - The Number Changes Over Time - eleven days after running the Baseline (B0), the same model on the same document refused 14% less.\* The 28% failure rate from the pilot isn't reproducible today at any configuration. A pilot modelled only once is modelling a snapshot; the CPCT basket becomes a standing index, re-run on a fixed cadence, quarterly to start - Verification is the Next Lever - Once routing is done, the frontier "judge" (Fable, Opus fallback) that checks the output is 40 - 54% of spend. Across 20 sampled tasks, a grader (Haiku) that was 14X cheaper agreed on every pass/fail verdict, with a consistent small offset. The finding here is that the cheap model can replace the frontier model judge. \*Fable's first attempt refusals dropped 35.5% to 21.5%, and Opus recovered all 23 remaining refusals. ## Frontier Judgment, Costed Out There are many possible takeaways from this exercise, and this is only the demo run, but perhaps the most interesting anecdote is the frontier costing: $0.09 per task at ~78% pass rate or $0.43 at 92% pass rate. That extra clarity at the frontier on judgment comes with a significantly higher cost. This is then up to the business to decide. This is an example of the type of insights gleaned from the CPCT rather than just playing around and estimating. Where this would really matter in a hypothetical professional services workflow buildout: - What documents are being judged? If the task here is very high-value to the business, then it's a no-brainer to capture the extra ~14% judgment on the frontier model - How is it being priced or billed internally? That added frontier judgment comes at an extra cost; that cost can be passed on to the customer and happily absorbed if it produces markedly better outputs - What are the costs of the errors? In this case, as a demo run, I was simply looking to understand the measurement process. There is no 'cost' for errors here measured out in business terms, but if there was, the price of lower frontier judgment would have a cost to the business in the event those manifested in errors - What are the labor hours for human review? In reference to the above point, a human needs to verify the work that is done at the end of each run. If lower frontier judgment results in the need for more manual task verification, that's a cost in labor that needs to be assessed ## CPCT Gaps and Next Steps This whole process was for prototyping and understanding, it was far from perfect. We started with an estimate on cost before spending a dollar (in Claude API credits), the actual reconciled amount came back 28% more expensive than expected, but with multiple caveats as to how and why. There are many dimensions that would change the efficacy of the outputs: a different task shape, different refusal rates from the frontier models, the model provider (in this case Claude) changing pricing for refused requests. It also brings up the models themselves. This experiment was run on Claude because I am familiar with Claude across multiple levers, including the API, and have experience with building an agent in a prototyping environment. Putting different models in would alter the whole set of expected outputs completely. There are absolutely cheaper options to run this via open-source alternatives or via [OpenCode](https://opencode.ai/)/[OpenRouter](https://openrouter.ai/), but the specific reasons for using Claude here were pointed out in the opening context. The TLDR of the post - LangFuse and telemetry enables 'agentic modelling.' The CPCT framework was designed to be able to test, optimize, and cost out a given workflow with real numbers. This makes the process of 'agentic modelling' possible for any business, creating an understanding of a) what the different models cost b) what those costs look like with a baseline pilot c) what those costs look like with agent optimization over time. The 14% decline in the second baseline (B0+FB) configuration vs. the baseline (B0) pilot shows how the modelling approach works. After forming an initial view and model on the pilot, running five separate configurations (B0, L1, L2, L3, B0+FB), and retesting the baseline with a fallback, a clearer picture of the agentic workflow emerged. Pricing agentic workflows means pricing completion risk, as agents run in loops; modelling provides a baseline expectation for costs and expands the canvas by testing multiple configurations on the same data set. The CPCT experiment will continue! ## Revisions - 2026-09-21 — The illustrative accumulation figure is now measured — see [Claude Commerce Agents, Measured (CPCT)](/notebook/claude-commerce-agents-measured-cpct/). - 2026-09-07 — Added an illustrative figure on context accumulation across turns. --- Written 100% by hand by [Joel](/about/) — no AI in the prose. Research and contextual support from Claude; shipped on Claude Code.