--- title: "Claude Commerce Agents, Measured (CPCT)" url: "https://lumosbusiness.com/notebook/claude-commerce-agents-measured-cpct/" date_published: "2026-09-21" date_modified: "2026-09-26" format: "essay" author: "Joel Finningley" thesis: "Claude Commerce Agents were recently launched with a blueprint and claims of larger cart sizes and better conversion. Digging into the blueprint and measuring the results via CPCT reveals unique insights into the agent's performance and potential cost to rollout. The expensive part was never the model." --- # Claude Commerce Agents, Measured (CPCT) **Thesis.** Claude Commerce Agents were recently launched with a blueprint and claims of larger cart sizes and better conversion. Digging into the blueprint and measuring the results via CPCT reveals unique insights into the agent's performance and potential cost to rollout. The expensive part was never the model. --- Recently I deep dived the [App Layer Thesis](/notebook/the-ai-app-layer-thesis-who-captures-the-value/) as it relates to AI. The broad thesis is that the companies who are most likely to capture the value in the industry are those who hold a deep level of domain expertise and vertical integration. The labs themselves business model is to build the models that power the application and capture revenue by renting out 'inference' to those same companies; however, the labs have proven capable of developing their own apps and taking advantage of their tremendous horizontal distribution power. What is playing out is a barbell, where the sophisticated players on both ends of the spectrum are capturing the value, while those in the middle are losing out. In the case of Anthropic and the recent launch of [Claude Commerce Agents](https://claude.com/blog/claude-for-commerce-agents), Anthropic is giving the app layer away for free without touching the wallet (or cart). This launch tests the boundaries of the thesis in the commerce space - eCommerce companies now have plausible options to run agents on their stores from platforms and frontier labs. This turned out to be the perfect opportunity to run the [CPCT (Cost Per Completed Task)](/notebook/agentic-modelling-what-does-it-cost-to-run-an-agent/) and see how the claims stack up vs. the measurement. ## What Anthropic Published vs. What Was Measured When Anthropic launched Claude Commerce Agents a few weeks ago, [Reuters outlined](https://www.reuters.com/business/retail-consumer/anthropic-launches-ai-agent-blueprints-retailers-ahead-holiday-shopping-season-2026-09-02/) the upside claims of the agents in production on a commerce storefront: one unnamed partner saw carts 30 - 35% larger, while customers were about 60% more likely to complete a purchase. Priceline's Penny and Shopify's Sidekick were cited as live (these predate the launch), but [Shopify's implementation](https://github.com/Shopify/claude-for-commerce-examples) is sample code and there is no current example of a non-partner brand shipping and implementing the code. Three weeks in, this is the first number on what it costs to run. Anthropic offers the blueprint for building either a shopping agent (customer facing) or a merchant agent (merchant facing) across retail, travel, telecom, and entertainment. What they don't offer are any numbers on how much it costs to run the agent. The blueprint itself is free, but to run an agent on a store requires API Credits, which is a cost per task. So I ran the CPCT Framework using 25 simulated shopping sessions and 26 scripted turns on the blueprint's retail vertical (87-product demo catalog) instrumented on the baseline client configuration without any changes to the underlying code (all this was done on Claude Code btw). The baseline config as Anthropic ships it is the Sonnet 5 model turns, Haiku for memory extraction, three cache breakpoints, and a few other specific levers. The CPCT was run through five configurations; a baseline (B0) where everything runs out-of-the-box, a B0' replicate where the same configuration is run a second time with nothing changed (ie. the noise floor), lever 1 (L1) with the cache off, lever 2 (L2) Haiku, and lever 3 (L3) with the iteration cap off. The data showed a negligible cost of running an agent; roughly a cent and a half per session means 100,000 sessions per month costs about $1,500. This is the cost of running the agent purely from an API Credits perspective, it doesn't include labor or other costs related to implementation on the store itself. This becomes valuable as a baseline understanding of the variables because this data can be mapped against other known variables on the cost side to start building an expected P&L. Additional data that needs to be extrapolated that brings this together includes: - Does it materially change conversion? (assuming they company selects the shopper agent) - Is the cart right? (in the CPCT run, the cart completed accurately in 23 out of 25 turns/carts) - Who catches any fabrications? (nothing in the out-of-the-box configuration does, CPCT added a frontier judge on a 20% sample) All combined, we start to get a more comprehensive picture of the business value an agent brings to the table. ## Claude Commerce Agents - Deeper Insights from CPCT | Config | What changed | Production cost (25 sessions) | vs. B0 | Cost per completed session (p50) | Correct carts | Rounds/turn | | --- | --- | --- | --- | --- | --- | --- | | B0 | Nothing: the blueprint as shipped | $0.4206 | +0.9% | $0.0154 | 23/25 | 2.50 | | B0' | Nothing: the same config run again (the noise floor) | $0.4127 | −0.9% | $0.0148 | 23/25 | 2.42 | | L1 | Cache off | $1.5131 | +263.2% | $0.0469 | 23/25 | 2.46 | | L2 | Haiku, thinking off | $0.1918 | −54.0% | $0.0074 | 23/25 | 2.42 | | L3 | Iteration cap off | $0.4309 | +3.4% | $0.0157 | 23/25 | 2.62 | *Invoiced and reconciled to the token (+0.04%, one attributed orphan call). Demo catalog, 25 sessions, single trial with a measured noise floor. "vs. B0" is against the mean of the two baseline runs ($0.4166).* The data above is the results from running CPCT on Claude's Commerce Agents blueprint for a shopping agent using hypothetical data from a retail store. There are three main insights that are worth pointing out, specific to agents: 1. Caching is worth 72.5% of production cost and costs nothing in quality: every turn re-sends the agent's prefix - system prompt, tools, skills, etc. - and costs about 9,500 tokens. With caching on, the prefix is written once and read from the cache at a tenth of the cost on every subsequent run. With caching off, the bill grew by 263% - same prompts, same answers, same 23 of 25 carts, and only the bill changed. It's worth noting that for caching to work, the prompt size must be above a minimum threshold, thus the rule is not universal and this came up on an [earlier intake workflow](https://lumosbusiness.com/notebook/agentic-modelling-what-does-it-cost-to-run-an-agent/). 2. Haiku halves the cost with no quality loss and no extra rounds: Anthropic's blueprint starts with Sonnet, but Haiku successfully executed the turns as reliably as Sonnet in this context (2.42 vs 2.50 tool rounds per turn); there was no inflation as was the worry. Haiku can't run adaptive thinking, so this compares Sonnet-with-thinking against Haiku-as-deployable (thinking off), which is the choice the merchant would actually face. 3. Verification is the sticky layer, where the costs and doubt both exist in tandem: adding a judger in the form of a frontier model (Fable in this case) along with a grader (Haiku) costs more per session than the L2 shopping loop. Fable was judged on a 20% sample, while a Haiku grader scored all 25. The cost per session was $0.011 vs. the L2 shopping loop at $0.0074, and while the grader tracked the judge on dimensions (87% on a point) but binary pass/fail flipped in 7 of 25 sessions between two identical baseline runs - for both verifiers. Grade on dimensions, not verdict. In one case, the memory extractor fabricated a customer preference; the judge caught it, the grader excused it. Thus, cost is not the only factor when judging a run, experimentation is required to catch the edge cases and evaluate the accuracy in aggregate. The total cost of the matrix was $4.17. Every call was traced in [LangFuse](https://langfuse.com/) and ledgered and reconciled against the final invoice to ensure accuracy (+0.04% delta). ## Would I run a Claude Commerce Agent on my store? (if I had one) I have worked in eCommerce on several occasions, so the hypothetical here is not that far-fetched to think through for thought experiment purposes … Yes! (with some qualifying criteria) - I would run it on a structured catalog - the agent's ability is determined by the data quality it ingests; unstructured data creates more complexity and that complexity is where the problems emerged in the CPCT runs - Run it beside search and browse, not inside of them - users are habituated to using 'search' and 'browse' on Shopify stores, the value add of an agent inside of them is limited; running the agent as a function on the storefront beside search and browse feels like the optimal behavior, not as a standalone function instead of them - Ensure the number of expected sessions is enough to amortize the integration cost - the actual integration on a storefront is going to cost multiple X what the tokens costs; therefore, I would need to be sure that cost can amortize against the expected increases in conversion and/or AOV The main necessities that would need to be in place to practically deliver on this are: - Evals authored against my catalog (the plugin automatically scaffolds them) - A human review of a small sample of what the memory layer writes about customers - Measurement from Day 1 on: cost per completed cart, cart correctness, and conversion (against a control) With these benchmarks and backstops in place, the Claude Commerce Agent implementation could be worth running for a mid-size Shopify store as an example. ## The Competition And The AI App Layer Thesis This run of the CPCT framework on Claude Commerce Agents perfectly illustrates the practical nature of the whole ['app layer thesis'](/notebook/the-ai-app-layer-thesis-who-captures-the-value/). Shopify has launched Sidekick for the merchant that is ready to deploy out-of-the-box. Shopify's Sidekick is built on Claude Sonnet and they also published their own implementation of Anthropic's blueprint, so Shopify is Anthropic's customer, partner, and competitor all in one. Shopify also has customer-facing agents (equivalent to shopping agents) in the Shop App, along with agentic storefronts, and a no-code storefront agent via [Shopify Inbox](https://help.shopify.com/en/manual/inbox), so they capture app layer market share from end-to-end. Anthropic comes in and launches Claude Commerce Agents, whereby they give away a free developer blueprint and only charge for the API tokens. Shopify represents the app layer of the stack, Anthropic represents the models and frontier labs. Shopify's own Sidekick app naturally uses Anthropic's inference to provide the service, but they earn back on subscription revenue by providing a better service to their merchants; Anthropic is essentially leasing their inference for a new commerce use case and gaining customers in the commerce vertical via the API token model. ### Which layer is better for the average storefront - merchant owned vs. platform owned? It's obviously too early to assess, but the presumed advantage of Claude Commerce Agents here is the ability to gain more control of the agent's capabilities. More configuration equals more opportunity to adjust the levers for greater conversion and related metrics. There is a required investment into Anthropic's version in terms of implementation, while the token cost is negligible for most merchants; this means that over an extended period of time, that cost has to be netted back (at a minimum) to create a positive ROI. There are certainly many businesses who won't want to go through the complexity and cost of running their own bespoke Commerce Agents, even with a blueprint in place. Thus, there is really no 'win' here, they are two completely different models. Shopify offers Sidekick with its existing plans, making it seamless to integrate but less customizable. Anthropic offers a more robust, customizable package that is more costly and complex to integrate, but comes with more upside. The running of the CPCT here illustrates the value the framework has. A simple execution of the 25 turns for $4.17 provides valuable data and insights to map out the full workflow and use case. Combined with a P&L and detailed specs, it creates clarity and structure for the decision on which to use. Overall, it will be great to see some future numbers published by a non-partner brand on Claude Commerce Agents about the cost and metrics associated with implementation and launch. Independent benchmarks with deeper analysis would also help to establish a better picture of the opportunity (or lack thereof) here. Finally, another frontier lab could launch a competing product and upend all the assumptions and analysis outlined in this post. But for now, the CPCT provides some baseline insight into the mechanics of these new agentic models for commerce. The expensive part was never the model! ## Revisions - 2026-09-26 — A consumer agent now buys from stores like the one measured here — see [Meta's Muse - Has the Demand Side Arrived to Agentic Commerce?](/notebook/metas-muse-has-the-demand-side-arrived-to-agentic-commerce/). --- Written 100% by hand by [Joel](/about/) — no AI in the prose. Research and contextual support from Claude; shipped on Claude Code.