Token and cost management: the first thing that hits you in AI production

So far we’ve looked at how to use AI as a work tool: models, agents, context. Now we switch sides: how to build AI solutions, integrating them into a real product, with real users and real costs.

And the first thing that will hit you when you put AI in production is cost. It’s not a detail. It can be the difference between a profitable product and one that sinks you if it scales.

What tokens mean in production

We’ve already seen that tokens are fragments of text: everything you send to the model and everything the model sends back is billed in tokens.

In production, this multiplies. If your product makes 10,000 calls a day, every wasted token becomes wasted money.

Personal use:  a few extra tokens go unnoticed
Production:    extra tokens × thousands of calls × every day = real cost

Choosing a model is a business decision

To see the impact, let’s take a concrete case: an eCommerce store with a shopping assistant agent. Every time a user asks the agent something (which product to buy, whether it’s in stock, how shipping works), a call is made to the model.

Now we run the same query with three Anthropic models:

The calculation that matters looks like this:

cost per query × 10,000 queries/day × 30 days = monthly cost

An example with illustrative values (the real prices per million tokens are in the console and on Anthropic’s pricing page, and they change over time):

Same query. Different model. Different cost.

That’s why choosing a model isn’t a technical decision: it’s a business decision. To answer “is this product in stock?” you probably don’t need the most powerful model. For a case that requires complex reasoning, maybe you do.

How to reduce tokens without losing quality

Three concrete practices.

1. A well-structured System Prompt

When you build an AI solution, the System Prompt is where the most important configuration lives: the agent’s role, the business rules, the tone.

If it’s poorly written, with redundant information, vague instructions or unnecessary context, you pay for those tokens on every call. A clean, precise System Prompt can significantly reduce the cost per call.

Long, redundant System Prompt × 10,000 calls/day = avoidable fixed cost
Clean, precise System Prompt  × 10,000 calls/day = same behavior, lower spend

2. Context caching

If you send the same context across multiple calls, you can cache it. Anthropic charges much less for cached tokens.

In the eCommerce case, the catalog description sent with every query is a perfect candidate: it doesn’t change between calls, so there’s no reason to pay full price for it every time.

3. Smart context truncation

When the conversation grows, don’t send the entire history. Send only the most recent relevant messages plus a summary of what came before.

Full history                  → higher cost and higher risk of degradation
Recent messages + summary     → lower cost and better model focus

This solves two problems at once: it lowers the cost per call and avoids the model degradation that long contexts cause, which we covered in the episode on tokens and context window.

Monitoring in production: measure before it scales

Last, and most important: measure everything.

The Anthropic console shows usage by API key, by model and by period. With that you can see tokens per day, cumulative cost and a breakdown by model. You can also set spending alerts so you find out before costs spike.

If your product grows from 1,000 to 100,000 users and you have no visibility into consumption, you’ll find out about the problem when the bill arrives.

The bare minimum to have from day one:

• Usage broken down by API key, so you know which part of the product consumes what.

• Usage by model, to spot where you’re using a more expensive model than you need.

• Spending alerts configured before launch, not after.

Conclusion

Managing tokens is not premature optimization. It’s the first thing you should think about when AI is part of your product.

Choosing the right model, writing a precise System Prompt, caching what repeats, truncating context with judgment and measuring everything is what separates a product that scales from one that becomes unviable as it grows.

In the next episode we’ll look at how to structure the architecture when AI isn’t just another feature but the core of the product.