That screenshot is my favourite example of something I badly underestimated: building an LLM system that works in Turkish is about five percent the model and ninety-five percent everything around it.

The model part is the easy part, which genuinely surprised me. The users ask in Turkish, the schema is in Turkish — column names, and categorical values like şube (branch) or zayi (waste) — and the answers need to come back in Turkish. I’d braced for this to be the hard problem and it mostly wasn’t: modern models handle Turkish comfortably, and I write the system prompts in Turkish too, so the whole pipeline stays in-language instead of translating in and out at the edges. That’s the one model-level decision I’d insist on — stay in Turkish end to end, never round-trip through English — because every translation hop is a place for a nuance or a proper noun to get quietly mangled.

The hard part is that the entire ecosystem around the model assumes English, and every one of those assumptions is a small landmine.

The CSV export is the perfect specimen, because it’s so dumb and it bit real users. An operator asks a question, likes the answer, clicks download, opens the file in Excel — and Excel, especially on Windows, does not assume a CSV is UTF-8. With no explicit marker it falls back to the system locale’s encoding, reads the multi-byte Turkish characters as if each byte were its own character, and produces the shredded mess from the screenshot. The file was always valid UTF-8. Excel just refused to believe it.

The fix is two bytes:

# Excel on Windows won't assume UTF-8 without a byte-order mark, so it
# misreads Turkish characters (çalışan -> çalışan). Prepending a BOM
# states the encoding explicitly, and the export opens correctly.
csv_bytes = "".encode("utf-8") + csv_text.encode("utf-8")

A byte-order mark at the front of the file. That’s the whole fix. It isn’t clever and it isn’t interesting, and finding it took an absurdly long time — because every tool I used (a Mac, a terminal, a reasonable editor) rendered the file perfectly. The bug only existed in the one place I wasn’t testing: a Turkish operator’s Windows laptop, opening Excel. The lesson isn’t about BOMs. It’s that localization bugs live in the gap between your environment and your user’s, and if you only ever look at your own screen, you will never once see them.

This is what I mean by ninety-five percent everything-else. “Thinks in Turkish” isn’t a capability you switch on at the model. It’s a property you have to carry through every layer: the prompts, the schema, the model’s output, the chart labels, the number and date formatting, the encoding on the file going out the door. Miss one layer and the whole thing feels broken even when the intelligence underneath is flawless — because users don’t grade your model, they grade the spreadsheet that opened wrong.

And the defaults fight you the whole way. The libraries default to English, the encodings default to whatever was convenient in California, every tutorial’s examples are pure ASCII. Building in a non-English language means noticing and overriding a long tail of these, each one individually trivial and collectively the actual job. I’ve started thinking of it as a tax you pay for operating outside the language the tools were designed for — invisible right up until a screenshot of çalışan lands in your inbox.

If I were advising someone starting a non-English LLM product: budget for the tax, and test in your users’ environment from day one, not your own. The model will speak their language fine. It’s the plumbing around it that defaults to English, and the plumbing is where the experience is won or lost.¹

From a consulting project building natural-language analytics for restaurant businesses. Customer details and schema are abstracted; the reasoning is as built. Code is illustrative.

Here’s what happened. A user types a question, and behind the scenes the agent does real work: it routes the question to the right specialist, writes SQL, runs it against the warehouse, then summarises the result into a sentence and a chart. End to end, that’s around eight seconds — a couple of LLM calls and a database query, none of it unreasonable. And the answer that comes back is correct.

But for those eight seconds, the screen showed nothing. A spinner, maybe, doing its little spin. And what I saw, over and over, was the user wait about four seconds, conclude it had hung, and refresh the page — which of course threw away the in-flight request and started the whole thing over, making it worse. They weren’t wrong to do it. Every signal the interface gave them said “nothing is happening here.” Silence reads as failure.

My first instinct was to make it faster. That’s the engineer’s reflex: latency too high, drive it down. I spent a little while there before admitting that eight seconds for two LLM calls and a query is already roughly where it’s going to be, and shaving it to six wouldn’t change anything — six silent seconds also reads as broken. I was optimising the wrong quantity. The problem was never the latency. It was the silence. Users don’t actually mind waiting if they can see that something is happening and that it’s happening for them.

So I stopped trying to make it faster and started making it legible. The pipeline already moves through distinct stages — routing, writing the query, running it, summarising — so instead of hiding those behind one spinner, I streamed them to the front end as they happened, over server-sent events:

routing your question…
writing the query…
running it against your data…
summarising the result…

Same eight seconds. Completely different experience. Now the wait has a narrative: the user watches the system think, each line proof that work is being done on their behalf, and nobody refreshes anymore because nothing ever looks stuck. I changed the perceived latency without touching the actual latency by a millisecond. For an interactive tool, perceived latency is the one that pays rent.

And then it didn’t work, which is the part worth writing down. I wired up the events, the backend emitted them in order, and the front end received… all of them at once, in a single clump, right at the end — exactly the silence I’d built this to kill, now with extra machinery. The backend was streaming correctly. Something between the backend and the browser was holding the whole response back and delivering it in one piece.

That something was the reverse proxy. By default it buffers responses — a perfectly sensible optimisation for normal request/response traffic, and the precise opposite of what streaming needs. It was collecting my carefully-streamed events into a buffer and flushing them together, defeating the entire point. The fix is two lines, and finding them took an embarrassing fraction of an afternoon:

location /api/ {
    proxy_buffering off;            # stream chunks through, don't accumulate
    add_header X-Accel-Buffering no; # belt and suspenders for the same thing
}

That’s it. That’s the difference between a progress stream and a spinner that sits dead for eight seconds and then dumps everything at once.

I keep that snippet around partly because I’ll need it again and partly as a reminder of what “productionising an LLM app” actually consists of. The prompt engineering and the model choice get the attention, but a real share of the work that decides whether people keep using the thing is unglamorous plumbing exactly like this: streaming, buffering, timeouts, the difference between correct and correct-and-legible. None of it shows up in a demo, because a demo is one person who already knows it works and is willing to wait. Production is a stranger who assumes it’s broken the moment it goes quiet.¹

So now, when something feels slow, I ask whether the problem is the duration or the silence before I spend a week chasing the duration. Often the cheapest fix isn’t making the work faster — it’s making the work visible.

From a consulting project building natural-language analytics for restaurant businesses. Customer details and infrastructure specifics are abstracted; the reasoning is as built. Code is illustrative.

Back up. Even after grounding the model in real values and canonicalizing its filters, generated SQL still fails sometimes. A subtle type mismatch, a function used slightly wrong, an aggregation that doesn’t quite parse. The naïve response is to surface the database error to the user — but my users are non-technical restaurant operators, and Cannot GROUP BY an aggregate of type FLOAT64 means precisely nothing to them. A stack trace is not an answer.

The good response is self-correction. When a query fails, don’t give up — hand the model back its own query and the exact error the database returned, and ask it to fix it. This works far better than you’d expect, because the error message is genuinely informative: the model wrote bad SQL not knowing the column was a string, the database says so plainly, and the model corrects it. Most failures resolve on the first retry.

The trap is the word “retry,” with no number attached.

Because some queries don’t get fixed. The question is genuinely ambiguous, or the data can’t answer it, and the model doesn’t know that — so it keeps producing new wrong queries, each one different, each one looking like progress. That’s the insidious part: an unbounded correction loop doesn’t sit there obviously stuck. It looks busy. It’s generating, executing, failing, generating again, and every cycle costs another LLM call and another few seconds while the user stares at a spinner and the token meter ticks up. Left alone, it’s a loop that mistakes motion for progress and bills you for the privilege.

So the correction loop is bounded — a deliberately small number of attempts — and the bound is the whole point:

def run_with_correction(question, max_attempts=...):  # deliberately low
    sql = generate(question)
    for attempt in range(max_attempts):
        if estimate_cost(sql) > COST_CEILING:        # dry-run BEFORE spending anything
            return clarify("that looks very broad — can you narrow it down?")
        ok, result, error = execute(sql)
        if ok:
            return result
        sql = regenerate(question, failed_sql=sql, error=error)  # feed the error back
    return clarify("I couldn't turn that into a query I trust — did you mean X or Y?")

Two guards, doing different jobs. The loop bound caps how many times the model is allowed to be wrong before the system stops and asks the user a clarifying question instead of spinning. And estimate_cost is a dry run — the warehouse will tell you how many bytes a query would process without executing it — so a pathological query that would scan an entire dataset gets caught and refused before it runs up a bill, not after. One guard bounds the model’s stubbornness; the other bounds its appetite.

Picking the bound was, like most of these constants, empirical — I watched what the retries actually did. What I found: if a query isn’t fixed within the first attempt or two, it’s almost never fixed by attempt five either — those later attempts are the model thrashing, not converging. So the ceiling is low, and crossing it isn’t treated as failure, it’s treated as a signal: this question can’t be answered as asked, so stop guessing and ask the human. Falling back to a clarifying question is a much better experience than either a stack trace or a four-second silence that ends in one anyway.

It holds for any agent with a feedback loop, not just this one: giving a model the right to correct itself is powerful, but “fix yourself” without “…up to N times” is an unbounded loop wearing a helpful smile. Any time you let an LLM react to its own output — retries, critiques, multi-step plans that revise themselves — you need a hard stop that doesn’t depend on the model deciding it’s done. The model is not a reliable judge of whether it’s making progress; that’s exactly the faculty that’s missing. So the bound lives in the harness, in plain Python, where it can’t be talked out of stopping.¹

The honest open question: the right fallback when you hit the bound is probably smarter than “ask the user.” Sometimes the model’s second attempt was closer than its third, and I throw that away. A better system would keep the best partial result and offer it with a caveat, rather than discarding the whole loop. I haven’t built that yet — it’s on the list, somewhere below the things that were actually on fire.

From a consulting project building natural-language analytics for restaurant businesses. Customer details, schema, and constants are abstracted; the reasoning is as built. Code is illustrative.

So the first fix is obvious and correct: before writing SQL, check whether the question is actually answerable, and if it isn’t, ask. Is there a time range? A scope? A metric? If something essential is missing, the agent says “over what period — this week, this month?” instead of inventing one. The router could already refuse when it wasn’t sure which domain a question belonged to; this is the same instinct one level down — refuse to proceed on missing parameters rather than fabricate them.

I shipped that, felt good about it, and watched it become annoying within a day.

Because the completeness check, applied naïvely, asks every time. The user types “show me last 30 days of sales,” gets their answer, and then types “what about by branch?” — and the agent, evaluating that second question in isolation, sees no time range and asks “over what period?” The user already said. Thirty seconds ago. They said it. Now they’re answering the same question again, and the magic of “just ask in plain language” has curdled into a form with a chatbot’s manners. An agent that re-interrogates you on every turn isn’t careful, it’s exhausting, and exhausting tools get abandoned no matter how correct they are.

The real requirement, it turned out, wasn’t “ask when something’s missing.” It was “ask when something’s missing and can’t be recovered from what was already said.” That’s a much narrower trigger, and it’s the difference between an assistant and a bureaucrat.

So the agent carries a short conversational memory — the parameters from recent turns — and a missing parameter is inherited from context before it’s ever treated as missing:

# session-scoped, bounded: LRU over active sessions, TTL eviction
state.recent = [...]          # parameters resolved on previous turns

def resolve(question):
    params = extract(question)
    for slot in REQUIRED:                 # time range, scope, metric, ...
        if slot not in params:
            params[slot] = state.recent_value(slot)   # inherit before asking
    missing = [s for s in REQUIRED if s not in params]
    return ask_user(missing) if missing else params

“What about by branch?” now inherits the 30-day window from the turn before and just answers. The agent only stops to ask when a slot is genuinely unrecoverable — when you’ve changed the subject, or when you never specified it in the first place and context offers no hint. Knowing when not to ask turned out to be as much of the design work as knowing when to.

The piece I’d flag for anyone building this: keep that state bounded, and treat that as a feature, not a limitation. The agents are otherwise stateless, which is exactly what lets them scale horizontally — any instance can handle any request. The moment you bolt on unbounded conversational memory, you’ve quietly introduced a coordination problem and a slow memory leak. So the session memory is an LRU with a TTL: it remembers enough recent turns to not be annoying, and forgets aggressively enough to stay cheap and stateless-ish. Bounded memory is the compromise between “useful in a conversation” and “doesn’t become a stateful service I have to operate.”¹

It’s not perfect, and the failure mode is instructive. Occasionally it inherits a parameter the user did mean to drop — they’ve moved on to a new question, but it’s phrased similarly enough that the old time range carries over silently. That’s the mirror image of the original bug: now it assumes by over-remembering instead of by guessing from nothing. The honest fix is to make the agent surface what it inherited — “for the same 30-day window:” as a quiet prefix on the answer — so a wrong inheritance is visible and correctable rather than silent. Visible-and-correctable beats silent — that keeps coming up across this whole project, and probably deserves its own post.

What I keep relearning: the impressive-sounding behaviour — an agent that asks smart clarifying questions — is the easy 80%. The last 20% — knowing when to not exercise the impressive behaviour — is what separates something people demo from something people use.

From a consulting project building natural-language analytics for restaurant businesses. Customer details, schema, and constants are abstracted; the reasoning is as built. Code is illustrative.

With one customer, a Text2SQL agent over their data is just an app. With two — and a third in the pipeline — it’s a multi-tenant system, and the one truly non-negotiable requirement is that no customer ever sees another’s numbers. A wrong chart is embarrassing. One restaurant seeing another’s revenue is the end of the contract.

So I did the responsible thing and sketched the proper multi-tenant architecture. One shared schema, a tenant_id on every table, row-level security, every query scoped by tenant. It’s the textbook answer, it’s efficient, and it scales to thousands of tenants. I had it on the whiteboard before I noticed I was about to make a serious mistake.

Here’s what I’d glossed over. In this system the queries aren’t written by me. They’re written by an LLM, at runtime, from a user’s plain-language question. The shared-schema design makes cross-tenant isolation a property of every generated query getting its WHERE tenant_id = … exactly right. And I’d just spent a week teaching that same model not to invent branch names. I was now proposing to make data isolation — the one thing I could not afford to get wrong — depend on the model’s discipline on every query, forever. One dropped predicate, one creative join, and tenant A is reading tenant B’s books.

You can defend against that, of course. You wrap generation in a layer that force-injects the tenant filter, you audit, you write tests. But now the most important guarantee in the whole system lives in a predicate I have to enforce correctly on every single LLM-written query for the life of the product. That’s an enormous surface area for something whose spec is “must never happen.”

So I didn’t build it. For two tenants — for the realistic near future of a handful — I went the other way: physical isolation. Each tenant gets its own BigQuery dataset, its own auth, and its own deployment of the same codebase, with the configuration swapped per tenant.

# one codebase, per-tenant config — isolation lives in the deployment,
# not in a WHERE clause the model has to remember every time
TENANT = load_config(os.environ["TENANT_ID"])
warehouse = Warehouse(project=TENANT.project, credentials=TENANT.creds)

# this instance's credentials can only reach this tenant's dataset.
# there is no cross-tenant query for the LLM to accidentally write —
# the data it shouldn't see isn't reachable from where it's running.

The difference is where the guarantee lives. In the shared-schema design, isolation is something the application has to do correctly on every request. In the per-tenant design, isolation is something the infrastructure makes impossible to violate. The model can write the worst query it likes; the credentials it runs under physically cannot reach another tenant’s data. I moved the guarantee from “we’ll get the filter right every time” to “there is nothing to get wrong.”

The trade-off is real, and it cuts the other way at scale. Physical isolation does not scale to thousands of tenants — past some point, standing up a deployment and a dataset per customer is the bottleneck, and the shared-schema model with its single efficient footprint wins decisively. It also costs more per tenant up front: a couple of managed containers and a dataset, tens of dollars a month each, multiplied out. Shared-schema amortizes all of that away.

But the failure modes are asymmetric, and that’s what settles it at my scale. The shared-schema model’s failure mode is cross-tenant leakage — the catastrophic one — and it sits one bug away at all times. The per-tenant model’s failure mode is “this got expensive and operationally annoying somewhere around N tenants” — a problem you watch approach for quarters and migrate against on your own schedule. One failure ends a contract; the other shows up on a cost dashboard. When the downsides are that lopsided, you don’t pick the architecture that’s elegant at a scale you don’t have. You pick the one whose worst case you can actually live with.

The mistake I almost made wasn’t technical. It was reaching for the architecture I’d be proud to have built — the one that handles thousands of tenants, the one that looks right on a whiteboard — instead of the one the problem in front of me actually needed. Two tenants don’t need multi-tenancy. They need to not see each other’s data, which is a different and much smaller requirement, and the smaller requirement has a much safer answer.

If this grows to fifty tenants I’ll revisit it, and shared-schema will probably win — by then the operational cost is real and the discipline to enforce tenant scoping properly is worth building. Recognizing which point on that curve you’re actually standing on, and resisting the pull toward the architecture that’s correct one curve over, is most of the job. I don’t always get it right. I got it right this time mostly because putting an LLM in the loop made the elegant option visibly scary.¹

From a consulting project building natural-language analytics for restaurant businesses. Customer details, infrastructure specifics, and numbers are abstracted; the reasoning is as built. Code is illustrative.

It would write WHERE branch = 'Alsanck'. Close to a real branch, but not it: a dropped letter. The query didn’t error — it’s valid SQL, valid column, just no such value. It returned zero rows. And the operator on the other end, who couldn’t read SQL and had no reason to distrust the answer, saw zero sales and concluded one of their locations had flatlined.

That’s the kind of bug that scares me — the one that doesn’t crash but quietly hands a confident, wrong answer to someone who can’t tell it’s wrong.

The fix looks obvious. The model wrote 'Alsanck', the real value is 'Alsancak', the edit distance is one. Just snap it to the nearest real value before running the query. I wrote exactly that, felt clever about it, and it was a while before I realised that “snap to the nearest real value” was about the most dangerous thing I could have done.

def canonicalize_filters(sql, distinct_values):
    # distinct_values: {column -> the real values in that column}
    for column, literal in equality_filters(sql):
        if literal in distinct_values[column]:
            continue                                  # already exact — leave it
        match, score = closest(literal, distinct_values[column])  # difflib ratio
        if score >= THRESHOLD:                        # one unambiguous near-miss
            sql = rewrite(sql, column, literal, match)
        # else: do nothing. a visible "no results" beats
        # a silent swap into the wrong real value.
    return sql

The instinct is to fix the prompt. Add instructions, add warnings, add examples of good joins. I did some of that, and it helped a little, and then I realised I was treating a structural problem as a wording problem.

The structural problem is schema linking — mapping a vague natural-language question onto the specific tables and columns that answer it. It’s the genuinely hard part of Text2SQL, and it gets worse as you add tables, not better. My data had several distinct domains — sales, inventory, waste, staffing, and so on — and stuffing all of them into one context turned every question into a needle-in-haystack search across tables that often used the same words to mean different things. More schema in the prompt meant more ways to be confidently wrong. No amount of prompt-polish fixes that; you’re asking one call to both figure out what the question is about and write correct SQL for it in a single shot.

So I split it. Before any SQL gets written, the question goes to a small classifier whose only job is to decide which domain the question belongs to, and dispatch it to a specialist that carries only that domain’s schema. Sales questions go to the sales specialist, which has never heard of the staffing tables and therefore cannot join to them. Decomposition shrinks each specialist’s schema-linking problem from “all tables” to “the handful that matter,” which is the difference between a search and a lookup.

The router is deliberately a different kind of component from the specialists. It runs at temperature zero — routing should be reproducible, not creative — and it returns a structured decision, not prose:

class RoutingDecision(BaseModel):
    specialist: str        # which domain agent should handle this
    confidence: float      # 0.0–1.0
    alternative: str | None # the runner-up, if it was close

decision = router.classify(question)   # temp=0, JSON enforced at the model layer

Pulling routing out into its own deterministic step bought me two things I didn’t fully appreciate until later. The first is that it’s testable in isolation: I can run a fixed list of real questions through the router and assert where each one lands, without executing a single query. The creative step (writing SQL) and the step I need to be boringly predictable (deciding what the question is about) are now separable, and I can hold each to its own standard.

The second is the part I’d actually put on a slide: confidence as a first-class output, and the right to refuse. A classifier that always returns its top guess is a classifier that’s confidently wrong on every ambiguous question. “How did the weekend go?” could be revenue or footfall or labour cost. The honest answer is “I’m not sure which you mean,” and the only way to give that answer is to look at how sure the router actually is:

if decision.confidence < CUTOFF:           # hand-tuned, deliberately cautious
    return ask_user(decision.specialist, decision.alternative)
    # "Did you mean sales or staffing?" — one cheap question beats a wrong report.
dispatch(decision.specialist, question)

That CUTOFF is the whole philosophy in one constant. Below it, the system stops and asks a one-line clarifying question instead of charging ahead. It costs the user a round-trip; it saves them a confident answer to a question they didn’t ask. For non-technical users who can’t read the SQL to catch the mistake, that trade is worth it every time.

The honest cost of all this: it’s an extra LLM call on every single turn. More latency, more tokens, a second thing that can fail. I took that trade on purpose, because the alternative — folding routing back into generation to save the call — gives back the determinism, the testability, and the clean place to put the confidence check. I’d rather pay for a step I can reason about than save a call on a step I can’t.

If I were starting again I’d reach for the router on day one instead of discovering it the hard way. The single-prompt version isn’t a smaller version of the right design — it’s a different design that happens to look right until the schema grows past the size of a demo. The most useful thing I built into this system wasn’t a cleverer prompt. It was a component whose job includes knowing when it doesn’t know.¹

From a consulting project building natural-language analytics for restaurant businesses. Customer details, schema, and tuning constants are abstracted; the reasoning is as built. Code is illustrative.

The model broke them constantly. Not the hard stuff — the distractors were genuinely good — but the counting. A passage that was ninety words when I asked for seventy. Five options instead of four. A stem that ran on for four sentences. These are exactly the things language models are worst at, because they don’t actually count; they generate text that feels about the right length, and “feels like seventy words” is regularly ninety.

My first instinct was the obvious one: add a second LLM call to check the first. A validator prompt — “here’s a question, does it follow these rules, reply with the violations.” It’s the natural move, everyone reaches for it, and for this particular job it’s slightly absurd. I’d be asking a model that can’t count to seventy to verify whether another model counted to seventy. Same weakness, doubled, plus another API call, plus latency, plus a new way to be wrong. You don’t fix “the model can’t count” by adding more model.

Because here’s the thing: I can count. Or rather, ten lines of Python can, perfectly, instantly, for free. Every rule that broke was a rule a deterministic check enforces flawlessly — word counts, option counts, sentence counts, length ratios. These aren’t judgment calls that need a model’s understanding. They’re arithmetic, and arithmetic is the one thing the LLM in the loop is structurally bad at and plain code is structurally perfect at. The validation belonged in code, not in another prompt.

def check(question):
    violations = []
    n = wordcount(question.passage)
    if not 60 <= n <= 80:
        violations.append(f"passage is {n} words; must be 60-80")
    if len(question.options) != 4:
        violations.append(f"{len(question.options)} options; must be exactly 4")
    if sentences(question.stem) > 2:
        violations.append("stem is more than 2 sentences")
    return violations   # empty == valid. no model, no tokens, no latency.

Run that before you call any model again, and most of the failures are caught for nothing. But catching them is only half of it. The half that made this actually work is what you do with the violations: you don’t discard the question and regenerate from scratch, and you don’t just retry and hope. You feed the specific violations back to the generator as targeted feedback — “the passage is 92 words, bring it under 80” — and let it fix that exact thing. A retry that knows precisely what was wrong succeeds far more often than a blind one, because you’ve turned a vague “try again” into a concrete instruction, and you’ve spent zero model calls figuring out what to say.

The division of labour is what makes this work, and it goes well past exam questions: let the model do the part that needs understanding, and let plain code do the part that needs precision. Generating a plausible reading passage with good distractors needs understanding — that’s irreducibly the model’s job, there’s no Python for “write something a fifth-grader would find interesting.” Verifying it’s the right length needs precision — that’s irreducibly code’s job, and routing it through a model just launders a deterministic fact through a probabilistic process and makes it worse.

There’s a cost-shaped reason this matters too, beyond correctness. Deterministic checks are free and instant; LLM calls are neither. In a system generating thousands of questions, putting the cheap, reliable check first — and only spending a model call on the targeted fix when something genuinely fails — is the difference between a pipeline that’s affordable and one that quietly triples your bill validating things a regex could have settled.

I think the reflex to solve every LLM problem with more LLM is one of the easy mistakes of building with these models right now. They’re so capable at the hard parts that it’s tempting to hand them the easy parts too. But the easy parts — counting, formatting, structure — are often precisely where they’re weakest, and where the boring old tools are not just cheaper but flatly better.¹

From an EdTech project building an LLM-based exam-question generator. Curriculum, client, and prompt specifics are abstracted; the reasoning is as built. Code is illustrative.

Those wrong answers have a name in assessment design: distractors. And a multiple-choice question is, almost entirely, its distractors. The correct option is fixed by the curriculum; there’s one right answer and everyone agrees what it is. The distractors are where all the craft lives, because a question with bad distractors isn’t a question at all — it’s a giveaway with extra steps.

Watch what goes wrong when you let a model generate distractors naively. You get three failure modes, over and over. The distractors are too obviously wrong — the question asks what plants need to grow and the options are sunlight, water, and “a bicycle” — so any student eliminates them on sight and the question tests nothing. Or they’re subtly also correct — phrased loosely enough that a smart student can argue one of the “wrong” answers is defensible, which turns the question into a trap and makes the teacher look careless. Or they’re implausible in a way that’s not even tempting — technically wrong but so unrelated that no real misconception would ever lead a student there.

What all three have in common is that the model is generating wrong answers from the answer’s point of view — “give me three things that aren’t this” — when good distractors come from the student’s point of view: what does a kid who half-understands this actually believe?

That reframe is the whole job. A good distractor is a wrong answer that a specific, plausible misunderstanding leads to. It’s the number you get if you add when you should have multiplied. It’s the definition that’s right for the neighbouring concept the student confused this one with. It’s the answer that’s correct for grams when the question asked for kilograms. Each good distractor is a little theory about how a student goes wrong — and a student who almost knows the material should feel genuine pull toward exactly one of them.

So the fix wasn’t a better prompt asking for “plausible wrong answers.” It was giving the generator an explicit menu of distractor strategies — ways students reliably go wrong — and having it build each distractor from one:

For each distractor, choose a strategy and apply it:
  - common misconception   : the wrong belief students actually hold here
  - right method, wrong step: correct approach, one realistic slip
  - adjacent concept        : the definition of a nearby idea they confuse this with
  - unit / scale error      : right number, wrong unit or magnitude
Reject any distractor that is: defensibly correct, or that no real
student reasoning would produce.

Naming the strategies did two things. It made the distractors good — each one now corresponds to a real way a student fails, so the question discriminates between kids who understand and kids who don’t, which is the entire point of an assessment. And it made them diverse: three distractors built from three different strategies cover three different misconceptions, so the question probes the concept from several angles instead of testing the same slip three times.

The reject rule in there earns its place, because it guards the one failure that’s worse than an easy question: the ambiguous one. A distractor that’s defensibly correct doesn’t just make the question easy, it makes it wrong — the student who knows the most might pick it for a good reason and get marked down, which is the exact opposite of what an assessment is for. So “is this option arguably also right?” is a hard gate every distractor has to pass, and it’s worth spending a check on.

The thing I keep turning over from this project is that the interesting problem wasn’t really about language models at all. It was about assessment design — about how children misunderstand fractions and photosynthesis — and the model was just the tool that needed to be taught that domain. I went in thinking I was building a text generator. I spent most of my time learning what makes a wrong answer worth offering.¹

From an EdTech project building an LLM-based exam-question generator. Curriculum, client, and prompt specifics are abstracted; the reasoning is as built.

It helps to be precise about why MMM needs so much. You’re not fitting a line. You’re locating, per channel, an adstock decay and a saturation curve plus a coefficient, all at once, from channels that move together so the data can barely tell them apart. That’s a lot of curve to pin down, and the information to do it comes from variation — spend going up while another goes down, a channel pausing, a budget shock. Calendar time isn’t really what you’re short of. Variation is. Sixty quiet days where every channel drifts up together contain almost no information about any individual channel’s curve, no matter how many rows it is.

The wrong response — the tempting one, because it always produces output — is to let the model run regardless and report whatever it returns. It will return something. A maximum-likelihood fit always hands you numbers; Bayesian sampling always hands you a posterior. The numbers will be precise-looking and the dashboard will render them and a UA lead will, reasonably, treat them as real and move budget. That’s the actual danger of data-hunger: not that the model errors out, but that it returns a confident answer indistinguishable on the surface from a good one, built on data that couldn’t possibly support it. A model that fails loudly is safe. A model that fails quietly, with a clean-looking number, is how thin data becomes a bad decision.

So the design principle I leaned on is that the model has to know when it knows nothing, and say so. Concretely, two things.

First, a graceful fallback to the null answer. When a market is below the data it’d need — too few installs, too little spend movement — the system doesn’t fit a heroic model on noise. It returns the honest default: a k-factor of 1.0, “no measurable halo,” no claim of lift it can’t support.

# below the data floor, the model declines to invent a signal.
# k = 1.0 means "no halo detected", not "we measured zero halo".
if not enough_signal(market):
    return Result(k_factor=1.0, basis="insufficient_data")
fit_mmm(market)

The distinction in that comment is the entire ethic. “No halo detected” is an admission of ignorance; “we measured zero halo” is a measurement. The fallback is making the first claim, never the second — and tagging the output with its basis, so downstream nobody mistakes a default for a finding.

Second, and this is what makes the fallback honest rather than just convenient: the priors do double duty. In the Bayesian setup, when the data is too thin to say much, the posterior simply stays near the prior — which is the correct behaviour, as long as you read it correctly. A posterior sitting on top of its prior isn’t the data confirming your belief. It’s the data having nothing to add, and the model honestly reporting your starting assumption back to you. The failure isn’t the model returning the prior. The failure is you reading “posterior ≈ prior” as a result. So the discipline is to always check how far the posterior actually moved, and treat “it didn’t move” as a signal that this market can’t support a model yet — not as confirmation that you were right all along.

And the maturity of a modeling system isn’t in how well it performs when data is rich — it’s in how it behaves when data is poor. Anyone can fit a clean model to two years of varied history. The professional question is what your system does on the market with sixty flat days, because that’s where the quiet, confident, wrong numbers come from. A system that degrades to “I don’t have enough to say” is worth more than one that always produces a number, because the always-a-number system is indistinguishable from the honest one exactly when it’s lying. Building the model was the easy part. Building it to know the edge of its own competence, and stop at it, was the part that made it safe to put in front of someone spending real money.¹

From work on a marketing-analytics system for a mobile-gaming portfolio. Thresholds, channels, and numbers are abstracted; the reasoning is as built. Code is illustrative.

Because a single global MMM estimates one set of channel effects — one TikTok coefficient, one saturation curve per channel — averaged across every market at once. And the markets are nothing alike. TikTok might be the dominant channel in one country and an afterthought in another; a dollar buys wildly different things in the US versus Brazil versus Indonesia; the saturation points differ by an order of magnitude because the addressable audiences do. Pool all of that and the model hands you the average channel effect across markets that share almost nothing — a blended number that’s correct for nowhere. Worse, it’ll confidently recommend shifting budget toward a channel that’s saturated in your biggest market just because it’s still cheap in a small one, because it can’t see the markets separately to know the difference.

So I split it: one MMM per country. Each market gets its own model, its own coefficients, its own adstock and saturation curves. Now the US model speaks for the US, and when it says TikTok is saturating, it means in the US, where you actually spend the money. The recommendations finally apply to a real place.

And the instant I did that, every model was starving.

This is the tax, and it’s unavoidable, so it’s worth stating plainly. MMM is data-hungry to begin with — you’re locating several nonlinear curves at once, and that needs a lot of history. Split your data by country and you’ve sliced that same history into N thinner piles, each feeding a model that’s just as hungry as the global one was. Your biggest markets have enough to fit on; your long tail of smaller countries have a handful of noisy days each, nowhere near enough to identify a curve. You traded one model that was confidently wrong for thirty models, half of which are individually too data-starved to trust. That’s not obviously a good trade, and pretending it is would be the easy lie.

What makes it a good trade is refusing to treat all thirty the same. Two things carry it.

First, a data threshold: a country only gets its own model if it clears a minimum volume. Below that line you don’t fit a desperate model on noise and present its output with a straight face — you fall back. Pool it into a regional or global model, or carry a portfolio-level prior. The small market gets a more pooled, more conservative estimate, the big market gets its own specific one, and you’re matching model granularity to the data each market can actually support instead of forcing one resolution on all of them.

# granularity follows the data, not the org chart
if country_volume(c) >= THRESHOLD:
    fit_country_model(c)          # enough signal to stand alone
else:
    fall_back_to_pooled(c)        # too thin: borrow strength, don't fake it

Second, and this is the bigger idea I only half-appreciated at the time: the right answer isn’t fully pooled or fully split — it’s partial pooling, and per-country-with-a-threshold is the poor man’s version of it. A proper hierarchical Bayesian model would let countries share a common prior and pull each country’s estimate toward the global mean in proportion to how little data it has — big markets dominated by their own data, small markets gracefully shrunk toward the portfolio average, all on one continuous dial instead of my hard in-or-out cutoff. I implemented the discrete version (own model above the line, pooled below) because it was simpler to build and operate and reason about, and it captures most of the benefit. But the hard threshold is a step function approximating a smooth one, with all the usual ugliness at the boundary — two nearly-identical small countries landing on opposite sides of the line and getting visibly different treatment. The hierarchical model is the right tool and it’s the upgrade I’d prioritize.

The way I think about it now: aggregation level is a bias-variance knob, and the extremes are almost never right. Fully pooled is maximum bias — one answer for places that have nothing in common. Fully split is maximum variance — every segment fits its own noise. The interesting work is always in the middle: how much should this segment speak for itself, and how much should it borrow from its neighbours, given how much data it actually has. Per-country MMM was my first real encounter with that question. It wasn’t the last — it’s the same question as cohort sizing, as geographic A/B analysis, as basically every problem where you have to decide how finely to slice before the slices turn to noise.¹

From work on a marketing-analytics system for a mobile-gaming portfolio. Markets, thresholds, and numbers are abstracted; the reasoning is as built. Code is illustrative.

So the central problem was never “can the model produce a plausible configuration.” Of course it can — it’ll produce a beautifully formatted, authoritative-sounding parameter set whether or not a single value is correct. The problem was making sure every value it produced came from the documentation rather than from the model’s own half-remembered training data. The specs are dense, version-specific, and full of values that look interchangeable but aren’t. A model running on parametric memory will cheerfully give you a parameter that was right in some other release, for some other vendor, with total confidence. The fluency is exactly what makes it dangerous.

This is what retrieval-augmented generation is for, and I think it’s often explained backwards — as a way to give the model “more knowledge.” That’s not the point that matters here. The point is to change where the answer comes from. Without retrieval, the model answers from its weights — an averaged, lossy compression of everything it read in training, with no notion of which source said what or whether it’s current. With retrieval, you fetch the actual relevant passages from the actual documentation at question time, put them in front of the model, and constrain it to build its answer from those passages. You’re not topping up its memory. You’re moving the source of truth out of the weights, where you can’t inspect or trust it, and into a set of documents you control.

question  ->  retrieve the relevant spec passages from the real docs
          ->  answer USING ONLY those passages, and cite which one for each value
          ->  if the passages don't contain the answer, say so — don't fill the gap from memory

That last line is the one that does the real work, and it’s the discipline most RAG systems are too lax about. Retrieval gets you the right pages; it doesn’t automatically stop the model from also leaning on its parametric memory when the retrieved text is thin. In a low-stakes setting, a model that quietly supplements the documents with its own guesses is fine, even helpful. In network configuration it’s the whole danger reintroduced through the side door — a value that looks retrieved but was actually invented. So a large part of the work was forcing the failure to be honest: when the documentation doesn’t contain the answer, the correct output is “the docs don’t specify this,” not a confident fabrication that happens to be ungrounded. An agent that admits the gap is safe. An agent that papers over it with a plausible number is exactly the thing you were trying to prevent.

The other half was making the grounding auditable, which falls out naturally once the answer comes from retrieved passages. Every value the system proposed could point back to the specific passage it came from. That traceability isn’t a nicety in this domain — it’s what lets a network engineer trust the output enough to act on it, because they can check the source rather than taking the model’s word. A configuration you can’t trace back to a document is a configuration nobody responsible will deploy, no matter how confident the model sounded. The citation is the difference between a suggestion an expert can verify and a black box they have to either blindly trust or ignore.

It’s the same pattern across everything I’ve built with these models since: an LLM gets useful in serious settings not when you trust it more, but when you arrange things so you have to trust it less. Retrieval grounds it in sources you can check; forcing it to abstain when the sources are silent stops it filling gaps with confident noise; citations make every claim verifiable. None of that makes the model smarter. It makes the output checkable — and checkable is the bar for anything where being wrong has a cost.¹

From my MSc thesis on agentic AI for telecom network configuration. The domain specifics are generalised; the design reasoning is as built.

The first instinct — mine included — was to treat this as a signal-quality problem and patch it. Model the missing conversions, reconstruct the probable attribution, stitch SKAdNetwork postbacks back into something resembling the old per-user view. A lot of very smart engineering went into this across the industry, and it always felt like what it was: building an ever-more-elaborate prosthesis for a limb that wasn’t coming back. You were estimating the per-user link, then building your decisions on the estimate as if it were the measurement, compounding one layer of uncertainty on top of another.

What actually helped was to stop asking “how do I recover attribution” and start asking “what was attribution ever for.” It was for deciding where the budget goes. And the marketing mix model I’d already built for organic lift answered that question without ever needing a per-user link — because MMM never looked at individual users in the first place. It works at the aggregate level: total spend per channel per day, total installs per day, the relationship between the two over time. It was, almost by accident, already privacy-proof. The thing ATT broke was a thing MMM never depended on.

That changed the posture completely. Instead of a degraded attribution signal propped up by modeling, I had a method whose required inputs — aggregate spend, aggregate installs — Apple’s changes didn’t touch at all. The privacy wall that demolished per-user attribution is no obstacle to a model that only ever read the totals.

But it forced one real change, and it’s the part worth writing down. On iOS, installs per channel are now the unreliable, privacy-fuzzed quantity — that’s exactly what SKAdNetwork degraded. So on iOS I stopped feeding the model paid installs and fed it the one number Apple can’t obscure: spend. You always know what you spent. Your own billing is ground truth, immune to anyone’s privacy policy. Android, where attribution still largely works, keeps using installs as the input. The platforms diverge on purpose:

# the input variable depends on what each platform can still measure honestly
if platform == "ios":
    media = paid_spend       # SKAdNetwork fogged installs; billing is ground truth
else:  # android
    media = paid_installs    # per-install attribution still reliable here

That looks like a small config branch, but the rule behind it matters: feed the model the most reliable observable, and let that differ by platform. On iOS the trustworthy observable is what left your bank account; on Android it’s still the install count. Rather than force both platforms through one pipeline and pretend iOS installs are as solid as Android’s — quietly poisoning the model with a number you know is fogged — you let each platform contribute the signal it can actually stand behind. The model on iOS answers “how do organic installs respond to spend,” the Android model answers “how do organic installs respond to paid installs,” and both are honest about their own inputs instead of one of them laundering a broken number.

There’s a real cost here. Spend and installs aren’t interchangeable inputs — spend folds in price (CPI moves with auction dynamics and seasonality, so a spend-based model partly measures cost fluctuation, not just volume), and the two platforms’ results no longer sit on the same axis, which makes a clean cross-platform total something you assemble carefully rather than read off. I accepted that. A coherent measurement built on a number that’s actually true beats an apples-to-apples comparison built on a number I know is fiction on one side.

The wider point: when a measurement breaks, your best move is often not to repair the measurement but to find a decision-making method that never required it. Reconstructing the old per-user signal was the locally obvious move and a strategic dead end — pouring effort into restoring a capability the platform had deliberately removed and would keep removing. Switching to a method that operates on data the privacy changes don’t touch was the move that aged well, because it stopped fighting the direction the whole ecosystem was visibly heading. The future of measurement in a privacy-first world looks a lot more like aggregate modeling and a lot less like following individual users around, and ATT was just the early, loud announcement of it.¹

From work on a marketing-analytics system for a mobile-gaming portfolio. Platform specifics, channels, and numbers are abstracted; the reasoning is as built. Code is illustrative.

Ridge regression — least squares with a penalty on the sum of squared coefficients — is exactly the same computation as finding the most probable coefficients under a Gaussian prior centred at zero. Not similar, not analogous: the same optimisation problem, the same answer, written twice. The penalty strength and the prior’s tightness are the same knob in two notations. Lasso, with its penalty on absolute coefficient values, is the same story with a different prior — a Laplace distribution, which is spikier at zero and explains, in one line, why lasso drives coefficients exactly to zero while ridge only shrinks them toward it. The frequentist “penalty term” and the Bayesian “prior” are the same act of telling the model what to believe before it looks at the data.

Once you see it, you can’t unsee it, and it reorganises a lot. Adding an L2 penalty is saying “I believe these coefficients are probably small, clustered around zero.” That’s not a neutral, assumption-free safeguard — it’s a belief, as opinionated as any Bayesian prior, you were just expressing it through the back door of an optimisation penalty instead of the front door of a probability distribution. I made this same argument about marketing mix models — that ordinary least squares and ridge both encode beliefs, they just don’t admit it — and this is the general version. There is no such thing as an unopinionated model. There’s only models whose opinions are stated openly and models whose opinions are smuggled in as defaults.

minimise:  ‖y − Xβ‖²  +  λ‖β‖²          # ridge: a penalty you tune
        =  (least squares)  +  (penalty)

equivalently, maximise the posterior under:
    y | β  ~  Normal(Xβ, σ²)             # the likelihood (the fit)
    β      ~  Normal(0, τ²)              # a PRIOR: "coefficients are ~small"
    with   λ  ∝  σ²/τ²                   # penalty strength = prior tightness

That λ ∝ σ²/τ² is the whole bridge in one line. A bigger penalty is a tighter prior — a stronger prior belief that coefficients are near zero. A penalty of zero is an infinitely wide prior: no belief at all, pure least squares, free to overfit. When you cross-validate to “tune the regularization strength,” you are, in Bayesian terms, letting the data tell you how strong your prior should have been. The two cultures are solving one problem and arguing about the vocabulary.

Why does this matter beyond being a neat fact? Because the prior framing tells you what you’re assuming, and the penalty framing hides it. “Coefficients are small around zero” is a fine default for many problems and a terrible one for others — if you have a predictor you know is strong, an L2 penalty is actively fighting you, insisting it should be small when you know it isn’t. Seeing the penalty as a prior makes that conflict legible: you’d never knowingly put a tight zero-centred prior on a coefficient you believe is large, but you’ll do exactly that, by reflex, when you slap a uniform ridge penalty on every coefficient including that one. The Bayesian view also points at the fix — different priors for different coefficients, which in penalty-land is the slightly awkward “feature-specific regularization,” and in prior-land is just… using what you know.

I don’t think one framing is correct and the other wrong. They’re genuinely the same thing, and I move between them by convenience: when I want a fast fit with a library, I think “ridge” and tune λ; when I want to reason about what the model assumes, or put a belief on a specific parameter, or get honest uncertainty, I think “prior” and reach for the Bayesian version. What changed for me wasn’t a technique — it was realising the wall between the two boxes was never there. Regularization was Bayesian all along; nobody told me, because the frequentist notation is careful never to mention it.

So now, when I regularize, I ask what prior I’m implicitly asserting, and whether I’d actually endorse it if it were written as a belief instead of a penalty. Usually I would. Sometimes, written plainly, the assumption is obviously wrong for the problem — and I’d never have noticed while it was disguised as a harmless-looking λ.¹

From the statistical-learning foundations of my coursework — the connection that quietly unified two things I’d kept apart. Code is schematic.

The mechanism is obvious once you say it out loud. You run a big Meta campaign. It drives paid installs, which attribution dutifully records. But it also pushes the game up the store charts, where new people discover it organically. Some of those paid users tell a friend, or just get seen playing. The campaign manufactured installs that nobody clicked an ad for — and your attribution tool, which can only credit an install it directly touched, files every one of them under “organic” and moves on. The spend gets none of the credit for the demand it actually created.

This is the halo effect, and it’s not a rounding error. If you optimize your UA purely on attributed paid ROI — which is what almost everyone does, because it’s what the dashboards show — you systematically underspend, because you’re crediting each channel only with the installs it directly touched and ignoring the wave of organic demand it set off behind them. You’re flying on an instrument that can’t see a chunk of the thing you’re trying to maximize.

So I stopped trying to measure paid ROI better and changed the question the model was answering. Instead of “how many paid installs did each channel get,” I pointed a marketing mix model at a different target entirely: organic installs. Not paid. Organic — the very installs attribution calls free — as the response variable, with paid spend per channel as the inputs.

That inversion is the whole idea, and it took me a while to be comfortable with how strange it looks. You’re regressing the thing you supposedly can’t buy onto the things you did buy, and asking: when paid spend moves, how much does the organic baseline move with it? Whatever the model can explain — the portion of organic installs that systematically rises and falls with paid spend, after adstock and saturation — is the halo. It’s paid-driven demand hiding in the organic numbers, and MMM can see it precisely because it works at the aggregate level, on correlations over time, instead of trying to trace individual clicks the way attribution does. Attribution asks “did this person touch an ad.” MMM asks “does organic move when spend moves.” Only the second question can find an install that nobody clicked.

The model decomposes organic installs into two parts. The baseline — the intercept, plus trend and seasonality — is the genuinely organic demand: what you’d get at zero paid spend, the brand, the back-catalogue, the season. The media contributions are the halo: the slice of organic installs that each channel’s spend is driving up on top of that baseline.

# target is ORGANIC installs. the decomposition splits them into the
# true baseline (what you'd get at zero spend) and the paid-driven halo.
contributions = create_media_baseline_contribution_df(
    media_mix_model=mmm,
    target_scaler=target_scaler,
    channel_names=channels,
)
# baseline  -> genuinely organic
# per-channel -> organic installs that paid spend manufactured

And then the number that actually changes the conversation — collapse it into a k-factor per channel, a virality multiplier:

# k = 1.15 means: every 100 paid installs from this channel come with
# ~15 organic installs in their wake that attribution credited to nobody.
k_factor = (paid_installs + halo_installs) / paid_installs

A k-factor of 1.0 means a channel buys exactly the installs it’s credited with and nothing spills over. Above 1.0 means it manufactures organic demand on top — and now you can compare channels on their true pull, paid plus halo, instead of the attributed-paid number that flatters the channels with no halo and punishes the ones quietly driving your charts. Two channels with identical attributed ROI can have very different real value once you count what they set off, and the team that knows that allocates budget differently from the team that doesn’t.

One caveat, and it’s a big one. This is a correlational decomposition, not a clean causal experiment. The model attributes the co-movement of organic installs and paid spend to the halo, but co-movement isn’t proof — a confounder that drives both (a seasonal surge, a press hit that coincided with a planned spend ramp) gets quietly absorbed into a channel’s contribution, and the model can’t tell that apart from genuine halo on its own. The gold standard for incrementality is a geo holdout or a proper lift test, where you actually withhold spend and watch what happens. MMM is the always-on, every-channel estimate you run when you can’t afford to stop spending everywhere to find out — and the right way to use it is to validate it against the occasional real lift test, lean hard on the priors, and read the k-factors as well-reasoned estimates rather than measured facts. I’d rather an honest estimate of the right quantity than a precise measurement of the wrong one, and attributed paid ROI is a precise measurement of the wrong one.

So your “organic” bucket is partly a measurement artifact — it’s where attribution files the demand it couldn’t trace, including the demand your own spend created. The most valuable thing a channel does might be the installs it doesn’t get credited for, and the only tools that can see those are the ones that stopped trying to follow the click.¹

From work on a marketing-analytics system for a mobile-gaming portfolio. Channels, k-factors, and numbers are abstracted; the reasoning is as built. Code is illustrative.

Take the timing one first. You run a burst of TikTok spend today. Some installs land today — but some land tomorrow, and the day after, as people who saw the ad get around to acting on it, as the creative circulates, as the algorithm keeps serving it. The effect of today’s spend is smeared forward over the following days, decaying as it goes. A model that credits today’s spend only with today’s installs misreads this completely: it sees spend, then a lagged bump in installs it can’t connect to the cause, and it either misses the effect or pins it on whatever else happened to move that day.

The fix is adstock (carryover): before the spend ever enters the model, you transform it so each day inherits a decaying echo of the days before it.

# geometric adstock: today carries a fading memory of prior spend.
# lam ~ 0  -> effect is almost entirely same-day
# lam ~ 0.8 -> a long tail; today's spend still matters a week later
def adstock(spend, lam):
    out = np.zeros_like(spend)
    out[0] = spend[0]
    for t in range(1, len(spend)):
        out[t] = spend[t] + lam * out[t - 1]
    return out

The decay rate isn’t something you set — it’s something you learn, and that’s the point. A channel with a long carryover (brand-ish, slow-burn) and one with an instant, all-same-day response get different decay rates, and the data tells you which is which. In the Bayesian setup this lam is just another parameter with a prior, inferred alongside everything else.

Now the second assumption, which is worse, because it’s the one that makes the model’s advice dangerous. Advertising saturates. The first slice of budget on a channel hits the cheapest, most responsive users; as you pour in more, you’re reaching down into people who are progressively less interested, and each additional dollar buys fewer installs than the last. The response of installs to spend isn’t a line, it’s a curve that bends over — steep at first, flattening toward a ceiling. Diminishing returns, the most reliable empirical fact in all of performance marketing.

A linear model cannot represent this, and the failure isn’t academic. If installs-per-dollar is a constant slope, the model thinks the channel never saturates — so its honest recommendation is put infinite budget here, because the marginal return never drops. Every linear MMM, asked where to spend more, will eventually tell you to bet everything on one channel, because it has no concept of “full.” That’s not a minor glitch — it’s the model confidently recommending the one thing every marketer knows is wrong.

So spend goes through a saturation curve — a Hill curve, an S-shaped transform with parameters for where the bend sits and how sharp it is — and only then does it hit the linear part of the model:

# Hill saturation: turns spend into effective spend, with a ceiling.
# half-saturation point K and steepness s are LEARNED per channel.
def hill(spend, K, s):
    return spend**s / (spend**s + K**s)

The pipeline per channel is the composition of the two: raw spend → adstock (smear it forward in time) → saturation (bend it for diminishing returns) → then a linear coefficient. Which is the thing that finally made MMM click for me: it isn’t a regression with marketing-flavoured variables. It’s a structured curve-fitting problem where the shape of each curve is the marketing knowledge. The adstock decay encodes “how long does this channel’s effect linger,” the saturation curve encodes “how fast does this channel get tired,” and the linear coefficient on top is almost the least interesting parameter in the whole stack.

The cost is that you’ve added two nonlinear parameters per channel, and they trade off against each other in ways that make the fit harder and the identifiability problem sharper — a strong-carryover-low-saturation curve and a weak-carryover-high-saturation one can mimic each other over a short window. This is exactly why the priors matter and why MMM is data-hungry; you’re asking the data to locate several curves at once. But there’s no shortcut worth taking. The linearity assumptions aren’t a simplification you can defend as “good enough” — one of them blinds the model to lagged effects and the other makes it recommend infinite spend. The curves aren’t sophistication for its own sake. They’re the minimum required to not be actively wrong.

I’ve kept one habit from this: when a model gives obviously broken advice, look at the shape it’s assuming before anything else. A linear model recommending infinite budget isn’t mis-tuned. It’s faithfully reporting that a straight line has no top — and the fix is to give it a curve that does.¹

From work on a marketing-analytics system for a mobile-gaming portfolio. Channels, parameters, and numbers are abstracted; the reasoning is as built. Code is illustrative.

I assumed I’d made a mistake. I hadn’t, not really. I’d just run face-first into the thing nobody tells you about marketing mix modeling: the data does not identify the answer. Several completely different stories about which channel drives what will fit your data about equally well, and ordinary regression picks one of them essentially at random — whichever one happens to drive the residuals to zero, including the ones that route a coefficient negative to cancel out a correlated neighbour.

The reason is structural. Marketing channels move together. When a game is doing well the team scales spend up across Meta and TikTok and Google all at once; when it’s cutting back, everything comes down together. So the spend columns are heavily correlated, and correlated predictors are poison for regression — the model can’t tell whether installs followed Meta or TikTok, because the two are nearly the same column. It splits the credit arbitrarily, and “arbitrarily” includes giving one channel a huge positive coefficient and its correlated twin a negative one. This is multicollinearity, and in MMM it isn’t an edge case, it’s the default condition. You can’t regularize your way out of it with ridge or lasso either — those stabilize the fit, but they shrink toward zero, which is its own unjustified opinion (“all channels are probably ineffective”) wearing the costume of neutrality.

What finally fixed it for me: the problem was never that I lacked an opinion about these channels. I had plenty. I knew roughly what a reasonable return looks like for paid UA in this space. I knew a channel can’t have a negative causal effect on installs — at worst it does nothing. I knew, within a factor, how the big networks tend to rank. I had all of this in my head, and I was running a method that threw it away and demanded the 60 days of data carry the entire load by themselves. No wonder it buckled.

So I switched to Bayesian MMM — in practice, Google’s LightweightMMM, which puts the whole thing in a probabilistic model and samples the posterior with NUTS via numpyro. And the single most important thing that buys you isn’t the fancy sampler or the credible intervals. It’s that the framework has a designed slot for the opinions you already hold. They’re called priors, and they turn “I have a hunch the model keeps ignoring” into a formal input the math has to respect.

# the model is told, before it sees a single day of data, roughly where
# each channel's effect should sit — and that effects are non-negative.
# the data updates these beliefs; it doesn't start from a blank slate.
mmm.fit(
    media=spend,                 # (n_days, n_channels), correlated columns
    media_prior=channel_priors,  # my domain belief about each channel's pull
    target=installs,
    number_warmup=1000, number_samples=1000,
)

The priors do the work that the data structurally can’t. Where the likelihood is flat — where the data genuinely can’t distinguish Meta from TikTok because they moved together — the posterior leans on the prior, and you get a sane, non-negative attribution near what you believed going in. Where the data is informative — a week where one channel moved and the others didn’t, a natural experiment the team didn’t know it ran — the likelihood dominates and the posterior moves off the prior toward what actually happened. The model spends its limited evidence updating the beliefs the data can actually speak to, instead of flailing on the ones it can’t. That’s exactly the behaviour you want, and it’s the behaviour OLS can’t give you because OLS has nowhere to put a belief.

Setting those priors well is its own craft, and an honest one — I’ve written separately about how I anchored them and the judgment that takes, because a prior is a claim and you should be able to defend it. But every MMM encodes prior beliefs. OLS encodes “I believe nothing, and I’m comfortable with a negative coefficient if it fits.” Ridge encodes “I believe every effect is probably near zero.” Neither of those is actually neutral — they’re just opinions held by accident, by people who didn’t realize they were choosing them. Bayesian MMM’s only real difference is that it makes you say your opinion out loud, in a place where someone can challenge it, instead of smuggling it in through your choice of regularizer.

I used to think putting priors on a model was cheating — tilting the result toward what I wanted. I had it backwards. The priors weren’t the bias. Pretending I didn’t have any was the bias, and the negative coefficient was what that pretense cost me. The model that states its opinions before it sees the data is the more honest one, because at least you can argue with it.¹

From work on a marketing-analytics system for a mobile-gaming portfolio. Channels, numbers, and priors are abstracted; the reasoning is as built. Code is illustrative.

Monte Carlo carries a reputation for being the honest-but-too-slow option, the thing you’d love to use for proper confidence intervals if only you could afford it. That reputation comes entirely from the naive implementation, the one that reads like the textbook description: for each cohort, for each of a thousand simulations, for each day of the horizon, draw and accumulate. Three nested Python loops. It’s correct, it’s readable, and it is unusably slow — millions of Python-level iterations per cohort, interpreter overhead on every one, and you’ve blown the Lambda timeout on a single game.

The shift is to stop picturing a thousand simulations happening one after another, and start treating the whole batch as one object. The samplers already hand you the entire block — np.random.beta with a size of (simulations × days) returns all of it in one call — and every operation after that is array-at-a-time, executed in NumPy’s C internals instead of the Python interpreter:

# the sampler already returns the whole (N_SIMS, horizon) block at once
retention = np.random.beta(a, b, size=(N_SIMS, horizon))
arpdau    = np.random.gamma(shape, scale, size=(N_SIMS, horizon))

ltv = (retention * arpdau).sum(axis=1)          # N_SIMS lifetime values, zero loops
p10, p50, p90 = np.percentile(ltv, [10, 50, 90])

There is no for over simulations and no for over days. The element-wise multiply happens across the entire (simulations × horizon) array in one vectorized operation; the .sum(axis=1) collapses each simulation’s horizon into a single LTV, leaving a thousand of them; the percentiles read the confidence band straight off that. The thousand simulations don’t run in sequence — they run as one block of arithmetic the CPU is built to chew through. Per cohort, it drops from “blows the timeout” to comfortably sub-second, which is what lets thousands of cohorts fan out across serverless workers and finish on schedule.

The performance isn’t a vanity metric, which is the part that justifies caring. Sub-second per cohort is what makes the what-if tool feel alive — a UA lead asks “what happens to portfolio ROAS if I move budget from this channel to that one?” and the answer comes back while they’re still looking at the screen, because re-running the simulation across the affected cohorts is fast enough to be interactive. Had I left the loops in, that feature couldn’t exist; you don’t build an interactive simulator on top of a computation that takes a minute. Speed changed what the system was for, not just how fast it did the same thing.

There’s a real cost to vectorizing, and I’d be lying to pretend otherwise: the code gets harder to read. A loop with named variables tells you what it’s doing at each step; a stack of array operations asks you to carry the shapes in your head — is this (sims, days) or (days, sims), what does axis=1 collapse here — and a transposed axis is a silent bug that produces plausible, wrong numbers rather than an error. I leaned on shape comments and a fast test that diffs the vectorized output against the dead-simple looped version on a tiny input, so I’d know immediately if a refactor quietly broke the math. The loop version earns its keep as the oracle even though it never runs in production.

The rule of thumb I’ve leaned on ever since: if you’re writing a Python for loop over your data points, you’re usually leaving a one-to-two-orders-of-magnitude speedup on the table. The fix isn’t a faster language or a bigger machine — it’s expressing the computation as operations over whole arrays, so the work drops into compiled code. “Monte Carlo is too slow for production” almost always means “my Monte Carlo has a Python loop in it.” Take the loop out and the technique you wanted to use all along is suddenly affordable.¹

From work on a cohort-LTV system for a mobile-gaming portfolio. Specifics, parameters, and numbers are abstracted; the reasoning is as built. Code is illustrative.

Both criteria are doing the same broad thing: estimating how well the model would predict new data, and penalising effective complexity so a model can’t win just by having more parameters to bend. The difference is in how they estimate the fit, and it comes down to one choice that sounds technical and turns out to matter.

DIC — the deviance information criterion — measures fit using the deviance evaluated at the posterior mean of the parameters. It takes your whole posterior, collapses it to its average point, and asks how well that single best-guess parameter set fits. It’s cheap, it falls straight out of the MCMC samples you already have, and for well-behaved models it’s perfectly reasonable. The hidden assumption is that the posterior mean is a good summary of the posterior — that the distribution is roughly symmetric and unimodal, so its average actually represents it.

WAIC — the Watanabe-Akaike criterion — doesn’t collapse anything. It works pointwise: for each individual observation, it averages the likelihood across the entire posterior, then sums the log of those averages. It uses the full posterior distribution rather than a single point from it, which makes it more principled when the posterior is skewed, heavy-tailed, or multimodal — exactly the situations where “the posterior mean” stops being a faithful stand-in for the posterior.

# DIC: fit measured at the posterior MEAN — one collapsed point
DIC = deviance(mean(theta)) + p_DIC

# WAIC: fit measured pointwise, averaged over the FULL posterior
WAIC = -2 * sum( log( mean_over_posterior( likelihood(y_i | theta) ) ) ) + p_WAIC

Once you see that, the cases where they disagree stop being mysterious. If your two models have similar fit but very different posterior shapes — say a Student-t model whose ν parameter has a long, skewed posterior — DIC’s collapse-to-the-mean throws away exactly the information that distinguishes them, while WAIC keeps it. So DIC can rank two models nearly tied while WAIC sees a clear gap, or, in nastier cases, they can flip outright. When they disagree, it’s usually DIC’s symmetry assumption quietly failing, and that failure is itself worth noticing — a posterior that breaks DIC is a posterior you should be looking at more closely anyway.

In my case they agreed on the winner — the Student-t model — which was reassuring rather than interesting. What was interesting was watching DIC report a smaller margin than WAIC. The Student-t’s advantage lived partly in how it handled the tails, which is partly a statement about the shape of its predictive distribution per observation, and that’s precisely the thing DIC averages away and WAIC retains. The two numbers weren’t contradicting each other. They were measuring the same thing through different amounts of blur, and the disagreement in margin was a hint about where the models actually differed.

So my rule now is boring and practical. Compute DIC because it’s free and it’s a fine first look. Reach for WAIC — or honestly, for proper cross-validation, which WAIC is an efficient approximation of — when the decision is close, when the posteriors aren’t tidy bell shapes, or when the two criteria disagree and you need to know which to believe. And when they do disagree, don’t just take the more sophisticated one’s answer and move on. The disagreement is a flag that something about your posterior isn’t as well-behaved as DIC assumes, and that’s usually worth a look before you trust either number.¹

From a Bayesian regression on public rental-listing data for Rome. Code is schematic — the real computations come from the MCMC output.

The reason there are two comes down to a fact about the input I couldn’t engineer away: cohorts arrive at wildly different levels of maturity, and a model that’s excellent for one is bad for the other. A cohort that installed this morning has almost no data — a day or two of retention, a trickle of revenue. A cohort from two months ago has a rich, fully-shaped curve. You’re asked to predict twelve-month LTV for both, from the same pipeline, and the honest truth is that no single model is good across that whole range.

The first model — call it the AR model — works the way the retention-curve approach suggests: fit retention, fit ARPDAU, integrate their product out to the horizon. When a cohort has enough data to actually fit those curves, this is the one you want — it’s mechanically faithful to how revenue accrues, and it’s precise. Starve it of data, though, and it’s worse than useless: you cannot fit a power law to three noisy points, and if you try, it’ll hand you a confident curve fit to nothing.

The second — the coefficient model — never fits a curve. It learns historical coefficients that map “revenue accumulated through day k” onto “revenue at the horizon,” normalized across cohorts to strip out scale. It’s cruder; it can’t capture a specific cohort’s curvature. But it degrades gracefully, because it needs almost nothing to produce a sane answer. On a day-old cohort it’s the robust choice precisely because it doesn’t try to be clever.

So they trade off exactly against each other along the maturity axis: AR is precise-but-fragile, coefficient is robust-but-blunt. The whole problem reduces to one question asked per cohort — which regime is this cohort in? — and the answer is just how much data it’s actually given me:

def choose_model(cohort):
    if cohort.avg_size_last_7d > VOLUME_CUTOFF:
        return AR_MODEL      # rich data: fit the curve, take the precision
    return COEF_MODEL        # sparse data: normalized coefficients, take the robustness

def predict(cohort):
    primary = choose_model(cohort)
    try:
        return primary.fit_predict(cohort)
    except FitError:
        return fallback_of(primary).fit_predict(cohort)   # never crash; degrade

High-volume cohorts get routed to precision; sparse ones get routed to robustness. And there’s a second safety net under the first: if the chosen model fails anyway — the AR fit doesn’t converge on a cohort that looked rich but was pathological — it falls back to the other one rather than throwing. The system’s contract is that it always returns a sane prediction, so “the preferred model didn’t work” can never become “the user gets an error.” Degrade, don’t die.

Which is why I think “what’s the best model?” is usually the wrong question — asking it keeps you tuning one model forever, trying to make it cover a range it structurally can’t. The better question is “what does this input let me get away with?” Here the input variable is data volume, and the answer is two specialists plus a cheap, legible router — and the router, the least glamorous part, is what makes the whole thing work in production.

One real weakness, and a sharp reader will spot it straight away: a hard threshold means two nearly-identical cohorts that land on opposite sides of the cutoff get handled by different models and can get visibly different predictions — a discontinuity right where the two models are least sure of each other. The better design is a blend: weight the two predictions by maturity and slide smoothly from coefficient to AR as a cohort ripens, instead of flipping. I know that, and I shipped the hard switch first because it was simple, debuggable, and good enough to be useful — the discontinuity sits in a region where both models are uncertain anyway, so it’s papered over by the confidence intervals. The smooth blend is the obvious next iteration. It’s just never quite outranked the things that were more broken.¹

From work on a cohort-LTV system for a mobile-gaming portfolio. Specifics, parameters, and numbers are abstracted; the reasoning is as built. Code is illustrative.