What the machine is actually doing

You have spent a career learning to read documents that want to be believed: borrower financials, appraisals, management responses. This module is about reading one more document that wants to be believed — the output of a large language model — and it starts with what the machine is actually doing when it writes.

It writes one most-plausible word at a time

A large language model is a system trained on an enormous amount of text to do one thing: given everything so far, produce a plausible next word. Then the next. Then the next, until the answer looks finished. That is the whole trick. The drafts it produces are not retrieved from a database of facts and they are not computed the way a spreading tool computes — each sentence is assembled, word by word, to be plausible in context.

This is why the machine is so good at the parts of your work that are about form: a workpaper narrative, a summary of a credit agreement, issue language in the house style. Plausible-next-word is exactly the skill those tasks reward.

It is also why the machine can produce a figure that was never in the file. When the plausible next token is a number, it writes a plausible number. A DSCR of 1.31x reads exactly as smoothly as a DSCR of 1.29x, whether or not either one ties to the spread. Nothing inside the machine distinguishes recalled from composed — that distinction lives in your verification, or nowhere.

The working frame for the rest of this program: a fluent junior reviewer with no memory, no accountability, and no ability to know what it does not know. You would never let that reviewer sign anything. You would absolutely let them draft.

Its entire world is the context window

The machine has no memory of your last session, your institution's rating scale, or the borrower you asked about yesterday. What it "knows" during a task is exactly what is in front of it — the instructions and documents in the current conversation. Practitioners call that working area the context window, and it is finite.

Three consequences you will meet in real files:

• Nothing persists. Ask it to apply the same add-back treatment it agreed to last week and it will not remember last week, though it may confidently act as if it does.
• Long files get thin. A four-hundred-page credit file may not fit, or may be summarized on the way in. The exhibit that matters most to you — the one amendment, the one guarantor schedule — can silently fall outside what the machine actually read.
• If it wasn't given the document, it doesn't have the document. The machine will still answer. An answer about a compliance certificate it never saw is composed, not read — and it will not be labeled as such.

The reviewer's question "what did you base that on?" has a precise machine version: was the source actually in the window? Module 3 turns that question into a control.

Why it sounds confident when it is wrong

Here is the property that costs review functions the most: the machine's confidence is a property of its prose, not of its evidence.

It learned to write from text that is overwhelmingly fluent and assured — reports, memos, filings. So it writes fluently and assuredly, whether the content beneath is rock-solid or invented. It does not hedge more when it is guessing, because it does not know it is guessing. There is no internal alarm that rings when the plausible next number is a fabricated one.

Pair that with how you read. Twenty years of reviewing files taught your eye that polish correlates with care: the analyst who writes a clean memo usually ran the numbers. That heuristic is now false. The machine produces the polish at zero cost, with or without the care. A well-formatted, confidently worded, correctly totaled draft is precisely what this technology emits by default — the shape of verified work, without the verification.

Module 4 catalogs the failure modes this produces. For now, one rule survives contact with everything: read the machine's tone as formatting. Only sources are evidence.

Same question, different answers

Run the same prompt twice and you will often get two different outputs. The machine composes rather than retrieves; by design it does not walk the same path every time. Vendors can reduce this variation, and the tooling can pin versions and log prompts — but you should assume, as a property of the technology, that output is not reproducible the way a calculation is reproducible.

For a review function this lands in two places:

• Workpapers. "Rerun it and check" is not a re-performance standard the way recomputing a ratio is. What makes AI-assisted work reproducible is the record around it — the prompt, the model version, the inputs, and your verification — not the hope that the machine says the same thing twice.
• Model changes. The system behind the tool is updated over time. The same prompt that behaved one way in March may behave differently in September.

The exercise below makes you feel this one in your hands.