The AI-OS Mental Model — ห้ารอยต่อที่การรับประกันเดิมพังThe AI-OS Mental Model — Five Seams Where Old Guarantees Break

🤔 Your team calls the AI system it operates "a chatbot" — but what if the thing you are actually operating is an entire computer, with no memory protection, no system-call authorisation, and a kernel whose source nobody can read?

In the previous post, Software 1.0 · 2.0 · 3.0, we saw that the three eras of software do not replace one another but stack as layers inside one deployed system, and that the engineer's leverage has moved from control flow, to dataset and loss, and now to the context assembled at runtime. We closed with Nong Kram's artifact inventory, naming which layer determines each piece of behaviour. The question left open: once the 3.0 layer becomes the layer that determines the most behaviour, what frame should we use to see it whole — because the mental model you choose decides which set of questions you ask of the system, and which set of risks you walk past without noticing.

The paper's answer^[1], and this post's, fits in one sentence: model the AI core as an operating system, not an application — weights are the CPU, the context window is RAM, retrieval is the file system, tool calls are system calls, agents are processes, and natural-language chat is the shell. But the real value of the analogy is not its elegance — it is its cracks. Every row of the table carries a guarantee that operating systems have given us for fifty years, and the AI side loses every single one. Knowing exactly where they are lost is the engineering map for the rest of this series. This post ends with the first artifact drawn on that map: Nong Kram's seam register, which post #8 will pick up and convert, row by row, into control rails.

1. The AI Core Is an Operating System, Not an Application

The conclusion first: Table 4 of the paper lays out six mappings — weights ≈ CPU, context window ≈ RAM, retrieval/RAG ≈ file system, tool calls ≈ system calls, agents ≈ processes, and natural-language chat ≈ the shell^[1]. The immediate benefit of the table is not that the analogy is clever, but that it moves your eyes: from looking at "a chat feature" to seeing an entire machine — a machine with working memory, files to read, an interface that reaches external authority, and several tasks running at once on a single computational core.

Why does a mental model deserve a whole post? I will vouch for this from real system reviews: a team that sees its system as an "app" tests it like an app — type a question, read the answer, feel satisfied, ship. A team that sees a "machine" asks the questions a machine's operator asks: is memory protected, who is allowed to make system calls, how far back and how precisely can we read the logs. Different question sets lead to different control sets — and Table 4 is the cheapest question-set changer I know of.

The real asset is the data and the protocols, not the chat

Here is the corner many organisations get backwards, and the paper states it plainly: the shell — the chat window everyone sees — is the most replaceable part of the whole machine. The real asset is the data and the protocols — the curated knowledge corpus, the tightly defined tool schemas, the context-assembly rules — the things that accumulate value and carry over across model versions^[1]. For KramKraft this translates directly: a well-organised policy/ and catalog/ corpus and a refund schema with its conditions written out will outlive the decision of which vendor's model to use, several times over.

And the sharpest sentence of this section of the paper describes what happens when an organisation does not see it this way: a chatbot glued onto an existing product is "one application, mistaken for a computer"^[1]. Organisations that "buy a chatbot" usually get one application with no machine underneath — no maintained file system, no system-call checking layer, no machine-level logs. Then, the day they want it to "do more", they wire tools straight into it, and the result is an entire computer assembled by accident, whose protection layers nobody ever designed — not even one of them.

💡 My view: use this analogy as a question generator, never as an architecture blueprint — nobody should build a scheduler or virtual memory for a model because this table says so. The value of Table 4 is that it makes operating-system-grade questions surface in the review room: where is our memory protection, who authorises our system calls, and who holds the kernel-level logs — those three questions are enough to open every piece of work in the rest of the series.

2. The Seams Where the Old Guarantees Break

Every analogy has the places where it breaks, and for Table 4 those places are the entire point: the operating-system side of every row carries a structural guarantee that software engineering has leaned on throughout its history, and the AI-core side keeps none of them^[1]. This section walks all six seams row by row, because the position of these cracks is the series' map of work: these seams are where the engineering is hard, and where all the controls will be placed.

A CPU that returns a sample, not an answer

A real CPU gives a guarantee we have forgotten is a guarantee at all: same instruction, same data, same result, every time, forever. The model does not give this — what it returns is a single sample drawn from a probability distribution^[1]. Run the same question again and get a different answer: not the symptom of a bug, but the machine's normal behaviour.

The caveat the paper attaches here is precise and very frequently misquoted: deterministic decoding — temperature zero, a pinned seed — removes only the sampling variance; it does not remove version sensitivity^[1]. It makes the word-selection rule reproducible, but the day the provider updates the model version, the entire underlying distribution shifts — the equivalent of swapping out the whole CPU while the old opcodes silently produce new results. This is why Nong Kram's M_v must be pinned to a specific version, and why the next post will force every remaining term to be pinned as well.

RAM with no memory protection

A real operating system has an MMU separating each process's address space, kernel memory split from user space, and code pages split from data pages — by design, one program cannot reach over and overwrite another's memory. The context window has none of this: the owner's system prompt, the customer's messages, retrieved document passages and tool outputs all sit in one channel as identical tokens, with no bit telling the model that this token is "an instruction from the house" while that token is "data from a stranger" — instructions and data share a single channel^[1].

That single channel is the structural basis of prompt injection^[1], and the more dangerous form is indirect prompt injection: the attacker does not even have to type into the chat — they only plant text somewhere the system will later read in by itself. Greshake and colleagues demonstrated this against real LLM-integrated applications back in 2023^[2], and OWASP places prompt injection first on its Top 10 for LLM Applications, 2025 edition^[3]. For Nong Kram, at least two roads into this channel exist today: customer-written product reviews embedded in the retrieved catalog pages, and the text of payment slips customers upload.

A file system with no stable read contract

The file system's contract is so simple nobody thinks of it: the same read returns the same bytes until someone writes. Retrieval has no such contract — "reading the same file twice" can return different sets, because the corpus has been re-indexed, the ranker has changed versions, or one new document was added and the ranking shifted across the board — both the corpus version and the ranking change the retrieved set^[1].

The engineering consequence is direct: two customers asking the identical question a day apart may get answers that "read" different versions of the policy, with no record anywhere of who read what. The fix is not to freeze the corpus forever but to treat retrieval as a versioned file system — snapshots carry version ids, and the set of passages read per answer gets recorded. Step 5 in the next section turns this into a task that finishes in a single day.

System calls that are requested in natural language

In a real operating system, the system call is a hard boundary: the kernel checks authority before acting, every time, no exceptions. In the AI core this boundary is requested in natural language — the model generates text that the harness interprets as a tool call^[1]. Whatever the model generates therefore has the status of a proposal, not an action, and since the model itself can be persuaded through the entire context channel, it is disqualified in principle from being the authorisation point: authorisation must be enforced outside the model, always^[1].

This is not a textbook hypothetical — the evaluation of agent hijacking, where an attacker uses injected content to steer an agent toward the attacker's own goals, is serious enough that NIST CAISI published a technical analysis on making exactly this kind of evaluation stronger^[4]. For KramKraft the meaning is direct and expensive: a refund is irreversible from the second the payment processor accepts the instruction, so an authorisation check standing outside the model must always be in the way — does one exist today? Step 4 will force that question to be answered plainly.

Processes with no innate privilege separation

Processes in an operating system are separated by design — separate address spaces, separate user identities, separate capabilities. Multiple agents (or multiple sessions of one agent) running on the same core have no privilege separation between concurrent tasks built in at all^[1]. The only thing standing between one task and another is the harness code we wrote ourselves — our discipline, not a property of the core.

Nong Kram today is a single agent, but it serves many customers at once. The questions the seam register must put on record: what prevents one customer's order data from bleeding into another customer's context, and what prevents one customer's session from requesting a refund against another customer's order — the team's answer today rests entirely on the correctness of the session-handling code, which has never once been tested from this angle.

A kernel whose source cannot be read

The operating system's last guarantee is the deepest one: when behaviour turns strange, we can always open the kernel's source and read it. Here the analogy breaks beyond repair — weights cannot be read the way code is read, and the core's behaviour is emergent rather than specified^[1]. Ask "which line made it answer this way" and there is no answer, because there is no line.

The way out is not surrender but a relocation of investment: when the source cannot be read, observability must step in to complement what source-reading used to do^[1] — record the machine's inputs and outputs completely enough to explain any run after the fact. This grows into the full decision trace in post #7, but it starts today with a single question: if Nong Kram gave a strange answer yesterday, do we hold enough data today to know why?

Six rows, all six broken — and not broken at random. Spread the cracks over the real path of one request (receive the message → assemble the context → retrieve knowledge → call tools → release the answer) and they line up as five seams along that path: input, dialog, retrieval, execution and output — exactly where post #8 will place the five control rails^[1].

3. The Seven Steps

The next seven steps take about half a day for a system of Nong Kram's size, and they produce a single artifact — the seam register — which I lay out in full in section 4. Do them in order, because each step feeds raw material into the next.

Step 1 — Lay your system's six components onto the mapping table

Write the six rows of the table in section 1 and fill in your system's "real things" column: the model name with its version, which file the context-assembly code lives in, what the retrieval sources are, the complete list of tools, what runs concurrently, and through which channel users talk to the system. The single rule of this step: no abstractions — every cell must point at a file, a service, or a repo that actually exists. Spread out the artifact inventory from the previous post next to you and this goes much faster, because most of the items were already hunted down there.

This step performs no analysis at all — it only forces you to see what parts the whole machine has, which sounds trivial enough to skip, yet a great many systems have never had this list written down anywhere. For Nong Kram the six rows already exist as the rightmost column of the table in section 1 — this step's job is to produce the same for your system.

Step 2 — Write down the classical guarantee you are unconsciously assuming, row by row

Append to each row a sentence starting "we are quietly assuming that…" and write it honestly: that the same input yields the same answer, that the system prompt can never be overridden, that retrieval will return the same set tomorrow, that the model can only call what we permit, that sessions separate themselves naturally, and that when necessary we will be able to "look inside". The most dangerous guarantee is the one that has never been said out loud — because nobody tests what nobody has ever said.

A real example from the KramKraft team (a tutorial-invented incident): when I asked what prevents Nong Kram from refunding beyond the policy, the first answer was "it's already written in the system prompt" — that sentence is exactly the unconscious assumption section 2 just dismantled: taking text in the context to be a constraint on the system, when it is only advice to a machine that can be persuaded through that very same channel. Write this assumption down on paper; step 4 will deal with it.

Step 3 — Hunt down your shared instruction/data channel, exhaustively

From the RAM seam: every place where third-party text can flow into the context is a road into memory that has no protection^[1][2]. Walk all of them and write them as a list. Nong Kram's came out at five:

• Customer chat messages (web + LINE) — untrusted by definition; this is the front door
• Passages retrieved from catalog/ — product pages carry customer-written reviews embedded in them, so a stranger's text walks in through the "shop knowledge" door
• Text from uploaded payment slips — arriving through OCR straight into the context as text
• Tool output: the note and address fields in order data — customers typed those values at checkout; even though they travel through our own database, they are still third-party text
• Prior chat history — everything above that ever entered once gets rolled back into the context again on the next turn

The item that usually startles the team is the fourth one — data from "our own" tool can carry a stranger's text in too. A trusted pipe does not make the content in the pipe trusted. This five-road list becomes the "what enters there" column of the RAM row in the register.

Step 4 — Stamp every tool call "requested, not enforced" — then name who enforces it today

Make a ledger of every tool in the system, then fill in the one column that matters: who enforces this today — answered by what is true, not by what is intended, and quite often the correct answer is "nothing". Nong Kram's ledger comes out like this:

# Nong Kram's tool-call ledger — surveyed honestly, as of today
get_order(order_id)               # read-only
                                  # enforced by: nothing — accepts any order_id,
                                  # not bound to the customer in this chat
refund(order_id, amount, reason)  # effectful — irreversible once the payment
                                  # processor accepts the instruction
                                  # enforced by: system-prompt text only
                                  # = no mechanism outside the model

The two occurrences of "nothing" in this ledger are the most valuable sentences this step produces — they are the admission that our system-call boundary has no kernel guarding it^[1], and they are the critical row of the register about to be born. NIST CAISI's agent-hijacking evaluation work cited in section 2 is the picture of what happens when this column is left blank in a real system^[4].

Step 5 — Treat retrieval as a versioned file system

Answer two questions on paper: one — what is the current corpus snapshot? If there is no version id to point at, there is no answer yet. Two — can the same question asked twice read different "files"? Trace when the corpus and the ranker change and who changes them. Then start recording two things per answer: the snapshot's version id and the list of passages that were read.

At KramKraft the real picture (a tutorial-invented detail) is that the team edits the policy in Google Docs and exports it for a nightly re-index — no version number, no record of which answer read which version. Two lines in the indexer's config plus one column in the log close this gap within a single day, and from that day on, every "the policy says…" from Nong Kram carries evidence of which version of the policy said it.

Step 6 — Plan observability for the opaque kernel

Play the incident-replay game with something real: take the strangest answer the system has ever given, and ask the team "what would have had to be recorded back then for us to explain it now?" The list that comes out of this game is the first draft of the trace spec — designed from real pain, not from a textbook. Nong Kram's came out like this:

# First draft of the trace spec — what must exist to explain "yesterday's strange answer"
model_id, model_version       # which computation engine actually answered
decoding                      # the temperature, top_p and seed actually used
context_assembled             # the fully assembled text, not just the template
corpus_snapshot, passage_ids  # which version of which "files" were read
tool_calls, tool_results      # what was requested, what came back
timestamp, session_id         # when, and in which conversation

The KramKraft team's motivation came from the "cracked vase" incident (a tutorial-invented case): a customer insisted Nong Kram had once said returns were accepted within 45 days, while the policy says 14. The team opened the logs and found only chat text — no idea how that turn's context was assembled, which policy passage of which version was retrieved — so they could not even determine whether the customer misremembered or the system really answered wrongly. That is the practical meaning of an opaque kernel with no observability to complement it^[1].

Step 7 — Assemble everything into the seam register

Combine the outputs of steps 1–6 into one table — the seam register — with five columns: the seam, the broken classical guarantee, what enters there, who enforces it today, and severity. Score severity with two simple questions: is this seam's worst-case damage reversible, and who can reach this seam's channel — a seam that is both irreversible and reachable by outsiders goes to the top of the table.

The reason this post's artifact is a table and not an executive summary: the paper's rule, which this series applies in every post, is that reliability questions are answered with an artifact, not an adjective^[1]. Nong Kram's full register is in the next section — commit it as docs/seam-register.md in the shop's repo, so it gets reviewed, diffed and updated exactly like code.

4. Nong Kram's Seam Register — This Post's Artifact

This is the real output of the seven steps run against Nong Kram — six rows, five columns, one minute to read and the whole machine is visible. The "who enforces this today" column answers for the system as it stands before this series steps in to fix it, not as the team wishes it were.

How to read it: start from the rightmost column. There is exactly one critical row — execution — because it is the only row where the damage is real money and irreversible. Next come three high rows that tie together into one cluster: untrusted text flows into the same channel as instructions (RAM), from sources with no version number (retrieval), and when something happens nothing can be explained after the fact (shell). This severity order is the priority order of the work — not the order of the lessons in this series.

💡 My view: the column that actually changes the meeting is "who enforces today". Printing the word nothing into a document everyone can see is completely different from knowing it quietly — once the word nothing is on paper, it can become a work item with an owner; while it stays in someone's head, it is merely one person's worry, and it leaves when they do.

Handover: commit this table as docs/seam-register.md in KramKraft's repo. The next post (#4) pins the terms the register says are still floating — the model version, the corpus snapshot, the context-assembly config — into one release manifest, and post #8 converts this very register row by row into the five control rails.

5. Validation Check — Audit Your Map With Artifacts

The series' rule, borrowed straight from the checklist at the end of the paper: every acceptance question must be answered with an artifact, not an adjective^[1]. Run this table against your own system — a row passes when you can point at a file, and fails when the answer is a feeling.

If a row fails, go back to the step it points at — and in my experience the row that fails most often and the row that matters most are the same one: the tool-call ledger. Do not move on to the next post while the "who enforces today" column is still blank — writing the word nothing is allowed and honest, but a blank is not.

6. The Road Ahead

This post hands over two things: a frame — your system is a whole machine, not one app — and a map: the six seams where the old guarantees are gone, plus a register that answers honestly who enforces each seam today, where most of the answers are: nobody yet. The real progress of this post is easy to measure: before reading, your team could not answer "who authorises our system calls"; after the seven steps, it can — even if the answer hurts.

What this post deliberately does not do is the pinning: the register says the model version floats, the corpus floats, the config floats, but it does not yet say how to stop them floating. That is the work of context engineering, and of the one-line equation from which the whole system's behaviour flows. The next post, Context Is a Control Artifact, unfolds that equation term by term and enforces one of the paper's strongest rules: changing any term is changing the program — so every term must be pinned in one release manifest and shipped together.