TechKiln

Running

[ work ]   llm-wiki   ·   running

Every agent starts knowing what went wrong last time

A repository of 69 rules, each written from an incident and carrying that incident with it, read by every agent before it does any work. Built and run by one person and a team of agents — which half did what is named further down.

Before

A lesson learned in one session lived in that session's memory, and a subagent never receives that. So the same mistake stayed available to every agent that had not personally made it.

Now

The rules live in their own repository and arrive before the first token of work. The planner and the critic read them before proposing or reviewing; the writer reads the one page that governs what it writes.

$ agent start --role planner

  rules.md — how work is done
  ------------------------------------------------
  R24  Never state a record as fact without
       checking it.
  R30  Do it yourself before handing it back.
  R47  Align once before starting, then work
       without asking.

  69 rules · 11 pages · injected before the
  first token of work

what the thing does

A mistake is made once, and then it belongs to every agent.

How a mistake becomes a rule

Transcriptevery sessionCorrectionmined by an agentHuman approvesnever automatedPagewith its incidentEvery agentreads it firstWeekly lint87 citations · 0 issues · and two axes it cannot check

One direction only — a rule enters where a human approved it

How it is used

Read before, not after

The planner and the critic read the rules before they propose or review — not as a checklist afterwards, when the work is already shaped.

A rule needs a person

Drafting is automated. Approval never is. A system allowed to write its own rules will write rules it invented, so that one step stays human.

Checked every week

A scheduled lint reads every page, counts whether each claim still carries its source, and reports the axes it cannot check itself.

What is true now

69

rules, never renumbered

11

pages

87

citations, lint green

79

commits, each one read

What it turned out to be good for

[ 01 ]

The miner found rules we had already written

Of twelve rules it proposed from raw transcripts, two already existed — rediscovered independently, from the incidents rather than from the page. That is the clearest evidence the detector reads behaviour instead of echoing us.

[ 02 ]

A rejected idea stays rejected

Everything ruled out keeps its own auditable page, with the measurement that ruled it out. When the same proposal comes back — and it does — it meets its own evidence instead of a vague memory.

[ 03 ]

A claim with no source becomes visible

87 citations, zero open issues. The lint also names the two things it cannot check, so a green run is never mistaken for a clean bill of health.

Who did which half

Both halves are named on purpose. The interesting part of this one is the seam — what is worth automating, and the one step that is not.

The human

  • Decides a correction is real
  • Approves every rule before it exists
  • Rejects — and the rejection is kept, with its reason

The agents

  • Mine the transcripts for the moments a correction was made
  • Draft the rule, and must supply a case that would fail it
  • Run the weekly lint and report what it did not check

Questions people asked

  1. What is llm-wiki?

    It is a list of rules that the AI assistants here read before they start any work.

    Here is the problem it fixes. When I catch one of them making a mistake and correct it, that correction only lives in that one conversation. The next assistant I ask for help is a fresh one — it never heard the correction, so it is free to make exactly the same mistake. And it does. Every one of them ends up learning the same lesson the hard way, over and over.

    So the corrections got written down in one place, and now every assistant reads it before it does anything. There are 69 rules so far. One of them says: never state a record as fact without checking it — written the day one of them told me something confidently that turned out not to be in the file it claimed to have read.

    Each rule carries the mistake it came from. That part matters more than it sounds: a rule with a real story attached is hard to wave away later, and easy to check against.

    What it is not: a manual, or a description of how anything works. It only contains things that went wrong once.

Ask about this one

How a rule gets drafted, what the weekly lint cannot check, why approval is the one step that is never automated — ask, and the answer goes up on this page.

Answers are published on this page, and your question may be quoted — edited or anonymised where that helps. Your email is never published and is only used to tell you an answer went up.