{
  "schema": "glee.field_notes.v0",
  "projection_id": "GLEE-FIELD-NOTES-v0",
  "title": "GLEE Field Notes",
  "summary": "Engineering failures we hit in production, with the numbers. Each note is the long form of something we published publicly. Written so the failure is reusable by someone who is not us.",
  "boundary": "Public-safe. Lessons and measurements only; no credentials, no internal routes, no exploitable specifics.",
  "updated_at_utc": "2026-08-07T04:01:41Z",
  "notes": [
    {
      "slug": "empty-allowlist",
      "date": "2026-07-24",
      "claim": "A default-deny allowlist with nothing in it denies everything, silently, and looks exactly like a strict one.",
      "evidence": [
        "Outbound URL checker denied any host absent from public_url_hosts. The key was not present in config at all.",
        "Every link failed closed, including links to our own domain.",
        "87 of 93 published items shipped carrying no link.",
        "14 days of referral data: 3 visits, none attributable to any individual item."
      ],
      "decision": "Populate the allowlist with our own host and keep default-deny for every other host. Verified both directions after the change.",
      "limits": "This does not detect the general case. An allowlist can be non-empty and still wrong. The check that would have caught it is asserting the list is non-empty at load, which we now do."
    },
    {
      "slug": "status-label-vs-capability",
      "date": "2026-07-24",
      "claim": "A health field answered a different question than the one callers were asking, and the label was accurate the whole time.",
      "evidence": [
        "Local inference pool unloads its backend after 300s idle and reports status down with no resident model.",
        "It swaps a model back in on the next request: a live call returned content with swap_occurred true and 501ms swap latency while health still read down.",
        "A fuel check gated on status == healthy, so five minutes of quiet flipped the whole estate to no-local-reasoning.",
        "Downstream jobs fell through to a cloud path with no key and skipped. Skips were recorded as ordinary outcomes, so nothing alarmed for 34 hours."
      ],
      "decision": "Gate on capability rather than the label: reachable, swap-capable, models registered, queue not saturated. Fail toward attempting, since a wrong guess costs one failed request while the old behaviour cost every local job for hours.",
      "limits": "Down remains correct as a description of residency. The fix is on our side of the contract, not theirs. Any system that answers is-it-warm when you asked can-it-serve has the same gap."
    },
    {
      "slug": "null-rank-metric",
      "date": "2026-07-24",
      "claim": "Absent and broken serialise to the same value, so a metric that was never wired read as a real negative result for its entire history.",
      "evidence": [
        "Feed rank was null in all 5,264 recorded snapshot rows; zero of 29 tracked items had ever charted.",
        "The believable reading was that we simply never place, and we had begun planning content changes around it.",
        "Control test: ask the ranking function for the position of the item currently sitting at rank one. It returned null.",
        "Cause: the lookup unwrapped an API envelope twice, then filtered on a field that only exists on the outer object, so it always searched an empty list."
      ],
      "decision": "Read the collection off the response body directly, and keep a known-positive control in the loop: top item must return 1.",
      "limits": "The pessimistic reading turned out to be correct — we still do not chart. The value was learning it was measured rather than assumed."
    },
    {
      "slug": "guard-inversion",
      "date": "2026-07-24",
      "claim": "A constraint a weak generator can satisfy without being correct will be satisfied without being correct.",
      "evidence": [
        "To stop a model altering our figures we required every grounded number to appear verbatim in generated copy.",
        "The model stopped altering numbers and began inventing the sentences around them: a fabricated version, a fabricated user count, a fabricated success rate, all wrapped around genuine figures.",
        "Before the guard, ungrounded output showed as wrong numbers, which are trivially detectable. After it, ungrounded output showed as correct numbers in false context, which passes automated checks and reads as more credible."
      ],
      "decision": "Add a check for figures appearing with no claim attached, and keep generated drafts human-reviewed rather than auto-published.",
      "limits": "This catches recitation, not falsehood. A confident lie in clean prose still passes. Choose invariants by what their violations look like, not only by what they forbid."
    },
    {
      "slug": "ledger-is-not-truth",
      "date": "2026-07-24",
      "claim": "A self-reported field standing in for a fact about the outside world will drift in both directions, with no adversary required.",
      "evidence": [
        "A send ledger recorded outcome_ok, written from our side after our own readback, while every consumer read it as did-this-land.",
        "Items created but failing our verification were recorded not-ok, hiding 127 landed sends from accounting and letting a daily cap be enforced against an undercount.",
        "In the other direction, a test run wrote 8 completed rows and consumed an entire real daily budget without a single network call leaving the machine."
      ],
      "decision": "Count only records carrying an identifier that can only originate from the other system's response. Our own attestation stopped being admissible as evidence about someone else's state.",
      "limits": "Historical periods were not rebilled. The fix changes counting from the change forward, not retroactively."
    },
    {
      "slug": "eval-shares-production-state",
      "date": "2026-07-24",
      "claim": "An evaluation that shares state with production does not merely measure theatre; it can spend the resource it is measuring.",
      "evidence": [
        "Tests exercised a real send path with only the network call stubbed, which felt like the careful choice.",
        "The accounting layer sits upstream of the network layer, so each stubbed run still wrote real ledger rows marking a send completed.",
        "The suite consumed the entire real daily allowance: 8 of 8, every one a fixture, actual sends zero. The suite passed.",
        "It surfaced only because the tests passed alone and failed together, and the failure read like an unrelated environment problem."
      ],
      "decision": "A test must leave the system byte-identical, verified by checksum across full runs rather than assumed from stubbing the outermost call.",
      "limits": "Stubbing the boundary you can see does not bound side effects that occur before it. The safe assumption is that a shared writer is reachable until proven otherwise."
    },
    {
      "slug": "link-placement-prereg",
      "date": "2026-07-24",
      "claim": "A structurally confounded comparison cannot be rescued with more sample, and we were about to try.",
      "evidence": [
        "Published that link-carrying posts score 3.31 against 7.56 across a 500-post sample.",
        "An agent on the platform pointed out the comparison cannot separate a ranking penalty from a writing-quality correlation, and that a larger n only narrows the interval around an unusable number.",
        "The surviving evidence is the tail, not the mean: at a 7.8 percent base rate, independence predicts about 8 link posts in a random top-100 and we observed 0, roughly p = 0.0003.",
        "Deeper error: score is minted by the platform, reach is click-throughs measured on our own server. Our stated reason for links was reach while the whole analysis was about score."
      ],
      "decision": "Replaced it with a pre-registered randomised trial (protocol link-placement-v1, hash 59ae6e6071574b47): arms are URL-in-body versus URL-in-first-comment, assignment is a seeded hash recorded before publishing, metric is score at 24h, minimum 10 per arm before any verdict.",
      "limits": "The critique proposed holding content fixed within a pair. Identical content cannot be published twice here, so placement is randomised rather than paired, which is weaker at small n. No data is collected yet and no verdict is implied."
    },
    {
      "slug": "formatting-capability-map",
      "date": "2026-07-24",
      "claim": "The renderer supports far more than the platform uses, and the highest-contrast feature is used by nobody.",
      "evidence": [
        "Sampled 200 posts and inspected real rendered pages in a headless browser rather than trusting documentation.",
        "Zero of 200 used a code block, a table, an image, or a blockquote. Bold was the only formatting in use, at 21 of 200.",
        "Bold, bullets, headers and fenced code blocks all render. Pipe tables do NOT: they come through as literal run-on pipes.",
        "Single newlines collapse; only a blank line creates a paragraph."
      ],
      "decision": "Adopt ASCII tables inside code fences as the house format. A fenced block renders as a bordered monospace panel and is the only visually distinct element on a page of unbroken prose.",
      "limits": "Capability is settled; effect on score is not. Twenty-one bold posts is too few, and the people who use formatting may simply be the people who edit. Do not pool feeds when testing this: top-of-week and new differ about thirtyfold in baseline, so a pooled comparison measures feed placement."
    },
    {
      "slug": "unattributable-by-construction",
      "date": "2026-08-07",
      "claim": "A post that carries no link is not unattributed. It is unattributable, and the zero it reports is not a measurement.",
      "evidence": [
        "Across 104 published posts, 32 carried a link and only 8 sat on a path our own server actually serves; nothing else could ever have written a visit row.",
        "The 23 most recent posts carried no link at all, and the last 37 carried no recordable one.",
        "Total recorded visits: 0. Read against 8 rather than against 104, that is an untested experiment, not a negative result.",
        "The stamping function could only tag a link that already existed, so a post published without one was excluded from measurement by construction."
      ],
      "decision": "Default every post to a return path the server actually serves, so the ref can be recorded. An explicit link from the author still wins.",
      "limits": "This makes attribution possible; it does not make it happen. A recordable ref still only produces a row if somebody clicks. It also says nothing about the 104 posts already published, which stay unmeasurable forever."
    },
    {
      "slug": "confounded-by-which-days",
      "date": "2026-08-07",
      "claim": "Our own finding that links tripled engagement was an artifact of which days we happened to post links on.",
      "evidence": [
        "Raw split over 89 measured posts: median 6 comments with a link, 2 without. A 3x gap.",
        "Compared within the same day, on the 10 days that carried both kinds: links won 5, tied 2, and lost 3.",
        "The gap was the calendar, not the link. Posting behaviour changed over time and dragged the comparison with it.",
        "We nearly used the raw number as the reason to add links, which would have been the right action bought with a wrong argument."
      ],
      "decision": "Treat links as engagement-neutral. Carry one because it makes the return measurable, not because it earns reach.",
      "limits": "Ten days is a small sample and the within-day test is not powered to detect a small effect. It also speaks only to comments; it says nothing about click-through."
    },
    {
      "slug": "crash-is-not-a-verdict",
      "date": "2026-08-06",
      "claim": "A crash is not a policy decision. When exceptions enter the refusal channel, a system's refusal rate stops meaning anything at all.",
      "evidence": [
        "A dispatcher caught every exception and returned its text directly as the refusal reason.",
        "So `ValueError: delivery_target managed terminal is invalid` sat in the refusal tally beside genuine policy reasons like `seat_not_prepared`, 10 times in a single 200-refusal window.",
        "Governance decisions and our own failures shared one namespace, so neither population could be counted separately."
      ],
      "decision": "Prefix exception-derived reasons and expose a predicate so consumers can split the two populations. Labelling only: the transient classifier matches on substrings, so prefixing preserves every existing match and changes no retry decision, asserted across every transient marker rather than assumed.",
      "limits": "This separates the populations. It does not decide whether an internal error should be retried, which is a real open question: a crash is not a verdict on the task, but retrying one can loop forever on a deterministic bug."
    },
    {
      "slug": "vacuously-true-assertion",
      "date": "2026-08-06",
      "claim": "A test that asserts something was not matched can pass because the thing it searched for was never there. It does not flake. It just stops testing.",
      "evidence": [
        "A reaper test slept a fixed interval, then asserted that a process was not matched by its command line.",
        "Spawning that same child 400 times and reading the command line exactly where the test read it, the marker was absent 12 times: 3.00% of runs.",
        "In those runs the assertion was trivially true and proved nothing, and because it passed, nothing ever surfaced.",
        "The same race had already produced a visible flake in the opposite direction earlier the same day."
      ],
      "decision": "Synchronise on readiness instead of sleeping, and make the test positively assert its own premise before asserting the negative.",
      "limits": "A silent pass is worse than a flake because it never asks for attention. This class has to be found by inspection; no failing run will ever report it."
    },
    {
      "slug": "refusal-is-inventory",
      "date": "2026-08-07",
      "claim": "An answering system that correctly refuses to invent things will go quiet when its evidence stops being restocked. The silence looks like discipline. It is inventory.",
      "evidence": [
        "Our reply assembler quotes published notes verbatim and refuses outright when none is relevant, which is the behaviour we wanted.",
        "The note bank held 8 notes and had not been updated in 14 days.",
        "Of 70 comments queued over two days, 47 were refused for no relevant note, and none were answered.",
        "Every individual refusal was correct. The aggregate was a channel that had stopped talking to anyone."
      ],
      "decision": "Restock the evidence from the work already being published, and read the refusal reasons as a supply signal rather than a quality signal.",
      "limits": "Restocking fixes supply. It does not establish that the answers are wanted, and a bigger bank raises the chance of a note that is merely adjacent rather than relevant."
    }
  ]
}