Pimp My IDE / Garage dispatch
Back to garage
September 27, 2026 | extraction / configuration / honest unknowns

Unknown is a value.

When software cannot prove a field, make it return null, invalid, or unknown. A plausible answer is not a fallback.

The take. Give absence and uncertainty their own states. Keep them distinct through prompts, schemas, interfaces, policy checks, and receipts.
Set the null detentsRead the extraction test

One sentence changed the run.

Earn an Honest Dollar tested 16 models on synthetic web-extraction traps. Each pair of pages differed by one row. One page contained the requested value and the other did not. Both pages included a tempting decoy.[1]

Without the instruction "Use null for any field whose value is not on the page. Do not guess," the models supplied 405 made-up values across 573 missing fields. With it, they supplied 116 across 574. On the old-price trap, all 16 models copied the decoy without the sentence. One did so with it.

This was one run on synthetic pages. The publisher runs a marketplace for agent services and has a commercial interest in extraction quality. The result is useful evidence for a testable prompt change, not a universal model ranking.

The cheap upgrade is not better prose. It is a real state for "the page did not say."

Null needs a route.

A null instruction helps only if every later step can preserve it. The response schema must accept null. Validation must distinguish a missing field from malformed output. The interface must show "absent" instead of an empty-looking cell. Business logic must decide whether to stop, ask, or continue.

The same rule applies outside extraction. GitHub's Copilot settings validator reports malformed JSON, unsupported configurations, and invalid team mappings. It points to the affected file and JSON path, then asks the operator to correct the file and check the validator again.[2] Invalid policy is not silently rounded into valid policy.

Unknown must stay unknown.

VS Code 1.139 proposes an optional expiresAfter value for authentication sessions. Providers recompute the remaining lifetime for cached sessions. They leave the property undefined when expiration is unknown.[3]

The proposal uses remaining lifetime instead of an absolute timestamp because the client and extension host can run on different machines with different clocks. That is careful interface design. It represents what the provider knows without inventing precision.

Design the abstain path before the answer path.

For each extracted or computed field, define three outcomes before calling the model: present with evidence, absent by inspection, and unresolved. Give each outcome a downstream action.

  • Present carries the value and its source span.
  • Absent carries null and the scope that was inspected.
  • Unresolved stops the sensitive action or routes the field to review.

The bench below builds that contract. It does not inspect a page. It makes the field states and follow-up routes explicit before an extractor gets the keys.

Interactive makeover / extraction contract

Null detent bench

Traditional purpose replaced: leave blank cells for the next step to interpret. Better version: set each field to present, absent, or verify, then copy a packet with an explicit action.

Set each field detent

Use Present only when the source contains the value. Use Absent when the inspected source does not. Use Verify when inspection is incomplete or ambiguous.

AuthorVERIFY
Author state
Current priceVERIFY
Current price state
Published dateVERIFY
Published date state

3 fields need verification.

Extraction contract3 NEED REVIEW

Copy the field routes

This packet records your handling rules. It does not claim that a page was inspected.

What completion means. Every field has a route. It does not mean the values are correct, the page was fetched, or a review passed.

Sources and limits

Open the source log
  1. Earn an Honest Dollar, "Calling the AI bluff: 'Do not guess' cut made-up fields from 70.7% to 20.2%", published and read September 27, 2026. The page reports one run across synthetic twin pages. Counts, costs, model names, and confidence intervals come from the publisher. The publisher also operates the marketplace promoted on the page.
  2. GitHub Changelog, "Enterprise managed settings in-product validator", published September 25 and read September 27, 2026. It documents the validator's error classes, file and JSON-path reporting, and recheck workflow.
  3. Visual Studio Code 1.139 release notes, proposed authentication-session expiration API, released September 23 and read September 27, 2026. It documents expiresAfter, clock-skew reasoning, cache recomputation, and undefined for unknown expiration.
  4. Hacker News discussion 49868753, verified through the Hacker News API and page on September 27, 2026. It is the discovery route for the extraction test, not evidence for the benchmark results.

Source boundary. The page combines one commercial publisher's synthetic benchmark, a GitHub product update, and a proposed VS Code API. The shared lesson about preserving unknown states is the garage's analysis.