Auditorpass

a step technicians keep reading two different ways is a procedure defect

Four technicians in eleven jobs stopped on the same step and each said, in their own words, that they did not know how clean is clean enough. That is not a hard job — it is a sentence in the procedure that does not say what it wants, and it is the case a paper checklist can never surface because paper has nowhere to put the objection.

The answer conformed to the contract and every assertion held.

modelgemini-3.5-flash

temperature0

scenarioscenarios/auditor/ambiguous-step-blocked-in-its-own-words.json

cassette7b4e32d2ca59d85f…

latency0.0s

tokens7,028

What the model was given

System instructionassembled from the contract schema, not hand-written

You are reading every job run against one procedure over a window of weeks, looking for defects in the PROCEDURE. The hardest judgement here, and the one you are for, is telling a broken form apart from a working one: a step that keeps failing because the machines are genuinely worn is the procedure doing its job, and reporting it as a defect would push a shop to loosen the rule that is catching real faults. A blocked step is your best evidence, because a technician who stopped and said why has written you a labelled defect report in their own words. Cite jobs and quote people; a finding with nothing behind it is an opinion, and this fleet does not ship opinions. Small numbers are not patterns — one job in forty is noise, and saying you do not have enough history is a correct and useful answer. If every instance traces to one person, that is training, not a procedure defect. Above all, you may say a bound is wrong; you may never say what the new bound should be. A figure has to come from the shop, and inventing one here would put a fabricated tolerance into a procedure by the back door.

Return JSON with these fields:
- mode (one of revise | no_defect | insufficient_history, required): revise when the procedure should change and you can show why. no_defect when you examined enough history and the procedure is behaving. insufficient_history when the window is too thin to distinguish a pattern from noise.
- understanding (string, required): What this procedure is and how it has actually been behaving over the window, in two sentences. Written every time so a wrong reading is visible before anyone acts on it.
- jobs_examined (integer, required): How many jobs you actually read. The denominator for every claim below.
- findings (array, required): Defects in the procedure. Empty unless mode is revise.
- considered_and_rejected (array, required): Patterns you noticed and decided were NOT procedure defects, each with the reason. A step failing often because the machines are worn belongs here, and so does anything traceable to a single technician. This list is how a reader knows you discriminated rather than reported everything that moved.

## The procedure under audit
{
  "key": "front-brake-service",
  "title": "Front brake service \u2014 Segway Xyber",
  "version": 3,
  "strictness": 2,
  "in service since": "2026-06-01"
}

## Its steps, as written
This is the text technicians are actually shown. Where a step is read two different ways by two people, the ambiguity is in these words.

[
  {
    "index": 1,
    "title": "Check the pads before you touch anything",
    "explanation": "If they are already past the limit the rest of the job changes.",
    "fields": [
      {
        "key": "pad_thickness",
        "kind": "measurement",
        "acceptance_rule": "within",
        "acceptance_min": 2,
        "acceptance_unit": "mm",
        "prompt": "Measure the thinnest pad and enter it"
      }
    ]
  },
  {
    "index": 2,
    "title": "Clean the caliper up",
    "explanation": "Contamination on the disc stops the job dead.",
    "fields": [
      {
        "key": "caliper_clean",
        "kind": "photo",
        "acceptance_rule": "must_show",
        "acceptance_description": "the caliper clean",
        "prompt": "Photograph the caliper once it is clean"
      }
    ]
  },
  {
    "index": 3,
    "title": "Fit the new pads and torque the caliper bolts",
    "explanation": "An under-torqued caliper bolt backs out under braking heat.",
    "fields": [
      {
        "key": "caliper_bolt_torque",
        "kind": "measurement",
        "acceptance_rule": "within",
        "acceptance_min": 6,
        "acceptance_max": 9,
        "acceptance_unit": "Nm",
        "prompt": "Torque both caliper bolts and record the reading"
      }
    ]
  },
  {
    "index": 4,
    "title": "Photograph the disc face for records",
    "explanation": "Kept for the file.",
    "fields": [
      {
        "key": "disc_face",
        "kind": "photo",
        "acceptance_rule": "must_show",
        "acceptance_description": "the disc face",
        "prompt": "Photograph the disc"
      }
    ]
  }
]

## The window you are reading
{
  "from": "2026-08-01",
  "to": "2026-08-20",
  "jobs in the window": 11,
  "note": "This is every job run against this procedure in the window, not a sample. A count you compute from it is the real count."
}

## Every job in the window
Each carries its step outcomes. A step that was not performed carries the reason the technician gave, in their words, and who gave it. Those sentences are the strongest evidence in this whole document: somebody stopped work and explained why, which is a defect report written by the person the procedure failed.

[
  {
    "id": "J-4401",
    "sealed_at": "2026-08-02",
    "asset_id": "XYBER-01",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "impossible",
        "reason": {
          "kind": "voice",
          "by": "Ana",
          "transcript": "clean enough for what? I've wiped it down but I don't know what you want me to prove here"
        },
        "blocker_kind": "other"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  },
  {
    "id": "J-4402",
    "sealed_at": "2026-08-03",
    "asset_id": "XYBER-02",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "performed"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  },
  {
    "id": "J-4403",
    "sealed_at": "2026-08-05",
    "asset_id": "XYBER-03",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "impossible",
        "reason": {
          "kind": "voice",
          "by": "Bruno",
          "transcript": "what counts as clean, there's always going to be some dust on it"
        },
        "blocker_kind": "other"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  },
  {
    "id": "J-4404",
    "sealed_at": "2026-08-06",
    "asset_id": "XYBER-04",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "performed"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  },
  {
    "id": "J-4405",
    "sealed_at": "2026-08-08",
    "asset_id": "XYBER-05",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "impossible",
        "reason": {
          "kind": "voice",
          "by": "Ana",
          "transcript": "again, I can't tell if this passes"
        },
        "blocker_kind": "other"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  },
  {
    "id": "J-4406",
    "sealed_at": "2026-08-09",
    "asset_id": "XYBER-06",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "performed"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  },
  {
    "id": "J-4407",
    "sealed_at": "2026-08-11",
    "asset_id": "XYBER-07",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "impossible",
        "reason": {
          "kind": "voice",
          "by": "Chen",
          "transcript": "is this about the disc or the caliper body? the photo shows both"
        },
        "blocker_kind": "other"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  },
  {
    "id": "J-4408",
    "sealed_at": "2026-08-13",
    "asset_id": "XYBER-08",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "performed"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  },
  {
    "id": "J-4409",
    "sealed_at": "2026-08-15",
    "asset_id": "XYBER-09",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "impossible",
        "reason": {
          "kind": "voice",
          "by": "Dev",
          "transcript": "no idea what good looks like on this one"
        },
        "blocker_kind": "other"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  },
  {
    "id": "J-4410",
    "sealed_at": "2026-08-17",
    "asset_id": "XYBER-10",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "performed"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  },
  {
    "id": "J-4411",
    "sealed_at": "2026-08-19",
    "asset_id": "XYBER-11",
    "steps": [
      {
        "index": 1,
        "title": "Check the pads before you touch anything",
        "status": "performed"
      },
      {
        "index": 2,
        "title": "Clean the caliper up",
        "status": "performed"
      },
      {
        "index": 3,
        "title": "Fit the new pads and torque the caliper bolts",
        "status": "performed"
      },
      {
        "index": 4,
        "title": "Photograph the disc face for records",
        "status": "performed"
      }
    ]
  }
]

What came backStructured output, not prose that was parsed afterwards: the contract schema is posted as the response schema, so the model returns this shape or fails.

{
  "mode": "revise",
  "understanding": "This procedure outlines the steps for performing a front brake service on the Segway Xyber, including pad inspection, caliper cleaning, pad installation, and disc documentation. While most steps are completed without issue, the caliper cleaning step is frequently blocked because technicians cannot determine what level of cleanliness is required or what the photo is meant to verify.",
  "jobs_examined": 11,
  "findings": [
    {
      "step_title": "Clean the caliper up",
      "field_key": "caliper_clean",
      "defect": "ambiguous_instruction",
      "what": "The instruction to clean the caliper and photograph it lacks objective standards, leaving technicians unable to verify if their cleaning is sufficient or what the photo is supposed to prove.",
      "jobs_cited": [
        "J-4401",
        "J-4403",
        "J-4405",
        "J-4407",
        "J-4409"
      ],
      "quotes": [
        "clean enough for what? I've wiped it down but I don't know what you want me to prove here",
        "what counts as clean, there's always going to be some dust on it",
        "again, I can't tell if this passes",
        "is this about the disc or the caliper body? the photo shows both",
        "no idea what good looks like on this one"
      ],
      "jobs_affected": 5,
      "proposed_revision": "The step needs to be revised to define clear, objective criteria for caliper cleanliness and specify exactly what the photo must capture. The shop must provide the specific cleanliness standards and reference images of acceptable versus unacceptable states.",
      "needs_the_shop": true,
      "confidence": 0.95
    }
  ],
  "considered_and_rejected": [
    "We considered whether the caliper bolt torque range of 6 to 9 Nm in Step 3 was problematic, but it was completed successfully in all 11 jobs without any technician complaints or failures, indicating the bound is currently workable."
  ]
}

What this scenario asserted

Assertions pin the part of an answer that decides something, never the whole object — exact-matching a rationale would fail the first time it was reworded, and a suite that cries wolf is abandoned inside a day. Where the reasoning is checked at all, it is to catch an agent that reached the right verdict for no reason.

	asserted	about	what happened
ok	equals	mode	is revise, want revise
ok	len_gte	findings	length 1, want >= 1
ok	mentions_any	findings.0.step_title	found ['clean', 'caliper']
ok	mentions_any	findings.0.defect	found ['ambiguous_instruction']
ok	mentions_any	findings.0.quotes	found ['clean enough', 'what counts as clean', 'good looks like', "can't tell"]