Router: a judge that returns nothing is a loud no-op, never a flat ranking #211

Open
opened 2026-10-21 01:43:33 +00:00 by teahubxyz · 4 comments
Owner

Today's Astra run (log #57, comment 2287, 2026-10-20 13:14) got no usable JSON back from the seat: (no note), 37 of 37 candidates unjudged, every score 1.911, and on that flat table it still sent #66, #198 and #202 to the cloud lane and promoted "start a new venture" to #1. A ranking where nothing was judged is not a ranking; the router queued on noise. Same class as #202 (a silent Object).

Changes, in .github/time/astra.py router_tick:

  • After parse(...), if out.get("assess") is empty (or fewer than, say, a quarter of the candidates are judged), post 🔭 Astra: the judge returned no assessments (<first 200 chars of the raw reply, or "empty">); nothing queued or sent today. with meta {"beat":"astra","router":mode,"queued":0,"unjudged":true} and return. Do not rank, pick, send to the cloud lane, or create cards.
  • Keep the raw reply's length and the stop reason in the run log, so a truncated reply (output limit) is visible as the cause.
  • Partial judging (some assessed, some not) keeps today's behaviour: unjudged rows score at value 3 / size M and the note says how many.
  • Test: a parse result with no assess posts the no-op comment and labels nothing; a result with 30 of 37 judged still ranks.

Follow-up, not this card: chunk the judging (batches of ~12 candidates per call) so one reply never has to carry the whole board, and let a cheap typed reader (Clef on Workers AI, now on the paid plan) do a first pass so the opus seat only judges the top band.

DONE: PR open with the above and test_time.py green in CI; the next run where the seat answers with no assess posts the no-op line and queues nothing.

Today's Astra run (log #57, comment 2287, 2026-10-20 13:14) got no usable JSON back from the seat: `(no note)`, 37 of 37 candidates unjudged, every score 1.911, and on that flat table it still sent #66, #198 and #202 to the cloud lane and promoted "start a new venture" to #1. A ranking where nothing was judged is not a ranking; the router queued on noise. Same class as #202 (a silent Object). Changes, in `.github/time/astra.py` `router_tick`: - After `parse(...)`, if `out.get("assess")` is empty (or fewer than, say, a quarter of the candidates are judged), post `🔭 Astra: the judge returned no assessments (<first 200 chars of the raw reply, or "empty">); nothing queued or sent today.` with meta `{"beat":"astra","router":mode,"queued":0,"unjudged":true}` and return. Do not rank, pick, send to the cloud lane, or create cards. - Keep the raw reply's length and the stop reason in the run log, so a truncated reply (output limit) is visible as the cause. - Partial judging (some assessed, some not) keeps today's behaviour: unjudged rows score at value 3 / size M and the note says how many. - Test: a `parse` result with no `assess` posts the no-op comment and labels nothing; a result with 30 of 37 judged still ranks. Follow-up, not this card: chunk the judging (batches of ~12 candidates per call) so one reply never has to carry the whole board, and let a cheap typed reader (Clef on Workers AI, now on the paid plan) do a first pass so the opus seat only judges the top band. DONE: PR open with the above and test_time.py green in CI; the next run where the seat answers with no `assess` posts the no-op line and queues nothing.
teahubxyz added the ready label 2026-10-21 01:43:33 +00:00

⏱️ Time: Claude Sonnet (Anthropic) is working on this now, on branch time/card-211.

⏱️ Time: Claude Sonnet (Anthropic) is working on this now, on branch `time/card-211`. <!-- time {"beat":"dispatched","launched":true,"worker":"anthropic"} -->

⏱️ Time: sonnet opened PR #212 (10 steps, claude-sonnet-5-5). Review and landing follow on the next beat.

⏱️ Time: `sonnet` opened PR #212 (10 steps, claude-sonnet-5-5). Review and landing follow on the next beat. <!-- time {"beat":"dispatched","worker":"anthropic","pr":212,"steps":10,"seat":"sonnet","model":"claude-sonnet-5-5"} -->

⏱️ Time: PR #212 is up for this card. haiku reviewed it on the PR.

⏱️ Time: PR #212 is up for this card. `haiku` reviewed it on the PR. <!-- time {"beat":"pr","pr":212,"launched":false,"seat":"haiku","as":"seat","review":"venice","seats":["haiku"]} -->

⏱️ Time: the reviewer objected on PR #212. Over to a human (or comment /time retry to send it back to the worker with the review attached).

⏱️ Time: the reviewer objected on PR #212. Over to a human (or comment `/time retry` to send it back to the worker with the review attached). <!-- time {"beat":"needs-human","pr":212} -->
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: teahubxyz/time#211