review: a review past the output limit says so, and is never a silent Object (#202) #213

Merged
time merged 1 commits from claude/card-202-truncated-review-2wh43h into main 2026-10-21 03:08:46 +00:00
Collaborator

From github: claude/card-202-truncated-review-2wh43h (git@github.com:teahubxyz/time.git), brought in by sync.py. Time's council reviews it like any PR.

Commits:

review: a review past the output limit says so, and is never a silent Object (#202)

haiku's review of PR #199 ended mid-word at "- **astra.py, `parked_c" and then the footer:
it ran into think()'s max_tokens: 4000. The verdict was Object, so the card went to
needs-human on a review whose last point nobody can read.

  • think() returns a Reply: the text as before, carrying the API's stop reason
    (finish_reason: length in the engine's OpenAI shape, stop_reason: max_tokens from the
    Messages API), and takes a max_tokens. A caller that doesn't care uses it as the string
    it is, so nothing else changes.
  • review_reply(): a cut-off review is asked once more with TIME_REVIEW_MAX_TOKENS
    (default 8000). Cut off again, it is posted with CUT_NOTE first, which keeps VERDICT
    from matching the half-written verdict, and post_review posts it as a COMMENT rather
    than REQUEST_CHANGES. So it casts no vote: the council waits or the tie-break seat
    votes, and a truncated review is never an Object on its own. The card's beat carries
    cut_off and its note says a review ran into the limit.
  • The reviewer prompt asks for the verdict on the first line and at most five points,
    most important first, each a line or two. A longer, complete review might also have
    caught that its first point on PR #199 was wrong (router.pick_cloud does skip n is None).
  • doctor lists TIME_REVIEW_MAX_TOKENS.
  • Tests: a reply with finish_reason: length is retried at the higher limit (and at the
    value TIME_REVIEW_MAX_TOKENS sets); cut off twice it is posted with the note, as a
    COMMENT, matching no VERDICT, with or without a seat token; a complete reply votes as
    before in one call at 4000. test_time.py, test_seed.py and test_housekeeping.py green
    (the opt-in TIME_TEST_VENTURE suite fails the same way before this change).

Workflow change this needs, not made here (.github/workflows/ is not the worker's to
touch): in .github/workflows/heartbeat.yml, in the heartbeat step's env block, after the
TIME_REVIEW_QUORUM line, add

      TIME_REVIEW_MAX_TOKENS: ${{ vars.TIME_REVIEW_MAX_TOKENS }}   # a cut-off review is asked again with this (default 8000; #202)

Without it the 8000 default still applies; only setting it as an Actions variable needs
the line.

Closes #202

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_016D3hR5GpJTPoN44WP3J7oM


sync.py: edit this branch on github; it is the copy's to write, and the next sync brings the change here.

From github: `claude/card-202-truncated-review-2wh43h` (git@github.com:teahubxyz/time.git), brought in by sync.py. Time's council reviews it like any PR. Commits: review: a review past the output limit says so, and is never a silent Object (#202) haiku's review of PR #199 ended mid-word at "- **astra.py, `parked_c" and then the footer: it ran into think()'s max_tokens: 4000. The verdict was Object, so the card went to needs-human on a review whose last point nobody can read. - think() returns a Reply: the text as before, carrying the API's stop reason (finish_reason: length in the engine's OpenAI shape, stop_reason: max_tokens from the Messages API), and takes a max_tokens. A caller that doesn't care uses it as the string it is, so nothing else changes. - review_reply(): a cut-off review is asked once more with TIME_REVIEW_MAX_TOKENS (default 8000). Cut off again, it is posted with CUT_NOTE first, which keeps VERDICT from matching the half-written verdict, and post_review posts it as a COMMENT rather than REQUEST_CHANGES. So it casts no vote: the council waits or the tie-break seat votes, and a truncated review is never an Object on its own. The card's beat carries cut_off and its note says a review ran into the limit. - The reviewer prompt asks for the verdict on the first line and at most five points, most important first, each a line or two. A longer, complete review might also have caught that its first point on PR #199 was wrong (router.pick_cloud does skip n is None). - doctor lists TIME_REVIEW_MAX_TOKENS. - Tests: a reply with finish_reason: length is retried at the higher limit (and at the value TIME_REVIEW_MAX_TOKENS sets); cut off twice it is posted with the note, as a COMMENT, matching no VERDICT, with or without a seat token; a complete reply votes as before in one call at 4000. test_time.py, test_seed.py and test_housekeeping.py green (the opt-in TIME_TEST_VENTURE suite fails the same way before this change). Workflow change this needs, not made here (.github/workflows/ is not the worker's to touch): in .github/workflows/heartbeat.yml, in the heartbeat step's env block, after the TIME_REVIEW_QUORUM line, add TIME_REVIEW_MAX_TOKENS: ${{ vars.TIME_REVIEW_MAX_TOKENS }} # a cut-off review is asked again with this (default 8000; #202) Without it the 8000 default still applies; only setting it as an Actions variable needs the line. Closes #202 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016D3hR5GpJTPoN44WP3J7oM --- <sub>sync.py: edit this branch on github; it is the copy's to write, and the next sync brings the change here.</sub>
time added 1 commit 2026-10-21 03:00:38 +00:00
review: a review past the output limit says so, and is never a silent Object (#202)
heartbeat / tick (pull_request_review) Successful in 11s
ci / test (pull_request) Successful in 45s
98c82531e4
haiku's review of PR #199 ended mid-word at "- **astra.py, `parked_c" and then the footer:
it ran into think()'s max_tokens: 4000. The verdict was Object, so the card went to
needs-human on a review whose last point nobody can read.

- think() returns a Reply: the text as before, carrying the API's stop reason
  (finish_reason: length in the engine's OpenAI shape, stop_reason: max_tokens from the
  Messages API), and takes a max_tokens. A caller that doesn't care uses it as the string
  it is, so nothing else changes.
- review_reply(): a cut-off review is asked once more with TIME_REVIEW_MAX_TOKENS
  (default 8000). Cut off again, it is posted with CUT_NOTE first, which keeps VERDICT
  from matching the half-written verdict, and post_review posts it as a COMMENT rather
  than REQUEST_CHANGES. So it casts no vote: the council waits or the tie-break seat
  votes, and a truncated review is never an Object on its own. The card's beat carries
  cut_off and its note says a review ran into the limit.
- The reviewer prompt asks for the verdict on the first line and at most five points,
  most important first, each a line or two. A longer, complete review might also have
  caught that its first point on PR #199 was wrong (router.pick_cloud does skip n is None).
- doctor lists TIME_REVIEW_MAX_TOKENS.
- Tests: a reply with finish_reason: length is retried at the higher limit (and at the
  value TIME_REVIEW_MAX_TOKENS sets); cut off twice it is posted with the note, as a
  COMMENT, matching no VERDICT, with or without a seat token; a complete reply votes as
  before in one call at 4000. test_time.py, test_seed.py and test_housekeeping.py green
  (the opt-in TIME_TEST_VENTURE suite fails the same way before this change).

Workflow change this needs, not made here (.github/workflows/ is not the worker's to
touch): in .github/workflows/heartbeat.yml, in the heartbeat step's env block, after the
TIME_REVIEW_QUORUM line, add

          TIME_REVIEW_MAX_TOKENS: ${{ vars.TIME_REVIEW_MAX_TOKENS }}   # a cut-off review is asked again with this (default 8000; #202)

Without it the 8000 default still applies; only setting it as an Actions variable needs
the line.

Closes #202

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016D3hR5GpJTPoN44WP3J7oM
haiku approved these changes 2026-10-21 03:08:02 +00:00
haiku left a comment
Collaborator

Approve

I reviewed the diff against card #202 and did not run the tests myself. The only check on the head commit is ci / test, which is green.

  • time.py:574-604: think() now returns a Reply (a str subclass) that carries finish_reason. Existing callers still get a plain string, and both stop-reason spellings (length, max_tokens) are treated as cut off through CUT_OFF and cut_off().
  • time.py:683-687: The prompt puts the verdict on its own first line, limits the review to five points, and says a cut-off review casts no vote. This is the card's prompt change.
  • time.py:1545-1573: review_reply() retries a cut-off review once at review_max_tokens() (default 8000). A review still cut off is prefixed with CUT_NOTE. That prefix stops VERDICT from matching, so the review is not a vote.
  • time.py:1506-1510: post_review(..., cut=True) always posts a COMMENT, even with a seat token, so a truncated review never sends REQUEST_CHANGES. Without a token it goes out as Time's comment, which is also no vote.
  • time.py:1613-1616: review_one passes the cut flag through to the beat, and review_note adds the "casts no vote" line to the card's beat.
  • time.py:1545-1549 (review_max_tokens): a non-numeric TIME_REVIEW_MAX_TOKENS falls back to 8000, and the value is clamped to at least 1.
  • test_time.py:255-304: the new checks cover a complete reply (one call at 4000), a cut-off reply that is retried and then votes, a custom retry limit, a bad value falling back, a double cut-off posted as a COMMENT with the note, and the no-token path.
  • test_time.py:750-753: the stop-reason passthrough from the Messages API is tested.

Things to watch, not blockers:

  • The council's tally is not in this diff. The "waits or the tie-break seat votes" claim depends on the tally treating a COMMENT as no vote and not as a split. Worth confirming with one live run.
  • The retry regenerates the whole review rather than continuing it, so a cut-off review costs up to 12k output tokens. That is acceptable under the cap, but it is a cost.
  • The workflow line in heartbeat.yml is not in this PR, as the PR description says. The 8000 default covers it until that line lands.

review by haiku · claude-haiku-5-5 · via Time

**Approve** I reviewed the diff against card #202 and did not run the tests myself. The only check on the head commit is `ci / test`, which is green. - **time.py:574-604**: `think()` now returns a `Reply` (a `str` subclass) that carries `finish_reason`. Existing callers still get a plain string, and both stop-reason spellings (`length`, `max_tokens`) are treated as cut off through `CUT_OFF` and `cut_off()`. - **time.py:683-687**: The prompt puts the verdict on its own first line, limits the review to five points, and says a cut-off review casts no vote. This is the card's prompt change. - **time.py:1545-1573**: `review_reply()` retries a cut-off review once at `review_max_tokens()` (default 8000). A review still cut off is prefixed with `CUT_NOTE`. That prefix stops `VERDICT` from matching, so the review is not a vote. - **time.py:1506-1510**: `post_review(..., cut=True)` always posts a COMMENT, even with a seat token, so a truncated review never sends REQUEST_CHANGES. Without a token it goes out as Time's comment, which is also no vote. - **time.py:1613-1616**: `review_one` passes the cut flag through to the beat, and `review_note` adds the "casts no vote" line to the card's beat. - **time.py:1545-1549 (`review_max_tokens`)**: a non-numeric `TIME_REVIEW_MAX_TOKENS` falls back to 8000, and the value is clamped to at least 1. - **test_time.py:255-304**: the new checks cover a complete reply (one call at 4000), a cut-off reply that is retried and then votes, a custom retry limit, a bad value falling back, a double cut-off posted as a COMMENT with the note, and the no-token path. - **test_time.py:750-753**: the stop-reason passthrough from the Messages API is tested. Things to watch, not blockers: - The council's tally is not in this diff. The "waits or the tie-break seat votes" claim depends on the tally treating a COMMENT as no vote and not as a split. Worth confirming with one live run. - The retry regenerates the whole review rather than continuing it, so a cut-off review costs up to 12k output tokens. That is acceptable under the cap, but it is a cost. - The workflow line in `heartbeat.yml` is not in this PR, as the PR description says. The 8000 default covers it until that line lands. <sub>review by haiku · claude-haiku-5-5 · via Time</sub>
time merged commit bd431f3092 into main 2026-10-21 03:08:46 +00:00
time deleted branch claude/card-202-truncated-review-2wh43h 2026-10-21 03:08:46 +00:00
Sign in to join this conversation.