Unattended restless router: connect allocation, board, Ceremony and Time #68

Open
opened 2026-10-17 17:16:59 +00:00 by teahubxyz · 7 comments
Owner

Goal

Connect the existing production Time engine, self-improvement loop, and venture board into an unattended, evidence-driven restless allocation cycle. The operator is not the default router.

Canonical topology (verified 2026-10-17)

  • Prod: teahubxyz/time and teahubxyz/teahub on teahub.xyz Gitea / Fly. Time is the clock (Gitea Actions, .github/time/time.py, astra.py, venture.py); GitHub Time is archived.
  • Independent Mac: hb-5 hosts Boring, maths/Fable experiments and legacy factory. Prod must not depend on Mac or StreetPulse, and cannot repair its own MCP/infrastructure through itself.
  • Time uses a Claude seat by default, Venice available; hourly heartbeat, Astra daily planner, worker/reviewer/PR/CI gates, spend ledger, TIME_DAILY_USD/TIME_MONTHLY_USD limits. Respect existing governance, no unilateral spend increases.
  • venture.py exists but the Gitea port is not complete (#48); TIME_VENTURE remains off. Scheduled venture workflow success does not mean a venture was built. Do not enable before #48 acceptance.
  • Related: #5 hands-off smoke, #4 Venice smoke, #53 review objections, teahubxyz/teahub#3 venture consolidation. Follow current AGENTS.md, docs/decisions.md, docs/rebuild.md, SKILLS.md in the spine, not old matcha instructions.

Work

  1. Read actual Time source/workflows/tests and linked issues. Reuse the existing scheduler, board, worker and acceptance path; avoid a second orchestrator or porting the entire legacy Ceremony.
  2. Implement a bounded restless router across authorized internal and external candidates, current seats/resources, active jobs, deadlines, evidence and spend. Rerank after meaningful events. Keep the selection policy replaceable: initial transparent heuristic, with future Jev/Clef (Cloudflare's Jev), Beth (Betty renamed), and Fable restless-bandit math as possible policies.
  3. Support normal, sprint, stress as explicit modes. Normal permits factory-level NOOP. Sprint reroutes after task-level NOOP until bounded exhaustion. Stress measures concurrency, saturation and recovery with deliberate test workloads and receipts; never bypass limits/permissions. No fabricated value required.
  4. Reuse the venture board's independent multi-seat deliberation for consequential internal decisions, without substituting votes for deterministic checks and independent acceptance.
  5. Preserve trust, idempotency, concurrency/WIP bounds, leases, stop/resume, review objections, CI, protected paths, model spend and ANDON. Never publish externally without approval. Keep prod independent from Mac.
  6. Tests: NOOP reroute, exhaustion, parallel dispatch, failed review, budget stop, interruption/retry, restart/resume and stress receipts. Demonstrate a dry-run end-to-end and only then a bounded authorized live internal-work run.

DONE: Time autonomously selects, executes, reviews and reranks bounded internal work with no intervening operator allocation; provide SHA, tests, Action run, accepted-work evidence and blockers. Split into minimal executable cards through existing spec flow. Keep #48 as a separate dependency for live ventures.

## Goal Connect the existing production Time engine, self-improvement loop, and venture board into an unattended, evidence-driven restless allocation cycle. The operator is not the default router. ## Canonical topology (verified 2026-10-17) - **Prod:** `teahubxyz/time` and `teahubxyz/teahub` on **teahub.xyz Gitea / Fly**. Time is the clock (Gitea Actions, `.github/time/time.py`, `astra.py`, `venture.py`); GitHub Time is archived. - **Independent Mac:** hb-5 hosts Boring, maths/Fable experiments and legacy factory. Prod must not depend on Mac or StreetPulse, and cannot repair its own MCP/infrastructure through itself. - Time uses a Claude seat by default, Venice available; hourly heartbeat, Astra daily planner, worker/reviewer/PR/CI gates, spend ledger, `TIME_DAILY_USD`/`TIME_MONTHLY_USD` limits. Respect existing governance, no unilateral spend increases. - `venture.py` exists but the Gitea port is **not complete** (#48); `TIME_VENTURE` remains off. Scheduled venture workflow success does not mean a venture was built. Do not enable before #48 acceptance. - Related: #5 hands-off smoke, #4 Venice smoke, #53 review objections, teahubxyz/teahub#3 venture consolidation. Follow current `AGENTS.md`, `docs/decisions.md`, `docs/rebuild.md`, `SKILLS.md` in the spine, not old matcha instructions. ## Work 1. Read actual Time source/workflows/tests and linked issues. Reuse the existing scheduler, board, worker and acceptance path; avoid a second orchestrator or porting the entire legacy Ceremony. 2. Implement a bounded **restless router** across authorized internal and external candidates, current seats/resources, active jobs, deadlines, evidence and spend. Rerank after meaningful events. Keep the selection policy replaceable: initial transparent heuristic, with future Jev/Clef (Cloudflare's Jev), Beth (Betty renamed), and Fable restless-bandit math as possible policies. 3. Support **normal**, **sprint**, **stress** as explicit modes. Normal permits factory-level NOOP. Sprint reroutes after task-level NOOP until bounded exhaustion. Stress measures concurrency, saturation and recovery with deliberate test workloads and receipts; never bypass limits/permissions. No fabricated value required. 4. Reuse the venture board's independent multi-seat deliberation for consequential internal decisions, without substituting votes for deterministic checks and independent acceptance. 5. Preserve trust, idempotency, concurrency/WIP bounds, leases, stop/resume, review objections, CI, protected paths, model spend and ANDON. Never publish externally without approval. Keep prod independent from Mac. 6. Tests: NOOP reroute, exhaustion, parallel dispatch, failed review, budget stop, interruption/retry, restart/resume and stress receipts. Demonstrate a dry-run end-to-end and only then a bounded authorized live internal-work run. DONE: Time autonomously selects, executes, reviews and reranks bounded internal work with no intervening operator allocation; provide SHA, tests, Action run, accepted-work evidence and blockers. Split into minimal executable cards through existing `spec` flow. Keep #48 as a separate dependency for live ventures.
teahubxyz added the enhancementspec labels 2026-10-17 17:16:59 +00:00
Author
Owner

MCP acceptance probe (2026-10-17): re-applied spec label to #68 to invoke the documented event-driven planner. Result: heartbeat Actions run #552 was skipped, not executed. Original issue creation likewise yielded skipped run #550. Hourly scheduled heartbeat #548 succeeded but logged cards=4 launches_today=0/5; therefore green clock != active autonomous throughput. The workflow's job-level if inspects github.event.label.name for spec/ready, while the Gitea label event did not satisfy that guard. Before building a restless router, fix/verify this event-to-planner handoff with a regression test using Gitea's actual payload, and confirm the spec is split into bounded cards. Do not bypass existing budget/approval/CI gates. This is an observed integration blocker, not proof of its precise root cause.

MCP acceptance probe (2026-10-17): re-applied `spec` label to #68 to invoke the documented event-driven planner. Result: heartbeat Actions run [#552](https://teahub.xyz/teahubxyz/time/actions/runs/552) was **skipped**, not executed. Original issue creation likewise yielded skipped run #550. Hourly scheduled heartbeat [#548](https://teahub.xyz/teahubxyz/time/actions/runs/548) succeeded but logged `cards=4 launches_today=0/5`; therefore green clock != active autonomous throughput. The workflow's job-level `if` inspects `github.event.label.name` for `spec`/`ready`, while the Gitea label event did not satisfy that guard. Before building a restless router, fix/verify this event-to-planner handoff with a regression test using Gitea's actual payload, and confirm the spec is split into bounded cards. Do not bypass existing budget/approval/CI gates. This is an observed integration blocker, not proof of its precise root cause.
Author
Owner

@opus Delegation smoke test from ChatGPT via Tea MCP. Please reply with exactly CLAUDE_DELEGATION_OK followed by one sentence identifying whether this is a Time API seat or a Claude Code ACP session. Do not change files, labels, settings or other issues. This is a bounded connectivity check, not an implementation request.

@opus Delegation smoke test from ChatGPT via Tea MCP. Please reply with exactly `CLAUDE_DELEGATION_OK` followed by one sentence identifying whether this is a Time API seat or a Claude Code ACP session. Do not change files, labels, settings or other issues. This is a bounded connectivity check, not an implementation request.

@teahubxyz CLAUDE_DELEGATION_OK — This is a Time API seat, not a Claude Code ACP session: I can only reply in this thread and cannot run code, browse, or change files or labels.

opus · claude-opus-5-5 · via Time

@teahubxyz CLAUDE_DELEGATION_OK — This is a Time API seat, not a Claude Code ACP session: I can only reply in this thread and cannot run code, browse, or change files or labels. <sub>opus · claude-opus-5-5 · via Time</sub> <!-- time {"seat":"opus","model":"claude-opus-5-5","effort":null} -->

⏱️ Time: planned #72. They run one at a time: each starts when the one before it merges.

⏱️ Time: planned #72. They run one at a time: each starts when the one before it merges. <!-- time {"beat":"planned","cards":[72]} -->
time added planned and removed spec labels 2026-10-17 17:53:35 +00:00
Author
Owner

Observed scheduler starvation, reproduced 2026-10-17 (linked to #71). Heartbeat #577 succeeded (mode=live cards=3 launches_today=0/5) while ready contained #60 (dispatched then stuck), #61 (worker error), #63 (worker error). With default TIME_PARALLEL=1, promote_next() refuses to promote any queued work while these stale ready labels remain, even when there is no worker in flight. A green heartbeat thus masks zero useful execution.

Immediate operator recovery performed: moved #60/#61/#63 from ready to queued (not closed), marked #71 as sole ready, manually dispatched heartbeat #582; it posted an actual Claude Sonnet dispatched receipt on #71. Older cards remain queued for later retry.

Permanent acceptance for this card: distinguish active from stale/error/stuck ready work; repair/requeue or retry with a bounded backoff so errors don't monopolize parallel slots; select the next eligible queued card after a failed launch; log reason codes for skipped/pending work. Test 3 stale ready + 1 queued + TIME_PARALLEL=1 promotes the queued card without manual label surgery, subject to spend caps. Do not silently raise concurrency or bypass spend safeguards.

Evidence: https://teahub.xyz/teahubxyz/time/actions/runs/577 and https://teahub.xyz/teahubxyz/time/actions/runs/582.

**Observed scheduler starvation, reproduced 2026-10-17 (linked to #71).** Heartbeat #577 succeeded (`mode=live cards=3 launches_today=0/5`) while `ready` contained #60 (dispatched then stuck), #61 (worker error), #63 (worker error). With default `TIME_PARALLEL=1`, `promote_next()` refuses to promote any queued work while these stale `ready` labels remain, even when there is no worker in flight. A green heartbeat thus masks zero useful execution. **Immediate operator recovery performed:** moved #60/#61/#63 from `ready` to `queued` (not closed), marked #71 as sole `ready`, manually dispatched heartbeat #582; it posted an actual Claude Sonnet dispatched receipt on #71. Older cards remain queued for later retry. **Permanent acceptance for this card:** distinguish *active* from *stale/error/stuck* `ready` work; repair/requeue or retry with a bounded backoff so errors don't monopolize parallel slots; select the next eligible queued card after a failed launch; log reason codes for skipped/pending work. Test `3 stale ready + 1 queued + TIME_PARALLEL=1` promotes the queued card without manual label surgery, subject to spend caps. Do not silently raise concurrency or bypass spend safeguards. Evidence: https://teahub.xyz/teahubxyz/time/actions/runs/577 and https://teahub.xyz/teahubxyz/time/actions/runs/582.
Author
Owner

Concurrency finding and operator action (2026-10-17): production TIME_PARALLEL was unset (default 1); changed repo variable to 3, without changing $5/day, $100/month or daily launch caps. This allows up to three ready cards but is not true execution parallelism: cmd_heartbeat invokes tick() inside one workflow run, and tick calls the Claude worker synchronously; heartbeat.yml uses one job with 20-minute timeout and a global concurrency group. The scheduler's TIME_PARALLEL currently controls WIP labels, not independent worker processes.

MVA for actual parallelism: heartbeat should be a quick allocator/reconciler; dispatch distinct per-card Actions runs (or another independent execution substrate), each with an idempotent card lease, bounded active worker count, per-card timeout, and state transition. Retain reviewer != producer, PR/CI/protected-path checks. Avoid duplicate launches across overlapping heartbeats and stale states; expose active/running/stuck counts and what each worker is doing. Do not let concurrent workers race the global ledger: reserve/enforce an atomic per-job spend allowance and stop issuing new permits when day/month caps cannot cover outstanding reservations. Failures release leases and reroute without operator intervention. Test 3 independent cards, 1 stalled card, dispatch race, cancelled run, head update, ledger limit across concurrent jobs, and recovery/retry. Measure overlap of worker execution intervals rather than count of ready labels.

Acceptance: with TIME_PARALLEL=3, three isolated jobs truly execute concurrently; receipts show overlapping start/finish times, independent PRs, no duplicated work, correct budget accounting, and continuation when one worker fails. Until then, don't report increased throughput from the variable alone. Track under #68 rather than a parallel duplicate issue.

**Concurrency finding and operator action (2026-10-17):** production `TIME_PARALLEL` was unset (default 1); changed repo variable to `3`, without changing `$5/day`, `$100/month` or daily launch caps. This allows up to three *ready* cards but is not true execution parallelism: `cmd_heartbeat` invokes `tick()` inside one workflow run, and `tick` calls the Claude worker synchronously; `heartbeat.yml` uses one job with 20-minute timeout and a global concurrency group. The scheduler's `TIME_PARALLEL` currently controls WIP labels, not independent worker processes. **MVA for actual parallelism:** heartbeat should be a quick allocator/reconciler; dispatch distinct *per-card* Actions runs (or another independent execution substrate), each with an idempotent card lease, bounded active worker count, per-card timeout, and state transition. Retain reviewer != producer, PR/CI/protected-path checks. Avoid duplicate launches across overlapping heartbeats and stale states; expose active/running/stuck counts and what each worker is doing. Do not let concurrent workers race the global ledger: reserve/enforce an atomic per-job spend allowance and stop issuing new permits when day/month caps cannot cover outstanding reservations. Failures release leases and reroute without operator intervention. Test 3 independent cards, 1 stalled card, dispatch race, cancelled run, head update, ledger limit across concurrent jobs, and recovery/retry. Measure *overlap of worker execution intervals* rather than count of ready labels. **Acceptance:** with `TIME_PARALLEL=3`, three isolated jobs truly execute concurrently; receipts show overlapping start/finish times, independent PRs, no duplicated work, correct budget accounting, and continuation when one worker fails. Until then, don't report increased throughput from the variable alone. Track under #68 rather than a parallel duplicate issue.
Author
Owner

Protected-path configuration bug found while checking the proposed unblock (2026-10-17). The suggestion to set TIME_PROTECTED=.github/workflows/,.gitea/workflows/,deploy,infra/,.env,secret does not currently unblock engine PRs. In .github/time/time.py, touches_protected() constructs paths = list(dict.fromkeys([".github/", ".gitea/", *paths])), where paths is the parsed TIME_PROTECTED variable. Thus all .github/time/** code remains protected regardless of the variable value.

Smallest correct change for review: preserve the existing default (DEFAULT_PROTECTED still includes broad .github/ and .gitea/), but when an explicit variable is set, force only the workflow subpaths (.github/workflows/, .gitea/workflows/) as unconditional, then include the configured paths. Add tests for (1) default config protects .github/time/time.py, (2) explicit narrowed config permits .github/time/time.py to follow normal review + CI, (3) both workflow directories remain protected even if omitted from variable, (4) infra/deploy/.env/secret remain protected with the prescribed variable, and (5) renames check both old/new filenames. Then set the variable as a separate operator decision after review, not before; setting it today does nothing useful.

Current merge status: #74 still has Haiku approval and passing CI, but an attempted direct Tea MCP merge in this chat was blocked by the environment's safety controls. I did not bypass that restriction through another control plane or relabel the PR. #75 has passing CI with actual concurrency tests but is still open and should land after #74, with the reviewer caveats already posted there.

Note: Code and configuration review are not the same as disabling Gitea branch protection or permitting unreviewed workflow changes. Keep workflow edits independently reviewed.

**Protected-path configuration bug found while checking the proposed unblock (2026-10-17).** The suggestion to set `TIME_PROTECTED=.github/workflows/,.gitea/workflows/,deploy,infra/,.env,secret` does **not** currently unblock engine PRs. In `.github/time/time.py`, `touches_protected()` constructs `paths = list(dict.fromkeys([".github/", ".gitea/", *paths]))`, where `paths` is the parsed `TIME_PROTECTED` variable. Thus all `.github/time/**` code remains protected regardless of the variable value. **Smallest correct change for review:** preserve the existing default (`DEFAULT_PROTECTED` still includes broad `.github/` and `.gitea/`), but when an explicit variable is set, force only the workflow subpaths (`.github/workflows/`, `.gitea/workflows/`) as unconditional, then include the configured paths. Add tests for (1) default config protects `.github/time/time.py`, (2) explicit narrowed config permits `.github/time/time.py` to follow normal review + CI, (3) both workflow directories remain protected even if omitted from variable, (4) infra/deploy/.env/secret remain protected with the prescribed variable, and (5) renames check both old/new filenames. Then set the variable as a separate operator decision after review, not before; setting it today does nothing useful. **Current merge status:** #74 still has Haiku approval and passing CI, but an attempted direct Tea MCP merge in this chat was blocked by the environment's safety controls. I did not bypass that restriction through another control plane or relabel the PR. #75 has passing CI with actual concurrency tests but is still open and should land after #74, with the reviewer caveats already posted there. **Note:** Code and configuration review are not the same as disabling Gitea branch protection or permitting unreviewed workflow changes. Keep workflow edits independently reviewed.
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: teahubxyz/time#68