labs/aicharts: decision confidences + per-request floor for calibration #41

Merged
teahubxyz merged 1 commits from labs/aicharts-calibrate into main 2026-10-13 15:31:06 +00:00
Owner

Live findings: typesafe/jev fails with 2021: Insufficient AI Gateway credits (TypeSafe bills through prepaid Cloudflare credits); Clef-flash works at ~0.4–0.5 s and got both probes exactly right, but at 0.21–0.25 confidence, so FLOOR=0.5 handed both to Llama. This exposes per-question confidences and the weakest question, lets a request set its own floor, and fixes bench.py's user agent (Cloudflare 403s Python's default).

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDkN7ukK1kXkqov8FaZxva


Generated by Claude Code

Live findings: `typesafe/jev` fails with `2021: Insufficient AI Gateway credits` (TypeSafe bills through prepaid Cloudflare credits); Clef-flash works at ~0.4–0.5 s and got both probes exactly right, but at 0.21–0.25 confidence, so FLOOR=0.5 handed both to Llama. This exposes per-question confidences and the weakest question, lets a request set its own floor, and fixes bench.py's user agent (Cloudflare 403s Python's default). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VDkN7ukK1kXkqov8FaZxva --- _Generated by [Claude Code](https://claude.ai/code/session_01VDkN7ukK1kXkqov8FaZxva)_
Sign in to join this conversation.