fix(ai): adaptive per-model token budget on truncation #78

Merged

kreativmonkey merged 3 commits from fix/ai-truncation-retry into main 2026-06-13 21:22:08 +02:00

Conversation 0 Commits 3 Files changed 7 +259 -3

kreativmonkey commented 2026-06-13 20:55:23 +02:00

Owner

Copy link

Problem

After #72, a recurring "recreate" AI job failed with:

{ "duration_ms": "25669", "error": "llm response truncated at token limit" }

The truncation detection works — but the generators returned ErrResponseTruncated straight to the caller, so the job failed. The ~22–25s durations of the successful jobs show the configured model is a reasoning model that occasionally spends its whole budget on hidden reasoning tokens and stops at finish_reason: "length" before finishing the JSON.

Fix (two clearly separated layers)

1. In-job recovery (fast). The voice-engine and campaign retry loops catch ErrResponseTruncated and double the token budget (escalateBudget, 2048 → 4096 → 8192, capped) and retry, so the current job still succeeds.

2. Cross-job learning (gentle, per model). Always starting at 2048 would make a heavy model pay for a truncated attempt + escalate-retry on every job. A process-wide budgetMemory adapts the starting budget per model — but only on a pattern, not a fluke:

• A single truncation changes nothing ("einmal ist keinmal").
• Only after a model truncates truncationsBeforeRaise (3) times does its starting budget creep up — and then by just one tokenBudgetStep (1024), not by doubling — capped at maxTokenBudget. The counter resets, so the next raise needs a fresh run of truncations.

So an isolated truncation is absorbed by the in-job retry and leaves the default alone; a genuinely heavy model slowly settles at a starting budget that stops it from looping. Model() was added to the Client interface to key the memory.

The memory is process-local (re-learns after a restart); persisting it across restarts is a possible follow-up. The 10-minute jobExecutionTimeout leaves ample room for the larger retries (each call ≈25s).

Tests (added, TDD)

• budgetClient simulates a reasoning model that truncates below a token threshold.
• Voice engine + campaign recover in-job by doubling the budget.
• A single truncation does not raise the starting budget; only after truncationsBeforeRaise truncations does a later job start one step higher.
• budgetMemory unit test: ignores one-offs, raises one step per pattern (no doubling), caps at maxTokenBudget, leaves other models untouched.

go build ./..., go vet ./..., go test ./... all pass.

Follow-up to #72 / #75.

🤖 Generated with Claude Code

## Problem After #72, a recurring "recreate" AI job failed with: ```json { "duration_ms": "25669", "error": "llm response truncated at token limit" } ``` The truncation **detection** works — but the generators returned `ErrResponseTruncated` straight to the caller, so the job failed. The ~22–25s durations of the *successful* jobs show the configured model is a **reasoning model** that occasionally spends its whole budget on hidden reasoning tokens and stops at `finish_reason: "length"` before finishing the JSON. ## Fix (two clearly separated layers) **1. In-job recovery (fast).** The voice-engine and campaign retry loops catch `ErrResponseTruncated` and **double** the token budget (`escalateBudget`, 2048 → 4096 → 8192, capped) and retry, so the current job still succeeds. **2. Cross-job learning (gentle, per model).** Always starting at 2048 would make a heavy model pay for a truncated attempt + escalate-retry on *every* job. A process-wide `budgetMemory` adapts the **starting** budget per model — but only on a *pattern*, not a fluke: - A single truncation changes nothing ("einmal ist keinmal"). - Only after a model truncates **`truncationsBeforeRaise` (3)** times does its starting budget creep up — and then by just **one `tokenBudgetStep` (1024)**, not by doubling — capped at `maxTokenBudget`. The counter resets, so the next raise needs a fresh run of truncations. So an isolated truncation is absorbed by the in-job retry and leaves the default alone; a genuinely heavy model slowly settles at a starting budget that stops it from looping. `Model()` was added to the `Client` interface to key the memory. The memory is process-local (re-learns after a restart); persisting it across restarts is a possible follow-up. The 10-minute `jobExecutionTimeout` leaves ample room for the larger retries (each call ≈25s). ## Tests (added, TDD) - `budgetClient` simulates a reasoning model that truncates below a token threshold. - Voice engine + campaign recover in-job by doubling the budget. - A single truncation does **not** raise the starting budget; only after `truncationsBeforeRaise` truncations does a later job start one step higher. - `budgetMemory` unit test: ignores one-offs, raises one step per pattern (no doubling), caps at `maxTokenBudget`, leaves other models untouched. `go build ./...`, `go vet ./...`, `go test ./...` all pass. Follow-up to #72 / #75. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kreativmonkey added 1 commit 2026-06-13 20:55:23 +02:00

fix(ai): retry with a larger token budget when a reply is truncated ... e7c4f97ced

After #72, truncated replies surface as ErrResponseTruncated — but the
voice engine and campaign generators returned that straight to the caller,
so a reasoning model that spends its budget on hidden tokens (≈25s, then
finish_reason "length") still failed the whole job.

Treat truncation as "needs more room, not a different prompt": on
ErrResponseTruncated the retry loops now double the token budget
(escalateBudget, capped at 8192) and try again instead of bailing. Prompt
re-wording is reserved for parse/validation failures as before. The 10-minute
job timeout leaves ample room for the larger retries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

kreativmonkey added 1 commit 2026-06-13 21:11:46 +02:00

fix(ai): learn a per-model token budget to avoid the truncation loop ... 94ed79878d

Always starting at defaultMaxTokens means a heavy/reasoning model pays for
a truncated attempt plus an escalate-retry on *every* job. Instead, remember
per model the budget that works:

- Add Model() to the Client interface.
- A process-wide budgetMemory records, per model, the starting budget. On
  truncation it raises that model's starting budget (escalateBudget, capped
  at maxTokenBudget); subsequent jobs for the same model begin there and skip
  the doomed default attempt.

The escalate-retry within a single job stays as the safety net for the first
occurrence (and after a restart, until re-learned).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

kreativmonkey changed title from fix(ai): retry with a larger token budget on truncation to fix(ai): adaptive per-model token budget on truncation 2026-06-13 21:11:58 +02:00

kreativmonkey added 1 commit 2026-06-13 21:18:54 +02:00

fix(ai): soften per-model learning — only adapt on a recurring pattern ... 4476c2ccb4

Raising the starting budget after a single truncation was too eager
("einmal ist keinmal"). Now:

- In-job recovery still doubles the budget so the current job succeeds.
- The learned *starting* budget only moves after a model truncates
  truncationsBeforeRaise (3) times, and then only by one gentle
  tokenBudgetStep (1024), not by doubling — capped at maxTokenBudget.

So an isolated truncation is treated as a fluke and changes nothing; a
genuine pattern nudges the default up slowly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

kreativmonkey merged commit 896ccc0a30 into main 2026-06-13 21:22:08 +02:00

kreativmonkey referenced this pull request from a commit 2026-06-13 21:22:08 +02:00

fix(ai): adaptive per-model token budget on truncation (#78)

kreativmonkey referenced this pull request from a commit 2026-06-17 09:09:33 +02:00

fix(ai): adaptive per-model token budget on truncation (#78)

kreativmonkey referenced this pull request from a commit 2026-06-27 08:54:30 +02:00

chore(main): release 0.1.1 (#4)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

wontfix

This won't be fixed

No labels

bug

duplicate

enhancement

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kreativmonkey/goloom!78

No description provided.