Posting this because it took me way longer than it should have, and the way it fails is basically invisible unless you know where to look.
We run LLM agents as Temporal workflows. The loop is: call a tool (activity), append the result to the message list, schedule an inference activity with the whole list as input, repeat. Nothing exotic.
One of these kept dying at 30 minutes. Not roughly 30. Exactly 30:00, which is its execution timeout. So naturally I assumed a slow tool. Pulled activity latencies, nothing. Checked the upstream API rate limits, fine. Added more logging, restarted things, the usual. Still died at 30:00 every single time. I was pretty convinced something upstream was hanging on us.
What finally cracked it was giving up on the dashboard and reading the raw history. temporal workflow show -w <id>, or the History tab in the UI filtered to WorkflowTaskFailed. And there it was, repeating every few seconds for half an hour:
WorkflowTaskFailed
cause: WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE
3.9 MB against the 2 MB limit.
So the workflow was trying to schedule the inference activity with a ~3.9 MB input. Server rejects the workflow task. SDK retries it. Workflow replays deterministically to the exact same state, emits the exact same command with the exact same payload, gets rejected again. Forever. From the outside it looks identical to a hang, which is why I spent so long looking for one.
Root cause was dumb in hindsight. One tool fetches commit details, and the service behind it returned the diff plus the full before and after content of every changed file. Roughly 90% of the bytes were stuff the model never read. We did have an S3 offload for big activity results, but it checked each result on its own, and none of them was over 2 MB. Five of them together were. Nothing was measuring the pile, only the items.
Fix ended up being two small changes. The commit-details endpoint now returns just the diff, and the agent loop trims older tool observations as the conversation grows so the message list can’t balloon no matter how many calls it makes.
A few things I’d tell past me:
If a workflow dies at exactly its timeout, go read history for WorkflowTaskFailed before you touch latencies. A workflow task retry loop and a hang look the same if all you’re watching is the clock.
When the input to your next step is an accumulator, measure the accumulator. Per-item checks tell you nothing about the sum.
And have an alert on WorkflowTaskFailed. We didn’t. We do now. Would have been a five minute debug instead of a multi-day one.
Genuinely curious how others handle this. We looked at a custom PayloadCodec that offloads anything over a threshold to blob storage and decided against it for now, mainly because it papers over a tool returning 10x more data than anyone needed. But I can see it as a safety net, and I might be wrong.