Repository navigation
LangGraph: prune finished subgraph runs, and refuse static breakpoints - #290
igalshilman wants to merge 2 commits into
Conversation
…d steps LangGraph stores each subgraph run in a namespace named after its task (node:task_id), and SqlCheckpointSaver(history="latest") only pruned within the namespace being written. Every subgraph run's last checkpoint, values and writes were kept for good: a graph agent (langgraph.agent(graph=...)) with a sub-agent subgraph stored a full copy of its messages each turn. Each aput now also deletes the subgraph runs that the graph's earlier steps started: those whose checkpoints name, in their metadata's "parents", a checkpoint of that graph older than the one stored (checkpoint ids sort in time order). Kept are: - the runs of the steps from the stored checkpoint on: with durability="async", LangGraph may store a checkpoint after a later step has started its runs, and with "exit" a paused run names the very checkpoint being stored; - a paused subgraph, as the graph's checkpoint stays until it resumes; - the runs below a subgraph with its own history (checkpointer=True, a namespace without a task id), which its own checkpoints decide: it can resume a paused step after the graph above it moved on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkTSkJ7FFKSJs2Vsub9
A static breakpoint (interrupt_before, interrupt_after) stops a run with nothing to wait for. langgraph.agent(graph=...) then answered with the run half done (often "", the text of a reply that only calls tools), and left its tool calls without results: the next message came after them, a model provider refused that history, and every later message on the key failed the same way until clear. check() now refuses a compiled graph with static breakpoints (such as create_agent(..., interrupt_before=["tools"])), pointing to interrupt(). A run that stops at a subgraph's breakpoint, which check() cannot see, fails with a terminal error, which closes the run so the next message starts afresh. A run counts as stopped there only when it has steps to go and no task has an interrupt: an interrupt() in a node's second subgraph call is on the node's task, though not in the subgraph snapshots LangGraph reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RePHkTSkJ7FFKSJs2Vsub9
There was a problem hiding this comment.
ℹ️ No critical issues. Two minor suggestions on the static-breakpoint path; the subgraph pruning looks right.
Reviewed changes
Reviewed both commits: subgraph-run pruning in SqlCheckpointSaver(history="latest") and the refusal of static breakpoints in graph agents.
- Prune finished subgraph runs —
aputnow also deletesnode:task_idnamespaces below the stored graph whoseparents[ns]checkpoint is older than the one being stored. Runs started by the current step, paused subgraphs, and anything below acheckpointer=Truesubgraph are kept. - Refuse top-level static breakpoints —
check()raisesTypeErrorfor a compiled graph withinterrupt_before/interrupt_after. - Fail a run stopped at a subgraph's breakpoint — after a run stops with steps still to go and no interrupt,
at_breakpointraises aTerminalError, whichanswer_onceuses to close the run. - Tests — pruning under
sync/async/exit, a paused subgraph resuming whole, acheckpointer=Truesubgraph, the refusal, andat_breakpointtelling a breakpoint from a second-callinterrupt(). The in-process tests pass locally; the Docker-harness test is CI-only.
claude-opus-5-5 | 𝕏
| start: Any | ||
| if asked: | ||
| resumes = await _resumed(ctx, app) | ||
| resumes = await _resumed(ctx, app, await _paused(app)) |
There was a problem hiding this comment.
The breakpoint check only runs inside the while loop. A run picked up again here (a retried attempt, or an earlier prompt's unfinished run that answer_once finishes first) gets resumes is None → start = None. LangGraph treats None input as "continue past the breakpoint", so if the attempt died between the subgraph stopping and close(...) committing, the retry runs the rest of the graph and answers instead of failing. I confirmed this with InMemorySaver: ainvoke(None) on a thread stopped at the test's pause subgraph finishes with hello x.
Technical details
# Retry path skips at_breakpoint and resumes past a subgraph's static breakpoint
## Affected sites
- python/restate/agents/langgraph/_graph.py:322-324 — the `asked` branch calls `_resumed` and falls back to `start = None` without checking `at_breakpoint(paused)`
## Required outcome
- A run found stopped at a subgraph's static breakpoint on entry fails with `TerminalError(BREAKPOINTS)`, the same way it does in the loop, instead of being resumed with `None`.
## Suggested approach
```python
if asked:
paused = await _paused(app)
resumes = await _resumed(ctx, app, paused)
if resumes is None and at_breakpoint(paused):
raise TerminalError(BREAKPOINTS)
start = Command(resume=resumes) if resumes is not None else None
```| """Raise ``TypeError`` if ``graph`` has no ``messages`` in its state, or has static breakpoints.""" | ||
| if "messages" not in graph.channels: | ||
| raise TypeError("the graph's state has no messages: build it on LangGraph's MessagesState") | ||
| if isinstance(graph, Pregel) and (graph.interrupt_before_nodes or graph.interrupt_after_nodes): |
There was a problem hiding this comment.
Optional: breakpoints on a subgraph that is a node can be seen up front. builder.compile().get_subgraphs(recurse=True) yields ('pause', ...) with interrupt_before_nodes == ['greet'] for the test graph. If check() walks these too (compiling a StateGraph first), the agent is refused when it is registered, not after a turn has already made model and tool calls. The runtime at_breakpoint check would then only be needed for subgraphs invoked from inside a node function.

Two fixes to graphs of your own in
restate.agents.langgraph(langgraph.agent(name, graph=...), andSqlCheckpointSaver), from a review of #288 and #289.history="latest"kept every subgraph runLangGraph stores each subgraph run in a namespace named after its task (
node:task_id, one per run), andSqlCheckpointSaver(history="latest")only pruned within the namespace being written. Every subgraph run's last checkpoint, channel values and writes were kept for good: a graph agent with a sub-agent subgraph stored a full copy of the sub-agent's messages every turn, the growthlatestexists to prevent.Each
aputnow also deletes the subgraph runs that the graph's earlier steps started: those whose checkpoints name (in their metadata'sparents) a checkpoint of that graph older than the one being stored. Checkpoint ids sort in time order. It keeps:durability="async", LangGraph may store a checkpoint several steps after a later step started its runs. With"exit", a paused run names the very checkpoint being stored;checkpointer=True, a namespace without a task id). That subgraph's own checkpoints decide them, since it can resume a paused step after the graph above it moved on.Static breakpoints broke a key for good
A static breakpoint (
interrupt_before,interrupt_after) stops a run with nothing to wait for. The agent then answered with the run half done (often"", the text of a reply that only calls tools) and left its tool calls without results. The next message came after those calls, a model provider refused that history (recorded as a 424 refusal), and every later message on the key failed the same way untilclear.create_agent(..., interrupt_before=["tools"])is a common way to get there.check()now refuses a compiled graph with static breakpoints, pointing tointerrupt(), which waits for a signal.Tests
tests/agents_langgraph_checkpointer.py:sync,asyncandexit. Before the fix, three runs left threeinner:<task_id>namespaces.asyncthe graph's checkpoints are stored after the subgraph has started.checkpointer=Truesubgraph survives the graph moving on (update_state), and is deleted once that subgraph's step is done.tests/agents_langgraph_graph.py:interrupt_beforeorinterrupt_afteris refused.200 "".at_breakpointtells a subgraph's breakpoint from aninterrupt()in a node's second subgraph call.Each test was checked against a broken variant of its fix and fails on it: keeping only runs of the stored checkpoint (
!=instead of<, which loses a paused subgraph's state underasync), ignoringparents, pruning below acheckpointer=Truesubgraph, and thepaused.next-only breakpoint check.tests/agents_langgraph*.py(62) and the LangGraph example tests pass. ruff, pyright and mypy are clean.Not in this PR
_interruptslooks for a subgraph's interrupts in the subgraph snapshots LangGraph reads (node:task_id), so aninterrupt()in a node's second subgraph call (node:task_id|1) is not waited for, and the turn answers half done, as before this PR. Fixing it means counting resumes from the pending writes of every namespace.from_tool; the graph path doesn't.context=ctxreplaces a graph's own runtime context (create_agent(context_schema=...)).Generated by Claude Code