Skip to content

How to watch and resume a run

Use this page to follow a run, to pause it, to stop it now, and to change its plan. It also shows how to go on with a run that stopped, and how to clean up after a run. The state of a run is in its journal. A stop loses no finished work.

Before you start

You have a workflow file and the agent pair for each stack. See how to bootstrap a single-stack repository. This page uses this workflow with two tickets.

workflow.toml
base-branch = "main"
run-branch = "run/demo"
mode = "assure"
adapter = "claude-code"

[stacks.python]
specialist = "python-specialist"
verifier = "python-verifier"
gates = ["python3 -m unittest"]
# The engine also reserves .claude/agents/ and .opencode/agents/ in every stack.
hotspots = ["workflow.toml"]
# The time limits of one gate run and of one agent session, in minutes.
gate-timeout = 30
agent-timeout = 90

[[tickets]]
id = "1"
text = "The export command reads the data set."
stack = "python"
blocked-by = []

[[tickets]]
id = "2"
text = "The export command writes a header row."
stack = "python"
blocked-by = ["1"]

The engine keeps the state of a run in ~/.local/state/delegate, outside the git directory. XDG_STATE_HOME or DELEGATE_STATE_ROOT changes the root. A run that an older version started is in .git/delegate. Each command below finds it there, so you change nothing. See where the engine writes.

Watch a run

1. Start the run

Start the run in one terminal.

delegate run workflow.toml

When the run starts, the command prints three lines: run <run id>, journal <path>, and follow it with delegate watch · pause it with delegate pause <short id>. Keep the run id. The short id is the part of the run id after the last -. status, watch, pause and --resume take the run id or the short id. When the run ends, the command prints a summary line, for example run built · 5s · 2 built · 0 failed · 0 skipped · 0 new · 0 cache.

2. See the state

In a second terminal, in the same repository, read the state of the run. With no run id, status and watch read the one active run.

delegate status
delegate watch --stall-minutes 30 --max-minutes 50

status prints the state of the run once, and exits. It prints the progress of the run: the percentage, the tickets that ended, the time used, the ETA, and the tokens as two counts. The two counts are the new tokens and the cache reads. Then it prints the ticket table, with one flat row for each ticket in plan order. A rail │ joins a row to the row above when that row is its blocker. The after column names a blocker that is not the row above. A row also gives the state, the phase, the verdict, the branch, the time used and the tokens. Then it gives the time of the last event, and the reason of a ticket that failed or was skipped. Each further phase of a running ticket is on a sub-row. status prints one snapshot, so it gives no time since the last activity of an agent: delegate status --json gives it. See the status reference for the columns. A program can read the same snapshot with delegate status --json. See the snapshot.

watch follows the run until it ends. In a terminal, it shows the live view. At the top, the run line gives the same progress as status: the percentage, the time used, the ETA, and the tokens. Under it, the ticket table has one flat row for each ticket in plan order, with the rail and the after column of status. A long chain of blockers does not make the table wider. A row gives the state, the first phase, the time used, the tokens, and the liveness of its agent: a spinner while the agent is active, quiet <time> after 30 seconds with no activity, and ⚠ idle <time> after the stall time. Each further phase of the running ticket is a sub-row under it. Below a divider, the feed shows one short line for each event of the ticket that runs now: the local time, the ticket, and what happened. For example, 10:51 #64 specialist finished · 36m · tokens 302k new, 9.7M cache · report ok. A problem and a stall are lines that start with ✗. watch draws the frame again at each poll. See the live view.

The option --stall-minutes makes watch exit when nothing changes for 30 minutes. The option --max-minutes makes it exit before a harness time limit. The journal names the event stream of each agent when the agent starts, so an agent that writes its stream is not a stall. The journal also names the log file of each gate when the gate starts, so a long gate that writes output is not a stall.

When standard output is not a terminal, watch prints one raw line for each event, for a script or a harness. To see the raw lines in a terminal, add --raw:

delegate watch --raw

See the raw lines.

With no run id, watch follows the one active run, also when a newer run ended. A run is active when its journal has no run-end and a live process holds the lock of its run branch. When more than one run is active, watch on a terminal shows a table of the active runs, the newest run first. Then it asks which run to follow: type its number, or push Enter for the newest run. When standard input or standard output is not a terminal, watch follows none: it prints one line on stderr that names each active run, and exits 2. See the run with no run id and the run picker.

When no run is active and the newest run has ended, watch with no run id shows none of its events. It prints one line with the result of that run, and exits. When the newest run stopped with no run-end, the line tells how to resume it. To see the events of a run that ended, give its run id:

delegate watch <run-id> --raw

See the newest run that ended.

3. Act on the exit code

The exit code of watch tells you why it stopped. For example, the run succeeded, failed, paused or halted, a step has a problem (only in the raw lines and the JSON lines: the live view shows a problem as a warning line, and goes on), or a limit is reached. See the exit codes of watch for each code.

You can run watch as a background task in your harness. The harness then alerts you when the command exits.

4. Go on after a limit

A watch that follows a journal prints a line position <seq> before it exits. After Ctrl-C, it prints the position in a line on stderr, and exits 130. To go on from that point, give the position. No event is printed twice.

delegate watch --from <position> --stall-minutes 30

With no run id, watch --from chooses the run as watch with no run id does: the one active run, also when a newer run ended, or the run that you choose when more than one run is active. When no run is active, it goes on in the newest run. To go on in a specific run, give its run id before --from, as the line of Ctrl-C does. See the run with no run id.

Pause a run

A pause stops a run at its next step boundary. It cuts off no agent, so no work in progress is lost.

1. Ask for the pause

In a second terminal, in the same repository, give the run id or the short id.

delegate pause <run-id>

The command prints run <run id>: pause requested; the run stops at its next step boundary and the path of the request. The run stops before the next step, gate or verifier starts. delegate status shows pausing until the run stops, and then paused. delegate run then prints the summary and the line below, and exits 3. watch exits 7.

run paused · 1s · 0 built · 0 failed · 0 skipped · 0 new · 0 cache
paused before the gate `python3 -m unittest` of ticket 1; go on with delegate run --resume <run id>

A run that stops before it reads the request does not show pausing. For example, Ctrl-C stops it. delegate status then shows a note that a pause request waits. A run with no live process cannot be paused: delegate pause exits 1. See the pause.

2. Go on after the pause

Resume the paused run with the same command as after Ctrl-C. See resume after Ctrl-C. Before you resume, you can add a ticket, or change a ticket that has not started. See change the plan of a run.

Stop a run now

A pause waits for the next step boundary, and that can be many minutes away. To stop a run now, use delegate stop. It can cut off an agent or a gate, but it loses no finished work.

1. Stop the run

In a second terminal, in the same repository, give the run id or the short id.

delegate stop <run-id>

The command sends SIGTERM to the process that holds the lock of the run. The run ends the process group of its running child, writes interrupted in the journal, and releases the lock. The command waits up to 30 seconds for the lock to be free. Then it prints two lines and exits 0:

run <run id>: stopped
go on with delegate run --resume <run id>

When a process still holds the lock at the end of the wait, the command says so on stderr and exits 1. A run with no live process cannot be stopped: a run that ended, a run that is paused and a run that stopped already. For these runs delegate stop exits 1 and gives the reason. See delegate stop.

2. Go on after the stop

Resume the run with the command that stop printed. See resume a run.

Change the plan of a run

A resume can add a ticket to a run. It can also change or remove a ticket that has not started. The built part of the run stays as it is. See the plan change for each rule.

1. Stop the run

Pause the run, as in pause a run, or stop it, as in stop a run now. You can also change a run that stopped, or a run that ended.

2. Edit the workflow

Edit the workflow file that the run started from. In this example, ticket 1 started before the pause. The edit adds ticket 3, and ticket 2 now waits for it.

[[tickets]]
id = "2"
text = "The export command writes a header row."
stack = "python"
blocked-by = ["1", "3"]

[[tickets]]
id = "3"
text = "The export command reads the options file."
stack = "python"
blocked-by = ["1"]

A new ticket goes at the end, or before a ticket that has not started. The blocked-by lists set the new order. Do not change or remove a ticket that has started. The resume refuses such a change, and names the ticket and the field.

In economy mode, a new ticket starts from the last built ticket before it in the chain. After a chain whose verification failed, the tickets of the chain are not built. So the resume refuses a new ticket after that chain, unless it also has --retry <tip> for the ticket at the tip of the chain. See retry after a failed verification of the tickets.

3. Check the change

delegate run --resume <run-id> --dry-run

The dry run changes nothing. It prints the old order, the new order, the verdict of each rule, and the text of each added or changed ticket.

run <run id>: plan change
old order: 1, 2
new order: 1, 3, 2
rule 1: ok: the built part is fixed: the built part is 1
rule 2: ok: a new ticket goes after the built part: added: 3
rule 3: ok: a ticket that has not started can change: changed: 2 (blocked-by)
rule 4: ok: no built work is changed: no new ticket comes before a ticket that has started
rule 5: ok: a new ticket gets its own gates and verifier: each new ticket gets its own gates and verifier
rule 6: ok: the journal records the change: the resume writes plan-change
rule 7: ok: a ticket that has started cannot be removed: removed: none
rule 8: refused: the coordinator reviews the later tickets: needs --reviewed for 2
rule 9: ok: the brief of each later ticket gets the section Plan change: the briefs of 2 get the section
later tickets to review, in plan order: 2
ticket 3 (added):
  The export command reads the options file.
ticket 2 (changed):
  The export command writes a header row.

A later ticket has not started, and comes after an added or changed ticket. Here ticket 2 is a later ticket. Without --reviewed, the dry run and the resume exit 1, and change nothing. The message on stderr names the tickets to review:

delegate run: the plan change needs a review of the later tickets '2': read each one against the added and changed tickets, change its text or its blocked-by where a precondition changed, then give --reviewed 2 or --reviewed all

4. Review the later tickets

Read each later ticket against the added and changed tickets. Change its text or its blocked-by when a precondition changed. Then run the dry run again with --reviewed 2. It exits 0 when each rule accepts the change.

5. Resume the run

delegate run --resume <run-id> --reviewed 2

--reviewed takes the ids of the later tickets, separated by commas, or all. The journal records plan-change. The brief of each later ticket gets the section Plan change. Its specialist reports blocked when the ticket no longer fits the new plan.

Resume a run

A run stops when you press Ctrl-C, when you send its process SIGTERM or SIGHUP, or when its process is killed with SIGKILL. Such a run has no run-end in its journal, and no process works on it. So delegate status shows the result stopped and the command that goes on with it: delegate run --resume <run-id>. After Ctrl-C, delegate run writes interrupted in the journal, prints the command that goes on with the run, and exits 130. See Ctrl-C. SIGTERM and SIGHUP do the same, so kill <pid> is the way to stop a detached run. The run first ends its harness and its gates with all their child processes, so no agent keeps working after the run stopped. See SIGTERM and SIGHUP. A paused run has no run-end too, and delegate status shows paused.

1. Resume after Ctrl-C

Give the run id. The workflow file is the one that the run started from. The engine rebuilds the state from the journal and starts at the first step that is not complete. It never builds a finished step again.

delegate run --resume <run-id>

A ticket with a recorded commit is not built again. A step that was open continues in its own worktree. See the resume reference for each case.

2. Resume after a kill

A process that SIGKILL ends cannot release the lock of its run branch, and it cannot end its children. Before you resume, check that no process of the harness still works in the worktree of the open step. To stop a run, use delegate stop <run-id> instead: the run then ends its children and releases the lock. Resume the run with the same command as after Ctrl-C. The kernel released the lock when the process ended, so the resume takes it over and needs no flag. When the run ends, the command prints a note on stderr that names the old process.

A lock that a process holds stays. A lock file that no process holds is free, also when it names another run id or a process id that another process uses now: --break-lock is not necessary. See the run lock.

A run that the invariant check ended cannot be resumed, also with --retry: start a new run. The same is true for a run that ended with run-branch-moved. A commit that the engine did not advance to is on the run branch. See the run branch check. A run that has a run-end can be resumed only with --retry, or with a change of its tickets. See the plan change.

3. Retry after a failed verification of the tickets

In economy mode, the tickets form a chain. A failure of the verification of the chain fails each ticket of the chain. Only the ticket at the tip of the chain goes on. The retry line at the end of the run names it.

A gate can fail one time and then pass, and no commit fixes such a gate. Retry the tip with no commit. First run the dry run, delegate run --resume <run-id> --retry <tip> --dry-run. It prints retry <tip>: phase chain, revives <ids>. Then run delegate run --detach --resume <run-id> --retry <tip>. The retry builds the other tickets of the chain again. It runs the gates again on the tip, and starts a fresh verifier. The fix-up rounds before the retry do not count against the limit.

A gate can also need a change of a hotspot path, and only you can make it. Commit the change on the branch of the tip, in its worktree. Then add --coordinator-commit <sha> to the retry. The retry line names this flag when the newest report of a fix-up of the chain names a hotspot path. See the retry in economy mode and the retry with a commit of the coordinator.

The fix-up of the chain can also need new rulings of the owner. Add the rulings as new tickets at the end of the workflow, and give the retry of the tip in the same resume: delegate run --resume <run-id> --retry <tip> --dry-run, then delegate run --detach --resume <run-id> --retry <tip>. The retry builds the tickets of the chain again. The first new ticket starts from the tip of the chain, so its branch holds the work of the chain. After the last new ticket, the verification of the chain runs again over the whole chain. Without --retry <tip>, the dry run and the resume refuse the new tickets: rule 5 of the plan change names --retry <tip>. The other way is a new run.

Clean up after a run

A run that ends built removes the worktree of each ticket, and deletes each ticket branch that the run branch holds. The run branch stays, because you merge it. A run that ended before that clean-up existed, or with keep-worktrees = true, keeps its worktrees and its ticket branches. delegate clean removes them. See delegate clean.

1. Merge the run branch

Merge the run branch into the base branch, as the merge steps of the agent-delegation skill tell.

2. See what the command removes

delegate clean --ended --merged --dry-run

The command changes nothing. It prints one line for each run: the worktrees and the branches that it would remove, each item that it would keep with the reason, and the space that it would free. --ended selects each run that ended built. --merged adds the run branch of such a run when the base branch or the default branch holds it. A run that is active, failed, paused or stopped keeps everything, and its line gives the reason. To clean one run, give its run id in place of --ended.

3. Remove it

delegate clean --ended --merged

The command removes what the dry run named, and records a cleanup event in the journal of each run where something went. A worktree with a change that is not committed stays, and so does its branch. A file that a gate makes, for example in __pycache__, is such a change when .gitignore does not name it. The command exits 0 when it removed or kept each item by these rules, and 1 on an error.

Abandon a run

Use this section when you give up on a run that is not built. A run that you abandon cannot be resumed. The decision goes to the journal, so the owner of the run can see it.

1. Check that no process works on the run

The run must be paused, halted, failed, or stopped with no run-end. If a process works on it, stop it first. See stop a run now.

2. Abandon it

delegate clean --abandon <run-id> --reason "<why you give up on the run>"

The reason is required. The command writes run-end with the result abandoned and your reason in the journal. Then it removes the worktree, the ticket branch and the hooks directory of each ticket. It removes them also when they hold work that is not committed or not merged. The command keeps the run branch and prints its name. Add --delete-run-branch to delete it too.

The command exits 1 and changes nothing in these cases. A process works on the run: the message names delegate stop <run-id>. The run ended built: the message names delegate clean. A run that was abandoned before needs no new decision. The command then only removes what is left.

3. See the result

delegate status <run-id> shows the state abandoned and the line note: abandoned: <reason>. The live view of delegate watch shows the reason on the last line of the feed. delegate run --resume <run-id> refuses the run and exits 1. To do the work again, start a new run. See delegate clean.

Report a run as a coordinator agent

A coordinator agent can start a run for a person. The person cannot see the agents, so the coordinator reports the run. This example starts the run detached from the tool shell.

delegate run --detach workflow.toml

The command first checks the workflow, the tier file, the lock and the branches. For a resume, it also checks a run that ended, the run branch, --retry and --coordinator-commit. A check that fails exits 1 and starts no process. Then the command prints run <run id> and log <path>, and exits at once. The run goes on in its own process, and writes its output to the log file of the run. See the detached run. delegate status and delegate watch with no run id read the one active run, or the newest run when no run is active. With more than one active run, status prints a table of the active runs. With no terminal, watch prints one line that names each active run, and exits 2: give the run id. The coordinator obeys these rules from the start of the run to its end.

  1. Start the run detached. Before the start, run delegate run <workflow> --dry-run, and read its quality check. Start the run with delegate run --detach, as in the example. The command starts the run process with the setsid system call, so the run is outside the process group of the tool shell. A harness can stop a background task at a time limit, and it can stop the processes of its tool shell. A run that only has nohup and & can stay in the process group of the tool shell.
  2. Follow the run with delegate watch. Start delegate watch as a background task, as in step 2. The harness then tells the coordinator when it exits. A background task has no terminal, so watch prints the raw lines, as with --raw. When it exits at a limit, start it again with --from <position>, as in step 4.
  3. Show delegate status as it is. Each time the coordinator tells the person the state, it shows the output of delegate status, as it is. The coordinator does not write its own script to read the journal or an event stream.
  4. Report a stall at once. A stalled agent is an agent with no activity for the stall limit, the time that --stall-minutes sets. watch then exits with the reason stall, as the stall tells. The coordinator tells the person at once, and gives the output of delegate status.
  5. Resume a stopped run. When the run process stops before the run ends, the coordinator goes on with delegate run --detach --resume <run-id>. The resume takes over a lock that no process holds, with no flag, as in the resume after a kill. A paused run goes on with the same command. After a change of the workflow, the coordinator first runs the dry run of the resume, and adds --reviewed. With --detach, a refused plan change or a refused retry prints its error on stderr, exits 1, and starts no process.

Version main