The API run is still on the chat clock

A planned stop still puts an in-flight API run on the chat clock. That clock defaults to zero. Cron has a separate floor. API does not.

If you are about to stop a gateway that is serving HTTP runs, read agent.restart_drain_timeout before you read the cron floor. The API run shares the first number. On a default config that number is 0, so the drain can report timed_out=True after 0.00s with api_at_start greater than zero and never enter the wait loop. I measured that split on a test double of this checkout. I did not stop the gateway this host is running.

This host's config sets that shared clock to 60. The stops already in this gateway log waited about 60 seconds, and every line that records api_at_start records 0. Do not paste the open report's 0.00-second table onto this machine. The code path that would produce that table is still the one in the tree, and it is still the one on the tip I fetched this morning.

What to read before you stop anything

Four checks, in this order. None of them require a stop.

1. Open the config the running gateway actually loads. Read agent.restart_drain_timeout. If the key is missing, the shipped default is 0. API runs share that 0 on the stop path.
2. Read agent.cron_drain_timeout only after that. Its default is 30. It is a cron floor. It does not cover API runs. On a host that has already raised the chat clock, the cron budget is lifted to at least that chat clock. It is not a second, shorter clock sitting under it.
3. Grep the gateway log for timed_out=True and read api_at_start on the same line. A 0.00s line with api_at_start greater than 0 is the signature in the open report. A line that waited about 60s with api_at_start=0 is a different class of work hitting a raised chat clock.
4. Print hermes cron --help if you think you can cancel a wedged job. On this install the verb list has no cancel. pause is not that verb.

Do not stop the gateway to produce the log line. Do not treat an in-band restart as the same budget as stop(). The restart wait counts API work. The stop drain, after that wait ends, puts API work back on the chat clock.

The report is open. The first sentence says it was not submitted.

GitHub issue #132989 is OPEN. I viewed it this morning. createdAt is 2026-10-05T00:21:34Z. updatedAt is 2026-10-05T00:30:08Z. Labels include type/bug, comp/gateway, and P1. The title is "Upstream Hermes bug report - gateway stop drain kills api_server runs."

The body opens with "Status: written, not submitted." The next sentences say submission is externally visible work and needs the owner, and that everything below the rule is the report. The tracker state is still OPEN. Those two facts sit next to each other. I am not resolving them. A status line inside the body is not the issue state.

The environment table in that body names Hermes 0.21.5, a Windows 11 install, Python 3.14.7, and a multiplex gateway on 127.0.0.1:8653. It says the config has no agent.*drain* overrides, so the process was on defaults. That is the reporter's host. It is not this host. I did not reproduce the four drain events they list. I did not open their client logs. The body correlates two of four events with client-visible cancels and says the other two owners were not identified. Read that table as their evidence, not as a measurement from here.

The summary they wrote is the claim I checked against source: any gateway stop, including a planned stop, destroys in-flight api_server runs immediately when the chat-turn budget is the default 0. API work is counted. It is counted against agent.restart_drain_timeout. Cron was given a separate deadline after #82161. API was not.

I did not re-read #82161's body. The citation I used is the comment and the regression file in this checkout.

Four counters. Two deadlines.

This checkout is 5000e29936df69d5209f7cf2eea8e5776cb4cbb1, committed 2026-09-29 11:55:48 -0400. gateway/run_shutdown.py in that commit is 124762 bytes, sha256 573d68b84ba94cb7ad402c66b70375626255cda9d82f4cc51887a5514a9eb276.

_drain_work_counts returns a four-tuple. The docstring names the order: agents, cron, api, deferred. The stop path logs all four. Counting API work is not the bug. The bug, if you are on the shipped default, is which deadline that count is compared to.

_drain_active_agents takes timeout and an optional cron_timeout. In this file, at lines 836-842:

deadline = started + timeout
cron_deadline = started + (timeout if cron_timeout is None else cron_timeout)

def _still_draining() -> bool:
    now = loop.time()
    agents, cron, api, deferred = self._drain_work_counts()
    return bool(((agents or api or deferred) and now < deadline) or (cron and now < cron_deadline))

timeout is the chat clock. The caller passes restart_drain_timeout. api and deferred are in the same parenthetical as agents. They do not see cron_deadline. If timeout is 0, deadline == started, now < deadline is false on the first check, and the wait loop is never entered for API work. timed_out is then any(self._drain_work_counts()). An API run still in flight makes that true. The caller then interrupts remaining work.

The comment directly above that split, lines 830-834, already describes this failure for cron. It says restart_drain_timeout defaults to 0 because interrupting a chat turn is announced and resumable, and that sharing one budget meant the default config could report timed_out=True after 0.00s with a cron job in flight and kill it, because the drain never entered the loop. The comment cites #82161. That sentence is now a description of what the same function does to an API run.

The caller is _stop_drain_active_work, line 1876 in this file:

ctx.active_agents, ctx.timed_out = await self._drain_active_agents(timeout, _cron_timeout)

_cron_timeout comes from resolve_cron_drain_budget. The log line that follows prints drain took, timed_out, active_at_start, cron_at_start, api_at_start, and deferred_at_start. If you only read active_at_start and it is 0, you have not checked the API counter. The report's clearest signature is active_at_start=0 with api_at_start greater than 0 and drain took 0.00s.

I grepped the fetched tip for that same return. origin/main:gateway/run_shutdown.py:842 is still:

return bool(((agents or api or deferred) and now < deadline) or (cron and now < cron_deadline))

I did not diff every line of the function. The grouping I grepped has not moved. A search of that tip for api_drain_timeout in config_defaults.py and run_shutdown.py returned nothing.

The probe, on a test double

I did not call hermes gateway stop. I did not send SIGTERM. I constructed a GatewayRunner the way the restart tests do, object.__new__, and called _drain_active_agents on that double. The method is the one in this checkout. The process is not the gateway in the user unit.

Four calls:

Case	timeout	cron_timeout	timed_out	elapsed
API only	0.0	30.0	True	0.0001s
Chat only	0.0	30.0	True	0.0001s
Cron only	0.0	0.2	True	0.2008s
Idle	0.0	30.0	False	0.0001s

The API-only case is the one the open report describes. The cron floor was 30, the shipped default. The chat clock was 0, the shipped default. The method returned timed_out=True without waiting. The chat-only case did the same. That is the regression the cron floor was written not to disturb.

The cron-only case used a floor of 0.2, not 30. I did that so the probe would finish. The shipped default floor is 30. The function that waited is the same function. It honored the cron deadline it was given, and it did not apply that deadline to the API case above it.

Idle returned timed_out=False immediately. An empty count does not time out. The variable that flips the API case is the live API count against a deadline that has already passed.

This is not a live stop. It does not prove what the running process has loaded. It proves what this checkout's method does when you pass the shipped defaults and an API count of 1.

The shipped numbers, called, not remembered

I imported the parsers from gateway/restart.py in this tree. restart.py is 18268 bytes, sha256 7ed15f9396c2073e3d7e966fd08aff3ab06bc8a97a5b929a916f87a53335900f.

The defaults the module exposes:

• DEFAULT_GATEWAY_RESTART_DRAIN_TIMEOUT = 0.0
• DEFAULT_GATEWAY_CRON_DRAIN_TIMEOUT = 30.0
• DEFAULT_GATEWAY_RESTART_AFTER_TURN_TIMEOUT = 1800.0

Those match hermes_cli/config_defaults.py in this commit. That file is sha256 0d8cf6c0b32310ab7bb4c1c8322d68d6c455a220d82825c5e3e690a84d94dc44. restart_drain_timeout is 0 at the key I read. The comment says 0 means interrupt immediately, and tells you to keep the value under systemd TimeoutStopSec or risk SIGKILL during cleanup. cron_drain_timeout is 30. The comment says an interrupted cron run is recorded as a permanent failure, so it must not inherit the chat clock's 0. restart_after_turn_timeout is 1800. The comment calls 30 minutes a safety valve for a wedged agent, not a target, and says 0 enters stop() at once.

Two parser traps, from calls I made:

• parse_restart_drain_timeout(None) returned 0.0. parse_restart_drain_timeout(0) also returned 0.0. parse_restart_drain_timeout(60) returned 60.0. The implementation passes raw or None into the zero-keeping parser. Explicit 0 collapses to the default, and the default is already 0, so you cannot use 0 on this key to mean something different from omitting it.
• parse_cron_drain_timeout(None) returned 30.0. parse_cron_drain_timeout(0) returned 0.0. Zero on the cron key is a real opt-out. The docstring says that opt-out puts cron back on the chat budget, which is the pre-#82161 behavior.

If you set cron_drain_timeout: 0 to "turn the extra wait off," you also turn off the only floor API runs do not have. You have not given API runs a floor. You have removed cron's.

The tests pin the split they were written for

tests/gateway/test_cron_drain_floor.py is the #82161 regression. I read it. I did not run pytest. The module docstring says the shared budget short-circuited on timeout <= 0 before the wait loop, and that the reported log line was drain took 0.00s, timed_out=True, cron_at_start=1.

test_chat_only_workload_keeps_the_zero_second_drain calls _drain_active_agents(0.0, 30.0) with one session in _running_agents and asserts timed_out is True and elapsed under 1 second. The docstring says the cron floor must not become a chat-turn wait. That assertion is why API, grouped with agents, still sees the zero. A fix that lets API ride the cron deadline would fail this test only if it also moved chat. A fix that gives API its own deadline would not touch this assertion. The test file does not contain that third deadline.

tests/gateway/test_api_server_active_work_drain.py does wait for API work, and it does time API work out. The timeout call I read is _drain_active_agents(0.1) with one argument. One argument means cron_timeout is None, so the cron deadline falls back to the same timeout. That test shows an API run can outlive a positive window. It does not show what happens when the chat clock is 0 and the cron floor is 30. That is the call I made in the probe. It is not the call in that file.

The file's other wait uses _drain_active_agents(2.0), again one argument, and finishes the API task inside the window. A green run of that file would not catch the default-config split. I did not run the file. I am describing the calls in it.

The restart wait is a different clock

_active_work_count adds four numbers: running agents, active cron jobs, active API runs, and deferred workers. _await_active_work_before_restart waits while _awaitable_work_count() is above zero, up to _restart_after_turn_timeout. Wedged work is excluded from that wait. The cap I imported is 1800 seconds unless the process loaded something else.

So an in-band restart does count API runs, for up to that cap, before it enters stop(). The open report says the same thing about restart_after_turn_timeout and the default 1800. I checked the count function. I did not invoke hermes gateway restart.

When that wait ends, or when the cap elapses, the code proceeds to stop(). stop() is the path with the two-deadline drain. An API run that is still going when the restart wait gives up is then judged on the chat clock, not on the 1800. If that clock is 0, the drain you just waited up to 30 minutes to avoid can still report 0.00s and interrupt the API run. The long wait and the short interrupt are not the same budget. Do not quote one of them as the other.

Pass --no-gateway-restart when cron updates Hermes is about the update flag, the in-band wait, and this host's unit. It is not this split. The flag still matters if a cron job is the process that would die in the restart it starts. It does not give API runs their own stop-path deadline.

This host is not on the shipped zero

The user unit is active and running. I read the unit file. I did not restart it.

• HERMES_HOME points at the default Hermes home, not a profile home.
• KillMode=mixed
• TimeoutStopSec=90
• ExecReload is kill -USR1 on the main pid
• The unit file does not set HERMES_RESTART_DRAIN_TIMEOUT

That home's config.yaml sets agent.restart_drain_timeout: 60 at the line I read. It does not set cron_drain_timeout. A profile config I also opened sets the same 60. The unit does not point at that profile home. The file that matters for this gateway is the one HERMES_HOME names.

I did not attach to the process and read the loaded attribute. The loader uses the env var if it is set, otherwise the config key, otherwise the default. The unit does not set the env var. The config key is 60. A past stop in the log waited 60.00s. That is consistent with the key. It is not a dump of the live object's field.

gateway.log on this host, read this morning:

• 43 lines contain api_at_start
• 0 of those lines have a value other than 0
• 3 lines contain timed_out=True

Those three:

• 2026-07-11, drain took 60.05s, cron_at_start=1. That line is the older format. It has no api_at_start field. I am not going to invent one.
• 2026-09-19, drain took 60.02s, active_at_start=4, api_at_start=0
• 2026-09-29, drain took 60.00s, active_at_start=1, api_at_start=0

I did not cause those stops. They are not the reporter's 0.00s events. They are evidence that stops on this host have waited about 60 seconds, and that the expanded log lines did not record an API run at the start of the drain. If your log looks like theirs, you are on a different budget than these three lines.

TimeoutStopSec=90 is the systemd bound, not the in-band wait. 60 is under 90. The comment in the defaults file says to keep the chat clock under that bound. If you raise restart_drain_timeout past what systemd will wait, you trade a clean interrupt for a kill during cleanup. I did not change the 60.

Raising the chat clock lifts cron. It does not name API.

resolve_cron_drain_budget returns max(drain, min(floor, ceiling)). The docstring says the result is never less than the chat drain. It only extends.

I called it:

• resolve_cron_drain_budget(0, 30, watchdog_delay=90, elapsed=0) returned 30.0. That is the shipped shape: cron gets 30, API stays on 0.
• resolve_cron_drain_budget(60, 30, watchdog_delay=90, elapsed=0) returned 60.0. Chat clock 60, cron floor 30, result 60.
• The same call with watchdog_delay=50 and elapsed=10 also returned 60.0. Once the chat drain is already above the cron floor, a tighter ceiling does not pull the result back down. max wins.

On this host, if the loaded chat clock is 60 and the cron floor is the unset default 30, cron work is not on a 30-second clock. It is on at least 60. The 2026-07-11 line, 60.05s with cron_at_start=1, is consistent with that lift. It does not prove which ceiling was in force that morning. The later lines with api_at_start=0 do not prove an API run would have been given 60 seconds. They prove those drains did not start with an API count. The method, given a non-zero API count and a timeout of 60, would wait on the chat deadline. I did not pass 60 into the probe. I passed 0, because that is the shipped default the report is about.

The workaround named in the issue is to set agent.restart_drain_timeout to 30, or to export HERMES_RESTART_DRAIN_TIMEOUT. This host already has 60 in the file the unit's home points at. That workaround is not an API-specific key. Chat turns share it. The issue says so, and the predicate says so. If you raise it so HTTP runs can finish, you also delay the interrupt of a chat turn by the same amount. Keep the number under TimeoutStopSec. On this unit that bound is 90.

There is still no agent.api_drain_timeout in this tree, and none on the tip I fetched. Setting the chat clock is the lever that exists. It is not the split the report asks for.

The tip has not split the clock

I fetched origin/main this morning. I did not fast-forward. Local HEAD is still 5000e299. The new tip is e1fdf003a668f97bf5a53d7675c1e70b1dcfec34, committed 2026-10-05 01:30:17 -0700. The subject is an update-path fix about a clashing local tag. I did not read that patch. A subject line is not a drain fix.

git rev-list --count HEAD..origin/main printed 2641. The reverse count is 0.

hermes --profile liam --version printed Hermes Agent v0.21.5+4599.g5000e29 (2026.9.24) · upstream e1fdf003 and Update available: 2426 commits behind. The banner SHA matches the tip I just fetched. The count does not. The profile source-check cache was written 2026-10-04 23:00:55 EDT. Its behind is 2426. Its target SHA is 27c02f6326. Commits from that SHA to the tip I fetched: 215. Last night's distance and this morning's distance are both real. They are not the same number. Three clocks on one Hermes version line is how to read that disagreement. I am not repeating it. I am telling you which number I would use before an update: the git rev-list count, 2641, not the cached 2426.

Do not hermes update this tree in an unplanned window. The predicate on the tip is still the two-deadline grouping. A search for api_drain_timeout on that tip returned nothing. Being 2641 commits behind does not mean the stop-path split has already been fixed upstream. I checked the line. It has not.

GitHub's latest release tag is a separate question from this fetch. I did not re-list releases this morning. Last night's note still had Latest at v2026.9.24. I am not treating the canary tags from that note as a stable release, and I am not installing them.

There is still no cron cancel

This is a different hole. Do not fold it into the drain bug.

#133057 is OPEN. I viewed it this morning. createdAt is 2026-10-05T02:50:51Z. updatedAt is 2026-10-05T06:34:43Z. Labels include type/feature, comp/cron, and P3. The title asks for hermes cron cancel. The body describes the reporter's wedged no-agent script job. That is not an incident on this host. I did not wedge a job to test it.

I printed hermes cron --help on this install. The verbs are:

list, create, add, edit, pause, resume, run, remove, rm, delete, status, runs, history, incidents, notepad, doctor, tick.

No cancel. No abort. No stop.

The parser in this checkout is hermes_cli/subcommands/cron.py, 12734 bytes, sha256 c07a27db9575afd77dfd7e1e0f0051241262706c11998d92f8b26369536f3222. The subparsers I read match that help text. pause help is "Pause a scheduled job." It takes a job id. It does not take a signal. I did not trace the pause handler through a live run. The issue says pause only blocks future fires. I am not upgrading that sentence into a runtime trace I did not do. What I can say is that the verb you would want, if a run is already in flight, is not on the help I printed.

A search of the fetched tip for add_parser("cancel" in hermes_cli/subcommands/cron.py returned nothing. The missing verb is not waiting in the 2641 commits I have not checked out. If a job wedges, this install still has no cancel command. Waiting for the scheduler timeout, or finding the process group by hand, is what the issue lists as the current options. I did not time out a job, and I did not send a signal to one.

pause will not save an HTTP run either. An API run is not a cron job. Canceling a cron id would not be the drain fix. They fail in different places. One is a missing CLI verb. The other is a deadline grouping inside stop().

What the log line is telling you

When a stop does happen, read the drain line as four counters, not one.

What you see	What it meant in the code I read
drain took 0.00s, timed_out=True, api_at_start greater than 0, active_at_start=0	API work was present. The chat clock had already expired. The wait loop did not run for that work. This is the report's signature. It is not in this host's log.
drain took about 60s, timed_out=True, active_at_start greater than 0, api_at_start=0	A chat-side agent hit the raised clock. This host has two of these, on 2026-09-19 and 2026-09-29.
drain took about 60s, cron_at_start=1, no API field	Older format. Cron was in flight. The duration matches a chat clock of 60 lifting the cron budget. It does not record an API count.
timed_out=False and every counter 0	Nothing to wait for. A 0.00s line in that shape is not the bug.
hermes --version says 2426 behind, git rev-list says 2641, banner SHA is the tip	The count is the profile cache from 2026-10-04 23:00 EDT. The SHA on the banner is live.

If you need HTTP runs to survive a planned stop, the lever in this tree is agent.restart_drain_timeout, set above 0 and under the stop bound your service manager will actually honor. On this unit that bound is 90, and the file already says 60. That change also delays chat interrupts. It is a workaround, not a separate API budget.

If you can restart in-band instead of stopping, the wait before stop() counts API runs, up to restart_after_turn_timeout. The default I imported is 1800. That wait still ends in the stop drain. Do not describe 1800 as the stop-path budget for API work.

If a cron job is wedged, do not look for hermes cron cancel on this install. It is not in the help text. pause schedules the future. It is not a kill.

What this does not mean

I did not stop the gateway. I did not reproduce the reporter's four events. I did not change config.yaml. I did not fast-forward the install. I did not run the pytest file. The probe was a test double of one method.

A green unit suite, including the API drain tests as they are written, would not by itself close this. Those tests do not call the default split. An architecture comment that cites #82161 is not a fix for the class that comment left on the chat clock. An open P1 is not a patch, and the body's "written, not submitted" line does not un-open the tracker state.

This host's 60-second lines are not a certificate that the next stop will wait 60 seconds for an API run. They are what the log recorded when the API counter was 0, or, in the July line, when the API counter was not printed. The method's behavior with a live API count and a timeout of 0 is the probe. The method's behavior with a live API count and a timeout of 60 is the same grouping, on a longer chat deadline. I measured the first. I read the second off the predicate. I did not pass 60 into the double.

Nothing here is a hardware change. Nothing here says the live tree should be updated this morning. The tip I fetched still groups API with the chat clock. Until that return changes, or until you set the chat clock on purpose and accept what that does to chat turns, a planned stop on the shipped default still interrupts an API run without waiting.

Sources

• NousResearch/hermes-agent#132989, viewed 2026-10-05. State OPEN. Body status line says written, not submitted.
• NousResearch/hermes-agent#133057, viewed 2026-10-05. State OPEN. hermes cron cancel request. Not reproduced here.
• This checkout 5000e29936df69d5209f7cf2eea8e5776cb4cbb1: gateway/run_shutdown.py lines 801-848 and 1848-1904, gateway/restart.py parsers and resolve_cron_drain_budget, hermes_cli/config_defaults.py drain keys, hermes_cli/subcommands/cron.py, tests/gateway/test_cron_drain_floor.py, tests/gateway/test_api_server_active_work_drain.py.
• In-process calls on a test double: _drain_active_agents(0.0, 30.0) for API-only, chat-only, and idle; _drain_active_agents(0.0, 0.2) for cron-only. Not a live stop.
• Parser and budget calls: defaults 0.0 / 30.0 / 1800.0; resolve_cron_drain_budget(0, 30, watchdog_delay=90) = 30.0; resolve_cron_drain_budget(60, 30, watchdog_delay=90) = 60.0.
• git fetch origin main this morning. Tip e1fdf003a668. git rev-list --count HEAD..origin/main = 2641. Predicate return still at line 842 on that tip. No api_drain_timeout. No cron cancel parser.
• hermes --profile liam --version after that fetch: banner SHA e1fdf003, cached behind count 2426, cache timestamp 2026-10-04 23:00:55 EDT.
• hermes cron --help on this install. Verb list as printed above.
• User unit file: KillMode=mixed, TimeoutStopSec=90, HERMES_HOME is the default home, no drain env var. systemctl --user show agreed the unit is active/running. Config in that home: restart_drain_timeout: 60. Gateway log counts as stated. I did not attach to the process.
• Prior note on the update flag and the 1800-second wait: Pass --no-gateway-restart when cron updates Hermes.