sjWtdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z ddl mZmZmZmZejeZgdZdZdZGd d eZe d d Gd dZd>dZd?dZd@dZdddAdZdBd"ZdCd%Z dDd)Z!d*dd d+dEd1Z"d*dd2dFd7Z#dd8dGd=Z$dS)Hu Unified deadline layer — one bounded-execution primitive, one timeout resolver. Phase 1 of the architectural fix for the timeout/hang backlog (https://github.com/NousResearch/hermes-agent/issues/85125). The tree currently carries at least six site-local deadline mechanisms, each built for one incident, none shared (tool_executor batch deadline, telegram ``_await_with_thread_deadline``, gateway turn lease, reasoning stale floors, ``human_wait_ceiling``, per-MCP-handler timeouts). Every new stall report grows that list by one. This module is the shared foundation the call sites migrate onto in later phases: * :func:`resolve_timeout` — one config-first resolution path for timeout values (``timeouts:`` section in config.yaml > legacy env var > default), so new surfaces stop inventing ``HERMES_*_TIMEOUT`` env vars (".env is for secrets only") and hardcoded literals stop ignoring user config (#63302, #53161, #43272 class). * :func:`clamp_timeout` — platform-safe clamping. Large user-supplied timeouts overflow ``time_t`` inside ``threading.Lock.acquire(timeout=...)`` / ``Thread.join(timeout=...)`` on macOS and kill whole tool batches (#83220). Clamping at the shared boundary fixes that class once, for every consumer. * :func:`run_bounded_async` — a wall-clock deadline for awaitables that does NOT depend on event-loop timers. ``asyncio.wait_for`` schedules its expiry on the loop; when the loop thread itself is blocked in a synchronous call (family A of the #84047 stall triage), every asyncio-based timeout in the process is silently disabled. This helper drives the deadline from a daemon ``threading.Timer`` (generalizing the proven telegram-adapter primitive) and abandons cancellation-shielded tasks instead of waiting for cancellation to complete. The telegram adapter's private copy (``plugins/platforms/telegram/adapter.py:_await_with_thread_deadline``) migrates onto this in Phase 2 of #85125 — do not let the two drift in the meantime; fix bugs here first. * :func:`run_bounded_sync` — the same contract for synchronous callables bounded from a synchronous context (daemon worker thread, abandoned on expiry). * :func:`kill_process_tree` — portable whole-tree termination so kill-on-timeout stops orphaning descendants (#71148, #59549, #84967, #68139 class). Existing site-local tree-kills that migrate onto this in Phase 4 of #85125: ``gateway/status.py`` (taskkill wrapper + psutil snapshot/reap pair) and ``tools/code_execution_tool.py`` (psutil recursive children kill). Design invariants: * Exceptions raised by the bounded operation propagate unchanged — callers keep their existing error handling. Only the *timeout* outcome is reified (as :class:`BoundedResult`), because that is the outcome the call sites keep getting wrong. * A timeout produced by this layer is OUR deadline, not the provider's. Callers that feed errors into ``agent/error_classifier.py`` should classify :class:`DeadlineExpired` distinctly from transport timeouts (the #59549 / #80323 misattribution class). * ``None`` timeout means unbounded, and non-positive resolved values are normalized to ``None`` (matching the existing ``HERMES_CONCURRENT_TOOL_TIMEOUT_S`` convention). ) annotationsN) dataclass)Any AwaitableCallableOptional)MAX_SAFE_TIMEOUT_S BoundedResultDeadlineExpired clamp_timeoutresolve_timeoutrun_bounded_asyncrun_bounded_synckill_process_treeg8~Ag@c$eZdZdZdfd ZxZS)r uNA deadline enforced by this layer expired. Distinct from transport/provider timeout types on purpose: when this is raised (or a :class:`BoundedResult` reports ``timed_out``), the timeout was Hermes's own bound — error classification must not attribute it to the provider (#59549 / #80323 misattribution class). labelstr timeout_sfloatcttd|dd|||_||_dS)Nzdeadline expired after z.1fzs: )super__init__rr)selfrr __class__s 2/home/agent/.hermes/hermes-agent/agent/deadline.pyrzDeadlineExpired.__init__qsA L9LLLULLMMM ")rrrr)__name__ __module__ __qualname____doc__r __classcell__)rs@rr r hsG##########rr T)frozenkw_onlycNeZdZUdZded<ded<ded<ded <d ed <dd ZdS)r uOutcome of a bounded operation. ``timed_out`` is the reified outcome; on completion ``value`` holds the operation's return value. Operation exceptions are never captured here — they propagate to the caller unchanged. bool timed_outrvaluer elapsed_sOptional[float]rrrreturncp|jr)t|jt|jpd|jS)z>Return ``value``, raising :class:`DeadlineExpired` on timeout.g)r&r rrrr')rs rraise_if_timed_outz BoundedResult.raise_if_timed_outs6 > L!$*eDN4Ic.J.JKK KzrN)r*r)rrrr __annotations__r,rrr r wsiOOOJJJJJJrr timeoutr)r*c|dS t|}n3#ttf$rtd|YdSwxYw||krtddS|dkrdSt |t S)u;Normalize a timeout value for platform wait primitives. * ``None`` stays ``None`` (unbounded). * Non-positive values become ``None`` (unbounded) — matching the existing ``HERMES_CONCURRENT_TOOL_TIMEOUT_S`` "0 disables the bound" convention. * Values above :data:`MAX_SAFE_TIMEOUT_S` are capped so they can never overflow ``time_t`` inside ``Lock.acquire`` / ``Thread.join`` on macOS (#83220). * Non-numeric values are treated as unset (``None``) with a warning rather than crashing the call path they were meant to protect. Nz(A)(A)rAkeyrrc|}|dD]&}t|tr||vrdS||}'|S)z=Walk ``a.b.c`` through nested dicts; return None when absent..N)splitr>r6)rArCnodeparts r_lookup_dottedrIsSD #$%% T)9)944Dz Kr)env_vardefaultrJ Optional[str]c tt|}|nt|ts= t |}||krt |Sn#t tf$rYnwxYwt d|||rrtj |d }|rI t t |S#t$rt d||YnwxYwt |S)u[Resolve a timeout in seconds for a dotted config key. Precedence (established by the ``providers.*.request_timeout_seconds`` pattern — config wins over the legacy env var): 1. ``timeouts.`` in config.yaml (dotted key walks nested maps, e.g. ``tools.concurrent_batch`` reads ``timeouts: {tools: {concurrent_batch: ...}}``) 2. ``env_var`` when set and non-empty (legacy bridge — internal mechanism and back-compat only; new surfaces must not grow new user-facing ``HERMES_*`` timeout env vars) 3. ``default`` The winning value is passed through :func:`clamp_timeout`, so ``0`` or a negative value means "unbounded" and oversized values are made platform-safe. Invalid (non-numeric) config/env values fall through to the next source with a warning instead of breaking the protected path. Nz6timeouts.%s: invalid value %r in config.yaml; ignoringzinvalid %s=%r; ignoring) rIrBr>r%rr r1r2r3r4osgetenvstrip)rCrKrJrawr'env_raws rr r s2. *,,c 2 2C  #t$$  c E>>(///"z*    OQTVYZZZL)GR((..00  L L$U7^^444 L L L8'7KKKKK L  ! !!s##AA.-A.9C&C>=C>task'asyncio.Future[Any]'Nonec~ |s|dSdS#t$rYdSwxYw)zGObserve an abandoned task's outcome so it never logs 'never retrieved'.N) cancelled exceptionr?)rTs r_consume_abandonedrZs[ ~~  NN             s (. << on_abandonCallable[[], Awaitable[Any]]cK |d{VdS#t$r tddYdSwxYw)zGRun abandonment cleanup fully fire-and-forget (its failures swallowed).Nzdeadline abandon-cleanup failedTr:)r?r3r@)r[s r_run_abandon_cleanupr^ soGjll GGG 6 FFFFFFGs&AArrrctd||t tjddS#t $r tddYdSwxYw)Nu[deadline] %r deadline (%.0fs) expired but the event loop has not processed the expiry after a further %.0fs — the loop thread appears BLOCKED in a synchronous call, which is why no asyncio timeout can fire. Dumping all thread stacks to stderr to identify the blocking frame.T) all_threadsz"faulthandler traceback dump failedr:)r3r4_LOOP_BLOCKED_DUMP_GRACE_S faulthandlerdump_tracebackr?r@)rrs r_dump_blocked_loop_diagnosticsrds NN  "   J#555555 JJJ 9D IIIIIIJs;&A%$A% operation)rr[dump_on_blocked_loop awaitableAwaitable[Any]&Optional[Callable[[], Awaitable[Any]]]rfr%cKt|tj}0|d{V}td|tj|z dSt j|}t jtj d fd d fd }d fd } tj |} d | _ | d} |r8tj tz| } d | _ |  t j|htj d{V\} } nB#tj$r0||t&wxYw|| vrs|d{V}td|tj|z | | | S||t&|;t jt-|}|t&t.d td dtj|z | | | S#| | | wxYw) u>Await ``awaitable`` under a wall-clock deadline independent of loop timers. On completion returns ``BoundedResult(timed_out=False, value=...)``; exceptions from the operation (including ``asyncio.CancelledError`` from a caller cancelling *us*) propagate unchanged. On timeout the underlying task is cancelled and **abandoned** — we do not await cancellation completion, because cancellation-shielded scopes (anyio, httpcore init, MCP SDK teardown) are exactly the paths that wedge forever. ``on_abandon`` (zero-arg callable returning an awaitable) is scheduled as detached best-effort cleanup for the half-built state the abandoned task may leave behind. Returns ``BoundedResult(timed_out=True, value=None)``. ``timeout=None`` (or a non-positive resolved value) awaits unbounded. NFr&r'r(rrr*rVcsddSdSN)setdone set_result)deadlineloop_processed_expirysr _mark_expiredz(run_bounded_async.._mark_expiredGsI!!###}} &    % % % % % & &rc2dSrm)call_soon_threadsafe)rsloopsr_expire_from_threadz.run_bounded_async.._expire_from_threadLs !!-00000rcTstdSdSrm)is_setrd)rrrrsr_watchdog_checkz*run_bounded_async.._watchdog_checkOs7$++-- = *5) < < < < < = =rT) return_whenz3[deadline] %r timed out after %.1fs; task abandonedr*rV)r time monotonicr asyncio ensure_futureget_running_loop create_future threadingEventTimerdaemonstartrawaitFIRST_COMPLETEDCancelledErrorcanceladd_done_callbackrZrornr^r3r4)rgr/rr[rfrr'rTrwrztimerwatchdogro_cleanuprsrqrvrrrs ` @@@@@rrr%s.g&&I N  EuET^EUEUX]E]imuz{{{{   + +D  # % %D'+'9'9';';H%O--&&&&&&& 1111111======== OI': ; ;EEL KKMMM*.H? 2 2O  !$ #Lx g.EGD!!%    KKMMM  " "#5 6 6 6   4<<==?? "!!!JJJJJJE 5IYIY\aIamvDEEE     OO    !!####  1222  !+,@,L,LMMG  % %&8 9 9 9LeU^___t44>CSCSV[C[gpx}~~~    OO    !!####     OO    !!####s'/+EL$?FAL$7B.L$$AM$)r on_timeoutfnCallable[[], Any]rOptional[Callable[[], None]]c t|}tj}|0tdtj|z d|Sit j d fd }t j|d|d }| |s~t d |||8 |n,#t$rt d d YnwxYwtddtj|z ||Sd vrd td dtj|z ||S)uRun ``fn`` in a daemon worker thread under a wall-clock deadline. On completion returns its value (exceptions re-raised in the caller). On expiry the worker thread is **abandoned** (daemon, so it cannot block interpreter exit), ``on_timeout`` (if given) runs best-effort in the caller's thread — e.g. to mark a backend suspect or kill a subprocess — and ``BoundedResult(timed_out=True)`` is returned. Intended for infrequent, seconds-scale blocking backend calls. Do NOT use per-item in hot loops: each call spawns a thread, and every timeout permanently leaks an abandoned daemon thread — a wedged backend called in a retry loop would accumulate them. ``timeout=None`` (or non-positive) blocks until ``fn`` returns. NFrkr*rVc d<n#t$r}|d<Yd}~nd}~wwxYwdS#wxYw)Nr'exc) BaseExceptionrn)rboxrors r_workerz!run_bounded_sync.._workersp 244CLL   CJJJJJJ  HHJJJJJDHHJJJJs% A *%A*AAz deadline-T)targetnamerz5[deadline] %r timed out after %.1fs; worker abandonedz#deadline on_timeout callback failedr:rr'r|)r r}r~r rrThreadrrr3r4r?r@r=) rr/rrrrrthreadrros ` @@rrrs,g&&I N  EuBBDDDNDTDTW\D\hltyzzzzC ?  D 000F LLNNN 99Y  NPUW`aaa  ! S  S S S BT RRRRR St44>CSCSV[C[gpx}~~~~ ||%j 50@0@DNL\L\_dLdpyBG H H HHs C""&D  D )sigpidintr Optional[int]c tjdkr ddlm}|}n#t$rd}YnwxYw t jddddt|gdd d | }|jdkS#t$r!t d |d Yd SwxYwddl }||j }g} ddl }|t|d}n#t$rg}YnwxYwd } t#j|} n#t&t(t*f$rd} YnwxYw | | |krt#j| |nt#j||d}n?#t&$rYn3t(t*f$r t d|d YnwxYw|D]>} | r| |d}/#t$rY;wxYw|S)uYTerminate ``pid`` and all its descendants, portably. Kill-on-timeout that signals only the direct child orphans process trees (cron scripts, in-container shells, browser daemons — #71148 class). * Windows: ``taskkill /F /T`` terminates the tree (``sig`` ignored; Windows has no equivalent). Console-window flash is suppressed via ``windows_hide_flags`` and the exit code is checked, so a dead or inaccessible PID reports ``False`` like the POSIX path. * POSIX: the descendant set is snapshotted via psutil (a hard dependency) BEFORE any signal — once the parent dies its children are reparented and can no longer be found by a parent walk. Then the process group is signalled when ``pid`` leads one (covers grandchildren in the same session in one syscall), and every snapshotted descendant is signalled individually — which also reaches descendants that created their OWN sessions (a child that called ``setsid``, exactly what user shell commands do; see tools/environments/base.py). ``sig`` defaults to ``SIGKILL``. psutil's identity-aware ``Process`` (PID + create time) means a recycled PID is never signalled. Returns True when the target (or any of its tree) was signalled, False when the process was already gone or every termination call failed. win32r)windows_hide_flagstaskkillz/Fz/Tz/PIDTF)capture_outputr/check creationflagsz-kill_process_tree: taskkill failed for pid %sr:N) recursivez+kill_process_tree: signal failed for pid %s)sysplatformhermes_cli._subprocess_compatrr? subprocessrunr returncoder3r@signalSIGKILLpsutilProcessrchildrenrOgetpgidProcessLookupErrorPermissionErrorOSErrorkillpgkill is_running send_signal) rrrrproc_signal descendantsr signalledpgidchilds rrrs2 |w  H H H H H H..00MM   MMM  >T4S:#+ D?a' '    LLH#X\L ] ] ]55  {oK nnSXX..77$7GG   Iz#  9 X    IdC GC         W %XXX BCRV WWWWWX  !! !!!#&&&     H  sk# 225A,,'BB*:C%% C43C4:DD+*D+/5E%% F!1-F! F!)+G G"!G")r/r)r*r))r*r6)rAr6rCrr*r)rCrrKr)rJrLr*r))rTrUr*rV)r[r\r*rV)rrrrr*rV) rgrhr/r)rrr[rirfr%r*r ) rrr/r)rrrrr*r )rrrrr*r%)%r __future__rrrbloggingrOrrrr} dataclassesrtypingrrrr getLoggerrr3__all__r ra TimeoutErrorr r r rBrIr rZr^rdrrrr.rrrs<<|#"""""   !!!!!!555555555555  8 $ $   &"! # # # # #l # # # $%%%&%*****@     " /"/"/"/"/"/"x    GGGGJJJJ*9=!% Y$Y$Y$Y$Y$Y$H/3 5H5H5H5H5H5Hx9=________r