Worker Management
Purpose
Event-driven worker management system enabling deployment and execution of workers written in multiple languages (Python, TypeScript, C#, .NET DLL, NativeAOT) that subscribe to Dapr pub/sub topics and process CloudEvents. Every process subscribed to a topic receives every message on it (one consumer group per sidecar), and each worker runs in exactly one process; see the Cross-Instance Coordination and horizontal scalability requirements below.
Topic and state-key naming for worker lifecycle events is defined in pubsub-topics/spec.md and built by the naming library from the virtufin.service.workmanager fragment. The lifecycle topic is sys.workmanager.worker.lifecycle; worker records live at sys.workmanager.worker.<worker_id>.
Requirements
Requirement: gRPC-only Control API
WorkManager SHALL expose business control operations only as the gRPC service virtufin.WorkManager; its HTTP port SHALL serve only health/readiness probes.
Scenario: Plain HTTP request
- WHEN a client sends a plain (non-gRPC) HTTP request for anything other than a probe
- THEN no worker-management operation SHALL execute
- AND clients SHALL use gRPC directly or gateway invocation over gRPC
Requirement: Worker Lifecycle Management
The WorkManager SHALL support creating, starting, stopping, and deleting workers. Workers SHALL persist their configuration and restore automatically on service restart.
Scenario: Create and start a worker
- WHEN a client creates a worker with a topic subscription and code payload, then starts it
- THEN the worker SHALL be persisted to the state store, and once its code is loaded it SHALL be subscribed to the topic and set to Running status
Scenario: Hot-reload worker code
- WHEN a client updates a running worker's code
- THEN the worker SHALL switch to the new code without restarting the service: a started worker is stopped (stop token,
StopAsync, lease revoked), reloaded and started again with a fresh context, never reloaded in place - AND a worker whose config says
autoreload=falseSHALL NOT be reloaded on a library release
Scenario: Auto-recovery on restart
- WHEN a Runtime pod restarts, or a worker's actor was idle-deactivated
- THEN the worker's next actor activation SHALL reload its code and resubscribe its topics from its Runtime record (
sys.workmanager-runtime.worker.<worker_id>) - AND the WorkManager SHALL touch the actor of every running worker on start-up and periodically (
RuntimeWorkerPokeService, default every 60 s), so a worker without inbound traffic is reactivated too, and a touch SHALL make sure the worker is started (a lapsed lease is claimed again)
Scenario: Worker version history
- WHEN a worker's code is updated
- THEN the previous code version SHALL be preserved in a versioned history
Scenario: Create returns before the code is loaded
- WHEN a client creates a worker
- THEN the WorkManager SHALL persist it and return with status
Loadingwithout waiting for the code fetch, dependency resolution or engine load - AND it SHALL load the code in the background and set the status to
Stopped(orRunning, see deferred start) once loaded, orLoadFailedon failure
Scenario: Start a worker that is still loading
- WHEN a client starts a worker whose status is
Loading - THEN the WorkManager SHALL accept the request and start the worker (subscribe it and set it to Running) as soon as loading completes
Scenario: Start a worker whose load failed
- WHEN a client starts a worker whose status is
LoadFailed - THEN the WorkManager SHALL reject the request with
FAILED_PRECONDITION
Requirement: Service Architecture
The WorkManager and its Runtime SHALL use the virtufin-api gRPC for all pubsub and state operations. They SHALL NOT call any Dapr pubsub or state API directly. The Dapr sidecar SHALL remain in use for actors, the secret store (Worker Secrets by Reference), service-invocation mTLS, distributed tracing, and metrics.
Scenario: Service persists worker state
- WHEN the WorkManager or the Runtime saves or retrieves a worker record
- THEN it SHALL call the API's
StategRPC service - AND it SHALL NOT call any Dapr state API directly
Scenario: Runtime subscribes to a worker topic
- WHEN the Runtime subscribes to a worker's input topic
- THEN it SHALL call
Pubsub.Subscribeon the virtufin-api - AND it SHALL NOT call
DaprPublishSubscribeClient.SubscribeAsyncdirectly
Requirement: Worker Lifecycle Events
The WorkManager SHALL publish worker lifecycle events to the sys.workmanager.worker.lifecycle topic, with ce-type = com.virtufin.workmanager.worker.lifecycle and ce-subject = worker/<worker_id>; the payload's state field SHALL name the transition, and the payload SHALL carry worker_id and topic. Names come from the naming library (virtufin.service.workmanager fragment, pubsub-topics/spec.md). Each event SHALL be published once, when its transition happens; a call or an actor activation that changes no state SHALL publish nothing. The states degraded, stopping and unknown are reserved and SHALL NOT be published. The state status carries a worker's own report (ReportStatusAsync: payload status, detail) or the Runtime's lease loss (status = lease-lost); it is not a catalog transition and is also kept in the worker's Runtime record.
Scenario: Worker registered
- WHEN
CreateWorkerhas accepted and persisted a worker - THEN the WorkManager SHALL publish a lifecycle event with
state = registered, before any event of its load
Scenario: Worker loaded
- WHEN a worker's code and dependencies finish loading
- THEN the WorkManager SHALL publish a lifecycle event with
state = loadedwithworker_idand the resolvedVirtufin.*library versions (libraries) in the data payload
Scenario: Worker load failed
- WHEN loading a worker's code or dependencies fails
- THEN the WorkManager SHALL publish a lifecycle event with
state = load-failedwitherror_typeanderror_messagein the data payload
Scenario: Worker started
- WHEN
StartWorkerstarts a stopped worker, or a worker started while loading finishes loading - THEN the WorkManager SHALL publish
state = starting, thenstate = onlineonce the worker is subscribed
Scenario: Start fails
- WHEN the Runtime fails to start a loaded worker
- THEN the WorkManager SHALL publish
state = starting, thenstate = failedwitherror_typeanderror_message, and the worker SHALL stayStopped
Scenario: Start of a running worker
- WHEN
StartWorkeris called on aRunningworker, or on aLoadingone (the start is only recorded) - THEN no lifecycle event SHALL be published for that call
Scenario: Worker stopped
- WHEN
StopWorkerstops aRunningworker, orDeleteWorkerdeletes one - THEN the WorkManager SHALL publish
state = stopped; stopping a worker that is not running SHALL publish nothing
Scenario: Worker deleted
- WHEN
DeleteWorkerhas removed a worker - THEN the WorkManager SHALL publish
state = deleted, after thestoppedof a worker that ran - AND a
DeleteWorkerthat fails SHALL publish nothing
Scenario: Worker updated
- WHEN
SetTagorSetTagshas saved a worker's tags, orLoadCodehas loaded new code into it - THEN the WorkManager SHALL publish
state = updatedwithchanged(["tags"]or["code"]) and the new value as stored:tags(the whole tag map) orcodeurl(null for inline code)
Scenario: Following the workers without polling
- WHEN a client keeps a list of workers current
- THEN every change of what
ListWorkersreturns SHALL have a lifecycle event (registered,deleted,updatedand the state transitions), so the client readsListWorkersonce when its lifecycle subscription is live and then applies the events - AND because delivery is best effort, the client SHALL read
ListWorkersagain after a resubscription
Scenario: Self-heal fails
- WHEN a Runtime actor activation cannot reload or resubscribe a
Runningworker - THEN the Runtime SHALL persist the worker as faulted, publish
state = failedwitherror_typeanderror_messageonce, and fail the activation so that a later activation retries - AND a later activation that fails again SHALL publish nothing, and the one that succeeds SHALL publish
state = online
Scenario: Healthy reactivation
- WHEN a Runtime actor activation reloads and resubscribes a worker that was not faulted
- THEN no lifecycle event SHALL be published
Scenario: Worker status report
- WHEN a started worker calls
context.ReportStatusAsync(state, detail) - THEN the Runtime SHALL publish a lifecycle event with
state = status,status = <state>anddetail, and keep both in the worker's Runtime record
Requirement: Worker Dead Letters
When a worker fails on a message (it throws, or a native worker returns an error status), the Runtime SHALL NOT retry it. It SHALL publish the original input CloudEvent exactly once to dlq.<topic> of the topic it arrived on (naming library DeadLetter), with the failure in extension attributes defined once in Virtufin.Worker.Host.WorkerAttributes, and SHALL then deliver the worker's next message.
Scenario: Worker throws on a message
- WHEN a worker throws while processing a message from topic
T - THEN exactly one dead letter SHALL be published to
dlq.T: the input event as it arrived, plusdlqworkerid,dlqactortype(the worker's Runtime actor type),dlqerrortype,dlqerrormessage(at most 1024 characters, no stack trace) anddlqfailedtime - AND the message SHALL NOT be redelivered to the worker, and the worker SHALL receive the next message on its topics
Scenario: Native worker error status
- WHEN a native worker's
Processreturns1or another non-zero status - THEN the message SHALL be dead-lettered exactly as if the worker had thrown
Scenario: Dead-letter publish fails
- WHEN publishing the dead letter fails
- THEN the Runtime SHALL log the failure at error level and continue with the next message, without retrying the publish
Scenario: No secrets in the dead letter
- WHEN a worker created with secrets fails on a message
- THEN the dead letter SHALL be the event as it arrived, before the worker's config and secret values were added
Requirement: Worker Wire Protocol
Every hop that carries a CloudEvent between the WorkManager, its Runtime and a
worker SHALL use the CloudEvents Protobuf format (io.cloudevents.v1): a request
SHALL be one CloudEvent message, a response one CloudEventBatch message whose
events are published in order (an empty batch means no output). Binary data SHALL
travel verbatim in binary_data. Errors SHALL be signalled by the transport, never
inside a CloudEvent. One codec (CloudEventProtoCodec, Virtufin.Worker.Host)
SHALL implement the mapping for every .NET hop.
Scenario: Binary data round trip
- WHEN a worker returns a CloudEvent whose data is bytes (e.g. an order book image)
- THEN every hop SHALL carry those bytes unchanged in
binary_data, without base64
Scenario: Subprocess frames
- WHEN the Runtime talks to a subprocess engine (Python, C# source, TypeScript)
- THEN each frame SHALL be a 4-byte big-endian length, a 1-byte frame type and a
payload:
processcarries aCloudEvent,resultaCloudEventBatch,errora UTF-8 message, andready,ping,pongandshutdownan empty payload
Scenario: Python workers see the same event
- WHEN the Python runner decodes a
processframe - THEN the worker's
Process(event)SHALL receive the CloudEvents-JSON-shaped dict it received before (dataparsed for JSON content,data_base64for bytes), and the runner SHALL encode the returned dict(s) back into aCloudEventBatchusing only the Python standard library
Scenario: TypeScript workers see the same event
- WHEN the TypeScript engine's Node bridge decodes a
processframe - THEN the worker's
Process(event)SHALL receive the same dict form a Python worker receives (every attribute a top-level key, text data indata, binary data base64 indata_base64, timestamps as RFC 3339 text), and the bridge SHALL encode the returned event(s) back into aCloudEventBatchwith no dependency beyond Node's standard library
Scenario: Subscription to worker actor
- WHEN the Runtime's topic subscription hands a message to the actor hosting a worker
- THEN it SHALL call the actor's
ProcessAsyncwith theCloudEventbytes and receive theCloudEventBatchbytes
Requirement: Polyglot Worker Engines
The WorkManager SHALL support executing worker code in multiple languages via a pluggable engine architecture. Built-in engines SHALL include Python (subprocess), TypeScript (a Node.js subprocess), C# source (Roslyn compilation in a CSharpSource.Bridge subprocess), pre-compiled .NET DLL (the default DotNetDllEngine via hostfxr + AssemblyLoadContext), and a native DLL engine (NativeDllEngine, see the Native DLL Engine / Native Worker ABI requirements below). Every engine SHALL be hosted by a WorkManager.Runtime pod through its per-engine actor; the WorkManager itself SHALL NOT run worker code.
Scenario: Python worker execution
- WHEN a worker with a Python engine processes a message
- THEN the worker code SHALL execute in a dedicated Python subprocess with a configurable timeout
Scenario: TypeScript worker execution
- WHEN a worker with MIME type
text/x-typescript(one ES module) orapplication/x-npm(annpm packtarball whosemainis the module) processes a message - THEN the module SHALL run in a dedicated
nodesubprocess speaking the subprocess protocol, its TypeScript type-stripped by Node without a compile step (erasable syntax only), with the worker'sStart(ctx)/Process(event)/Stop()exports or a default-exported worker object or class (start/process/stop) as its entry points, any of them optional and possibly asynchronous - AND a module that fails to load SHALL fail the load with Node's error, without waiting for a timeout
Scenario: TypeScript worker packages
- WHEN a TypeScript worker is loaded
- THEN the engine SHALL install its dependencies with npm into the worker's own directory: a module's bare import specifiers (Node built-ins and relative imports excluded) at their latest versions, a package's
dependencieswith every@virtufin/*one at its latest version - AND
@virtufin/*SHALL resolve only from the private registry (npm scope mapping) with the Runtime's registry token passed through the environment and never written to disk; install scripts SHALL NOT run - AND the Runtime's registry credentials SHALL NOT be in the worker subprocess's environment
Scenario: C# source worker execution
- WHEN a worker with a C# source engine processes a message
- THEN the code SHALL be compiled at runtime via Roslyn and executed in the
CSharpSource.Bridgesubprocess
Scenario: C# source worker packages
- WHEN a C# source worker starts with
#r "nuget: <Id>",#r "nuget: <Id>, <version-or-range>"or<Id> [<version>]directives - THEN the engine SHALL resolve those packages and their dependencies with the same installer, feeds, credentials and version policy as DLL workers (
Virtufin.*at the newest release in range), compile against them, and load host-shared assemblies (Virtufin.Worker.Host, CloudEvents,Virtufin.Api.Client) from the host so the worker sees the host's types - AND an unresolvable package, a non-NuGet
#r, or a compile error (with its line number) SHALL fail the load
Scenario: C# source worker entry point
- WHEN a C# source worker defines exactly one non-abstract class implementing
IWorkerwith a public parameterless constructor - THEN the engine SHALL run it the way the DLL engine runs a DLL worker, publishing every output in order
- AND a source without such a class MAY instead declare a
CloudEvent ProcessAsync(CloudEvent)method; two such classes SHALL fail the load
Scenario: DLL worker execution (in-process, default)
- WHEN a worker with MIME type
application/x-dotnet-dllprocesses a message - THEN the worker DLL SHALL be loaded into an isolated
AssemblyLoadContextand executed inside the hosting Runtime process
Scenario: Engine extension
- WHEN a new engine implementation conforms to the
IEngineinterface - THEN it SHALL be added as a Runtime actor type and mapped from its MIME type (
RuntimeActorTypeNames), without changing how the WorkManager routes workers
Requirement: Event-Driven Message Processing
Workers SHALL subscribe to Dapr pub/sub topics and process incoming CloudEvents. Each output CloudEvent's publishtopic extension, when present, SHALL determine its publish topic (the extension is removed before publishing); otherwise its type field SHALL.
Scenario: Message processing
- WHEN a CloudEvent is published to a topic a worker subscribes to
- THEN the worker's code SHALL be invoked with the event data and any output events SHALL be published to the topic named by their
publishtopicextension, or else theirtypefield
Scenario: Several outputs from one input
- WHEN a multi-output worker returns several CloudEvents for one input
- THEN every one of them SHALL be published, in the order returned, each to its own topic by the rule above
- AND an empty result SHALL publish nothing
Scenario: Shared per-event rules and conformance vectors
- WHEN any host runs a worker on an event (the WorkManager.Runtime with any engine, or an in-process local host such as the devkit's
LocalWorkManager) - THEN it SHALL apply the per-event rules of
Virtufin.Worker.Host.WorkerEventRules, not its own copy:PrepareInput(a missingdatacontenttypebecomesapplication/json, the worker's config fills only attributes the event leaves out, the worker's secrets are set over everything),FinishOutputs(a missing id becomes a fresh one, a missing source the input's, secret attributes are removed, thetypehas no default),Route(by the rule above; an output with neither is not published) andDeadLetter(the dead letter of the Worker Dead Letters requirement) - AND the host SHALL pass the conformance vectors
conformance/worker-event-rules.v1.jsonshipped in theVirtufin.Worker.Hostpackage (codec cases and rule vectors), using a vector'sexpect_pythonwhere the worker runs in the Python or the TypeScript engine (the dict-form wire path both share)
Scenario: Concurrent message handling
- WHEN multiple messages arrive for the same worker
- THEN they SHALL be processed concurrently (the engine's
ProcessAsyncis re-entrant; the worker author is responsible for internal synchronization)
Requirement: Worker-Addressed Topics
A CreateWorker topic MAY contain the placeholder {worker_id} (the naming library renders it for wrk.<worker_type>.command.<worker_id>). The WorkManager SHALL replace every occurrence with the id it mints for the worker, before the worker is registered anywhere, and SHALL persist and report the expanded topic. A topic names what its data is, not who happens to consume it: commands meant for one worker instance SHALL be addressed by that instance's id, never by a caller-side run or session tag.
Scenario: Command topic addressed by the worker
- WHEN a client creates a worker with topic
wrk.httpget.command.{worker_id} - THEN the worker SHALL be subscribed to
wrk.httpget.command.<its id>, andGetWorker/ListWorkersSHALL report that expanded topic - AND the client SHALL derive the topic from the id
CreateWorkerreturns (to target a trigger or publish commands)
Scenario: Orphans stay isolated
- WHEN a client dies before deleting its workers and triggers, and a later client creates workers of the same kind
- THEN the orphaned triggers SHALL only reach the orphaned workers, because each worker's command topic is its own
Scenario: Commands across a move
- WHEN commands are published on a running worker's command topic while the worker moves to another Runtime pod (a rebalance, a pod replaced or killed, a reopened stream), when no pod is subscribed to it
- THEN the activation the worker continues on SHALL receive each of them exactly once, in order, after the commands it had not yet handled: the Runtime SHALL persist a cursor per worker (
sys.workmanager-runtime.worker.<id>.commands: since when, and the ids handled since), saved after each command it handles (processed or dead-lettered), and SHALL open a continuing worker's command topic with the API'sreplay_sincefrom it (less a clock margin), skipping what is listed - AND a worker that is stopped and started again, or newly started, SHALL start at the tail (its cursor is deleted on stop and delete); its other input topics SHALL follow Scenario: Inputs across a move
- AND a replay the API refuses SHALL be logged as an error naming the worker and the time, and the topic SHALL open at the tail
- AND a command handled but not yet recorded when its pod died MAY be delivered again (at least once at a crash)
Scenario: Inputs across a move
- WHEN events are published on a running worker's input topics (any topic but its command topic) while the worker moves to another Runtime pod (a rebalance, a pod replaced or killed, a reopened stream), and the worker's config does not say
replay=off - THEN the activation the worker continues on SHALL receive each of them exactly once, in topic order, before the later live ones: the Runtime SHALL persist per worker the cursor of each input topic (
sys.workmanager-runtime.worker.<id>.inputs: the id of the last event finished, and a time at or before its stream entry), batched (at most one State write per worker every few hundred milliseconds, never one per event) and settled when the worker stops on that pod while it still holds the lease; and SHALL catch a continuing worker up per topic from its cursor (the API'sreplay_sincewithreplay_only, less a clock margin, skipping through the cursor's event), merged with the topic's shared live stream by CloudEvent id, without affecting other workers on the topic - AND a catch-up the API refuses (too many entries or bytes retained, no stream history), one that fails, or a replay that no longer holds the cursor's event SHALL be logged as a warning and published as a lifecycle event with state
gap(topic,from,to,reason), and the worker SHALL continue live - AND a worker whose config says
replay=offSHALL start every input topic at the tail after a move (workers that resynchronise themselves, such as the order-book workers);StopWorkerandDeleteWorkerSHALL delete the input cursors, so a stopped or new worker starts at the tail - AND an event finished but not yet recorded when its pod died MAY be delivered again (at least once at a crash); workers SHALL be written to tolerate a repeated input event or document that they do not
Requirement: Trigger Ownership
A trigger MAY name the worker it exists for (owner_worker_id on Trigger.ScheduleTrigger). When the WorkManager deletes a worker it SHALL delete every trigger owned by that worker, after the worker itself is deleted, so a trigger never outlives the worker it drives. A failure to delete owned triggers SHALL be logged and SHALL NOT fail the worker deletion. Triggers without an owner SHALL be unaffected.
Scenario: Deleting a worker deletes its triggers
- WHEN a worker that owns triggers is deleted
- THEN its owned triggers SHALL be deleted and SHALL stop firing
- AND triggers owned by other workers, and unowned triggers, SHALL remain
Requirement: Cross-Instance Coordination
Messages SHALL reach Runtime processes by per-sidecar delivery, not competing consumers: the pub/sub component gives every sidecar its own consumer group (consumerID: {podName}), so every process that holds a subscription for a topic receives every message on it (Pub/Sub Topics spec, "Subscription Implementation Pattern"). A process subscribes to a topic only while it hosts a worker for it, and each worker is hosted by exactly one process (Ownership via Dapr Actor Placement), so each worker processes each message exactly once. A message that reaches a process with no worker registered for the topic (e.g. one that left between delivery and dispatch) SHALL be dropped, acked with no redelivery (an explicit business-logic skip per the Pub/Sub Topics spec's "Dead-Letter Policy on Subscribe Exceptions" requirement, not a processing failure).
Scenario: Message lands on a replica with no matching worker
- WHEN a message reaches a Runtime process that has no worker registered for that topic
- THEN the message SHALL be dropped and acked with no redelivery (an explicit business-logic skip, not a processing failure)
- AND no re-routing SHALL occur: every process hosting a worker for the topic receives the message itself
Scenario: Every subscribed process receives every message
- WHEN two Runtime pods each host a worker subscribed to topic
orders - THEN each process SHALL receive every message published to
orders, and each worker SHALL process it once - AND no message SHALL be load-balanced between the processes
Requirement: Ownership via Dapr Actor Placement
Each worker SHALL be a Dapr actor in the WorkManager.Runtime, of the actor type of its engine (RuntimeActorTypeNames, from the MIME type), keyed by the worker id. Placement keeps at most one activation per id, so exactly one Runtime process hosts a worker. The WorkManager is the catalog and router: it SHALL host no worker code and SHALL drive every worker operation (create, load, start, stop, delete, code reload) by calling the worker's actor. There SHALL be no instance directory, peer forwarding or reclaim; a worker record's InstanceId is informational.
Scenario: Operation on any WorkManager replica
- WHEN a worker RPC (
StartWorker,StopWorker,DeleteWorker,LoadCode) reaches any WorkManager replica - THEN the replica SHALL call the worker's actor, and placement SHALL deliver the call to the Runtime process hosting it
Scenario: Activation on another pod
- WHEN a worker's actor is activated on a Runtime pod that never loaded it (failover, rebalance, idle reactivation)
- THEN the activation SHALL load the worker from its Runtime record and resubscribe its topics if it was running
Scenario: Deactivation stops delivery on the old host
- WHEN a worker's actor is deactivated on a Runtime pod (placement moved it, idle timeout, pod shutdown)
- THEN that pod SHALL unsubscribe the worker's topics (closing a topic's stream when no other worker on the pod uses it) and release the worker's engine, without changing the worker's Runtime record
- AND messages the pod still receives on those topics SHALL NOT be delivered to the worker's actor from that pod, so a worker moved by placement processes each message once, on its new host
Scenario: Graceful drain on Runtime shutdown
- WHEN a Runtime pod is asked to stop (SIGTERM)
- THEN it SHALL first stop every started worker (stop token,
StopAsync, lease revoked) within a bounded budget, then close its topic streams, so no new message enters a worker's inbox - AND it SHALL then let each worker finish the messages it already holds, in order, and publish everything in its outbox (responses and dead letters), within a bounded drain budget
- AND it SHALL then release every engine, and only then exit; what is still queued when the budget runs out is dropped and logged with a count per worker
- AND the deployment SHALL give the pod a termination grace period that covers the drain and keep the sidecar available for it
Requirement: Worker Management Client Contract
There SHALL be one client contract for managing workers, IWorkManagerClient (Virtufin.WorkManager.Client), implemented by the cluster client (GatewayWorkManagerClient, through the api gateway) and by the in-process host (LocalWorkManager). Both SHALL accept the same WorkerSpec (CodeSource, mime type, topics, config, tags, secrets) and expose the same state model (Loading, Stopped, Running, LoadFailed) and operations (create, load code, start, stop, delete, list with tag filters, tags). CodeSource SHALL be a URL (with an optional credential), inline content, or an already constructed worker instance; the instance form SHALL be accepted by in-process hosts only. Local hosts SHALL load URL and content sources through pluggable IWorkerLoaders that mirror the Runtime's engines (Virtufin.WorkManager.Engines.Local: nupkg and C# source).
Scenario: Same spec on both hosts
- WHEN a
WorkerSpecwith a nupkg URL or C# source content is deployed through the cluster client and through a local host with the engine loaders - THEN both SHALL load the same code, apply the same per-event rules and produce outputs identical by content
Scenario: Instance rejected by the cluster
- WHEN a
CodeSource.Instanceis given to the cluster client - THEN it SHALL fail with a not-supported error before any RPC is sent
Scenario: Start while loading
- WHEN
StartWorkerAsyncis called on a worker whose code is still loading, on either host - THEN the start SHALL take effect once the load finishes, and SHALL fail with a load-failed error if the load fails
Requirement: Worker Code Sources
Workers SHALL accept code from inline Base64-encoded content or from HTTP(S) URLs. URL-based sources SHALL be validated against SSRF and host allowlist constraints as defined in the cross-cutting security requirements. URL-based sources MAY carry a per-request credential (see Worker Source Authentication) used only for that fetch.
Scenario: Inline code worker
- WHEN a worker is created with a Base64-encoded code string
- THEN the decoded code SHALL be passed to the engine for execution
Scenario: URL code worker
- WHEN a worker is created with a URL pointing to code
- THEN the code SHALL be fetched, SHA-256 verified, and passed to the engine
Scenario: Blocked URL
- WHEN a worker's code URL targets a private IP or a host not in the allowlist
- THEN the worker creation SHALL fail with a security rejection
Requirement: Python Sandboxing
Python worker subprocesses running raw source code (text/x-python) SHALL run with restricted interpreter flags and a package allowlist; all non-allowed imports SHALL be blocked. A Python worker packaged as a wheel (application/x-python) is NOT subject to this blocklist/allowlist: its dependencies are declared and vetted via its build-time Requires-Dist manifest instead of being restricted at runtime.
Scenario: Allowed import
- WHEN a raw-source Python worker imports a package listed in
PYTHON_ALLOWED_PACKAGES - THEN the import SHALL succeed
Scenario: Blocked import
- WHEN a raw-source Python worker imports a package not in the allowlist
- THEN the import SHALL fail with an error
Scenario: Wheel-packaged worker import
- WHEN a Python worker packaged as a wheel imports any module
- THEN the import SHALL succeed regardless of the blocklist/allowlist, since its dependencies were vetted at publish time via the wheel's declared
Requires-Distmanifest, not restricted at runtime
Requirement: Worker Contract (IWorker)
The IWorker interface (Virtufin.Worker.Host) SHALL define StartAsync(IWorkerContext context, CancellationToken stop), ProcessAsync(CloudEvent input, CancellationToken ct) returning Task<IReadOnlyList<CloudEvent>> (zero or more outputs for one input, in publish order) and StopAsync(CancellationToken deadline). There SHALL be no separate single- or multi-output contract, so no host dispatches on the worker's type. Every host (the Runtime's engines and bridges, the devkit's local WorkManager, the test hosts) SHALL call StartAsync once before the first ProcessAsync of an activation and StopAsync after the last. The context SHALL be handed over once, at StartAsync, and be the activation's: the worker's id, an IVirtufinApi (state, pubsub, triggers, gateway) whose publishes route through the host, a logger, the worker's Config and Secrets snapshots, State (the worker's own keys under sys.workmanager.worker.<id>.state.<key>, refused when a value contains a secret value, deleted with the worker), PublishAsync (through the host's outbox under the output rules, gated by ownership) and ReportStatusAsync. Nothing SHALL be ambient: a worker that needs the platform reads it from the context, never from the environment, so the same worker runs under the Runtime (the configured gateway) and under a local host (an in-memory api). The stop token SHALL be cancelled when the host loses ownership of the worker (lease lost, stop, reload, delete, deactivation, drain); from then on the host SHALL publish nothing of the worker's, SHALL never call ProcessAsync again, and SHALL call StopAsync with a bounded deadline.
Scenario: The context reaches the worker
- WHEN any host starts a worker
- THEN
context.WorkerIdSHALL be the worker's id in that host,context.Apithe host's api,context.Configandcontext.Secretsthe values the host stamps on every input - AND a C# source worker SHALL compile against the context's types (logging abstractions,
Virtufin.Api.Client) without declaring them
Scenario: Start before the first Process
- WHEN a host has an input for a worker it has not started
- THEN it SHALL start the worker first (the Runtime refuses the input with
FAILED_PRECONDITIONwhen the worker is not running)
Scenario: A host publish is routed like an output
- WHEN a worker calls
context.PublishAsync(or itscontext.Apipublish methods) - THEN the host SHALL apply the output rules (a fresh id, secret attributes stripped), route the event to its
publishtopicelse itsce-type, drop one with neither, and publish it in order with the worker's responses through its outbox
Scenario: Ownership lost
- WHEN the host loses ownership of a started worker
- THEN it SHALL cancel the stop token, refuse every later publish of the worker's, call
StopAsyncwith a deadline, and never callProcessAsyncon that activation again
Scenario: Async worker process
- WHEN the Runtime invokes a worker's
ProcessAsyncmethod - THEN the call SHALL be awaited asynchronously without blocking threads
Scenario: Zero, one or several outputs
- WHEN the engine hosts a worker and calls its
ProcessAsync - THEN it SHALL publish every returned CloudEvent in order
- AND an empty list SHALL publish nothing
Scenario: Every engine carries the lifecycle
- WHEN a C# source, Python, TypeScript or native (ABI v3) worker is started or stopped by the Runtime
- THEN its engine SHALL hand it the activation's context and call its start and stop exactly as the DotNetDll engine does: the subprocess engines over the subprocess protocol (
start/stopframes, the start acknowledged within the engine's startup timeout and never under the per-message timeout), the native engine through theStart/Stopexports - AND the worker's publishes, status reports (and for subprocess workers its state and fresh secrets) SHALL go back to the host as host calls, from any of the worker's threads and at any time between its start and the return of its stop, through the host's outbox and ownership gate
- AND from the moment the host's stop token fires, the engine SHALL refuse the worker's publishes
- AND the lifecycle conformance vectors SHALL pass through the Runtime on each engine
Scenario: Python worker lifecycle
- WHEN a Python worker module defines
Start(ctx)and/orStop()next toProcess(event) - THEN
StartSHALL be called once per activation before the first event with actxcarryingworker_id,config,secrets, astoppingevent,publish,report_status,state(get/save/delete/keys) andget_secrets, andStopafter the last event withctx.stoppingalready set - AND a refused host call SHALL raise
HostRefusedand a stale etagStateConflict; the devkit'sLocalWorkManagerSHALL offer the samectx
Scenario: A host publish without a route
- WHEN a worker publishes through its context an event with neither
publishtopicnortype - THEN every engine SHALL drop it (logged) and publish the rest
Requirement: Worker Lease
A started worker SHALL hold a lease: on every start and every self-heal the Runtime SHALL claim the API's publication fence keyed by the worker id (a fresh token superseding the persisted one), persist the token in the worker's Runtime record, start the worker only after the claim, renew the lease periodically (default every 1 s, lease 10 s) and check it locally before every outbox batch and host publish. A rejected renewal (another activation claimed the worker) SHALL stop the worker on this activation (gate closed, stop token, StopAsync), leave its topics and publish a status lifecycle event (status = lease-lost), the record being the successor's. It SHALL start the worker there again only once nobody holds the lease: a poke or call reaching it SHALL, at most once per renewal interval, reload the persisted record and, if it still says running, claim under its own superseded token, which the fence refuses while the successor's lease is live. Renewals that fail until the lease lapses SHALL stop the worker the same way and mark it faulted (failed); the next poke or call SHALL claim and start it again (online). The lease SHALL be revoked on stop, delete, deactivation, reload and the shutdown drain.
Scenario: Takeover
- WHEN a successor activation claims a worker's lease while the old activation still runs
- THEN the old activation SHALL publish nothing after its next renewal, SHALL call the worker's
StopAsyncwith the stop token cancelled, and SHALL not touch the Runtime record
Scenario: Reactivation claims with the persisted token
- WHEN a worker's actor activates on a Runtime pod and its record says running
- THEN the activation SHALL claim the lease with the record's token, start the worker and resubscribe; a refused claim SHALL fault the worker and fail the activation
Scenario: Superseded activation after the successor is gone
- WHEN an activation whose lease was taken over receives a poke or call after the successor's lease was revoked or expired, and the persisted record says running
- THEN it SHALL claim the lease, start the worker and resubscribe, so the worker never stays listed running while not started; while the successor's lease is live, or once the successor stopped the worker, it SHALL not start it
Requirement: Worker Commands
The WorkManager SHALL expose InvokeWorkerCommand(id, command, payload, content_type, attributes, timeout_ms): the command SHALL become a CloudEvent of type com.virtufin.workmanager.worker.command (source urn:com.virtufin:workmanager, subject worker/<id>, the command name in the command extension, the attributes as string extensions, the payload as data) that goes through the worker's actor ProcessAsync like any input, serialized with its inbox; the outputs SHALL be returned flattened and SHALL NOT be published. The local WorkManager SHALL implement the same (IWorkManagerClient.InvokeCommandAsync).
Scenario: Command to a running worker
- WHEN a client invokes a command on a running worker
- THEN the worker's
ProcessAsyncSHALL see the command event with its config and secrets stamped, and the client SHALL receive every output in order, none of them published
Scenario: Command to a worker that is not running
- WHEN a client invokes a command on a stopped, loading or load-failed worker
- THEN the WorkManager SHALL answer
FAILED_PRECONDITIONwithout calling the actor
Requirement: Topic Dependencies
A worker's input topics (CreateWorker topics) SHALL be its whole dependency: it subscribes to them and SHALL NOT name the worker that produces them. The WorkManager SHALL NOT match topics to producers. An orchestrator outside the platform (a notebook, a script, the TUI) SHALL open the producers of the topics it needs (for venue data the WebSocketFeed workers that serve them, WebSocket Feed, Feed Orchestration); workers and producers MAY start in any order. The devkit's former requires=<type>/<name> attach/detach hook is removed (worker-devkit 0.40.0): a config key of that name is ordinary config.
Scenario: A worker and its feed start in any order
- WHEN a market-data worker is created with its input topics and an orchestrator applies the feeds that serve them, in either order
- THEN the worker SHALL receive the frames once both run, and neither SHALL send the other anything
Requirement: Worker Base Classes
Virtufin.Worker.DevKit SHALL provide the abstract WorkerCore<TInput, TOutput> (an IWorker constructed with a worker type and an IPayloadCodec<TInput, TOutput>, owning the envelope helpers BuildResponse and Error) and, on it, the abstract WorkerBase<TInput, TOutput>: it decodes the input's data with the codec, calls the typed ProcessAsync(CloudEvent, TInput) override, and returns its one result in the response envelope (BuildEnvelope), or no output for null. Virtufin.Behavior.DevKit (built in the worker-devkit repo) SHALL provide BehaviorWorkerBase, a WorkerCore<JsonNode, JsonObject> that returns every output of a bound behaviour process (behaviour spec, Worker Translation). There SHALL be no command-dispatch base class in C#: a worker that takes commands reads them from its typed input.
Scenario: Worker named by its worker type
- WHEN a worker of type
<type>(e.g.httpget) builds a response or an error - THEN its source SHALL be
urn:com.virtufin:worker:<type>, a response SHALL have ce-typecom.virtufin.<type>.responseon topicwrk.<type>.response, and an error ce-typecom.virtufin.<type>.erroron topicwrk.<type>.error(WorkerTypeNames, from the naming library) - AND an incoming
replytopicSHALL change the topic of both, never the ce-type
Scenario: Response envelope
- WHEN
WorkerBasebuilds a response - THEN it SHALL give the response a new
id(never the input's), setsource,type,timeandcorrelationid, and set thepublishtopicextension to the response topic
Scenario: Unhandled exception
- WHEN the typed
ProcessAsyncoverride throws - THEN
WorkerBaseSHALL return an error response with messageInternal error: <exception message>
Scenario: API client
- WHEN a worker needs the API gateway
- THEN it SHALL use
context.Api(theIVirtufinApiof itsIWorkerContext: the host's configured gateway under the Runtime, a local host's in-memory API otherwise) - AND only an explicit other endpoint (the
apihostconfig) SHALL be dialled throughWorkerApiEndpoint.CreateClient(endpoint), which attaches the host's key only to the host's own configured gateway
Requirement: Python Worker Base Classes
The Python worker devkit (virtufin.worker_devkit, in the workmanager repo) SHALL provide WorkerBase(worker_type=...) with an abstract handle(event) returning none, one or a list of CloudEvent dicts, named by the worker type like the C# WorkerBase. It SHALL also provide CommandWorker, ApiWorker and ApiCommandWorker.
Scenario: Command dispatch
- WHEN a
CommandWorkerreceives an event whose data is a JSON object with acommandfield - THEN it SHALL call
handle_command(input, command, data)
Scenario: Command errors
- WHEN the event data is not valid JSON, has no
commandfield, orhandle_commandreturns None - THEN the worker SHALL return an error response (
Invalid JSON in event data,Missing 'command' field,Unknown command: <command>)
Scenario: API workers
- WHEN an
ApiWorkerorApiCommandWorkeris created with an optionalapi_host/api_port - THEN it SHALL hold an API client for that endpoint, else for the host's configured gateway (
self.api);ApiWorkerpasses it toprocess_async(input, api)
Scenario: Unhandled exception
- WHEN
handleraises - THEN
processSHALL return an error response with messageInternal error: <exception>
Requirement: Correlation ID Propagation
Worker responses SHALL propagate the correlationid extension attribute from the input CloudEvent to the output CloudEvent. The CloudEvent.WithCorrelationId(input) extension method SHALL handle this as a no-op when absent.
Scenario: Response with correlation ID
- WHEN a worker creates a response or an error via the base class helpers (
BuildResponse/Errorin C#,response/errorin Python) - THEN the output CloudEvent SHALL carry the same
correlationidas the input CloudEvent if present
Scenario: Input without correlation ID
- WHEN a worker processes an input CloudEvent without a
correlationidextension - THEN the output CloudEvent SHALL NOT have a
correlationidextension
Requirement: In-Process DotNet DLL Engine
The WorkManager SHALL support an in-process DotNetDllEngine (default for MIME type application/x-dotnet-dll) that loads pre-compiled .NET worker DLLs directly into the AOT-compiled Runtime process that hosts the worker. The engine SHALL embed the .NET runtime (CoreCLR) into the host process on first use via the hostfxr C API and SHALL load the worker DLL into a per-worker, collectible AssemblyLoadContext. The engine SHALL be AOT-compatible: the engine code itself is compiled with <IsAotCompatible>true</IsAotCompatible> and SHALL NOT use reflection on its own types; the only reflection SHALL be on the JIT-loaded worker assembly.
Scenario: First LoadCodeAsync initializes CoreCLR
- WHEN the first
LoadCodeAsyncis called for MIME typeapplication/x-dotnet-dll - THEN the engine SHALL load
libhostfxrviaNativeLibrary.Loadand callhostfxr_initialize_for_runtime_configto initializeCoreCLRin the host process; the cost SHALL be paid once per Runtime process
Scenario: Subsequent LoadCodeAsync in the same process
- WHEN a second
LoadCodeAsyncis called afterCoreCLRis already initialized - THEN the engine SHALL skip the
hostfxrinitialization and load the new worker DLL directly
Scenario: Per-worker AssemblyLoadContext
- WHEN a
LoadCodeAsyncis called with a worker nupkg - THEN the engine SHALL extract the worker DLL (and its sibling dependencies) from the nupkg's
lib/<tfm>/entries, create a new collectibleAssemblyLoadContext, and load the worker assembly into it
Scenario: Host resolution
- WHEN the JIT-loaded worker assembly references
Virtufin.Worker.Host - THEN the engine SHALL resolve the reference to the
Virtufin.Worker.Hostinstance already loaded in the Runtime process (so the worker and the engine see the same type identity forIWorker,IWorkerContext,IVirtufinApiand CloudEvents) - AND
Virtufin.Worker.DevKit(the authoring layer:WorkerBase, codecs, envelope helpers) SHALL load into the worker's own assembly load context from the worker's dependencies, so it may be newer than the deployed runtime as long as its Host requirement is satisfied
Scenario: Direct in-process dispatch
- WHEN
ProcessAsyncis called on the engine with a CloudEvent - THEN the engine SHALL call the worker's
IWorker.ProcessAsyncdirectly, with no socket, no JSON serialization, and no subprocess scheduling; the per-call latency SHALL be sub-microsecond
Scenario: Runtime discovery on target
- WHEN the Runtime runs on a target that does not have a compatible .NET runtime installed
- THEN the first
LoadCodeAsyncSHALL fail with a clear error message that names the missing runtime and the discovery paths (DOTNET_ROOT, system install, side-by-sidehost/fxr/<version>/)
Scenario: Roll-forward to a newer runtime
- WHEN the Runtime is built for .NET 10 and the target has a .NET 11+ runtime installed
- THEN the engine SHALL use the newer runtime; the generated
runtimeconfig.jsonSHALL declare"rollForward": "Major"
Requirement: Native DLL Engine
The WorkManager SHALL support a NativeDllEngine for executing workers
shipped as a per-architecture native shared library (.so / .dll /
.dylib). The engine SHALL run in-process inside the (NativeAOT-compiled)
Runtime and SHALL NOT require a separate worker-host subprocess or a
JIT build stage. The engine SHALL be registered for MIME type
application/x-native-dll. Per-architecture entries SHALL live under
runtimes/<rid>/native/ inside the worker zip, matching the NuGet
convention for native packages.
Scenario: Native worker execution
- WHEN a worker with MIME type
application/x-native-dllprocesses a CloudEvent - THEN the engine SHALL
NativeLibrary.Loadthe matchingruntimes/<rid>/native/<file>entry from the worker zip, resolve theentry_pointandfree_resultexports declared inmanifest.json, encode the input CloudEvent as a protobufCloudEvent, invoke the nativeProcessfunction, and decode the returnedCloudEventBatch.
Scenario: Per-architecture entry selection
- WHEN a worker zip contains entries under
runtimes/<rid>/native/for at least one supported RID (linux-x64,linux-arm64) - THEN the engine SHALL select the entry whose
<rid>directory matchesRuntimeInformation.RuntimeIdentifierof the running process and SHALL load the correspondinglib<library>.so,<library>.dll, orlib<library>.dylibper the platform (see Native Worker Library Naming).
Scenario: Unsupported architecture
- WHEN the worker zip does not contain a
runtimes/<rid>/native/entry for the running architecture - THEN the engine SHALL throw
NotSupportedExceptionlisting the supported archs (linux-x64,linux-arm64).
Scenario: Missing manifest
- WHEN the worker zip does not contain
manifest.jsonat the root - THEN the engine SHALL refuse to load the worker and throw a clear error naming the missing file.
Scenario: ABI version mismatch
- WHEN the
manifest.jsondeclares anabi_versionthat is not in the engine's supported set - THEN the engine SHALL refuse to load the worker and throw a clear error naming the supported versions.
Requirement: Native Worker ABI
The NativeDllEngine SHALL define and enforce a stable C ABI between the
managed engine and the native worker. The ABI SHALL consist of: (a) the
Worker Wire Protocol messages (CloudEvent in, CloudEventBatch out); (b) two
C function exports Process and FreeResult; (c) a VirtufinHost struct
carrying host callbacks; (d) the worker nuspec's virtufin* elements declaring
the ABI version (virtufinAbiVersion), the library basename and the export
names. The engine SHALL vendor virtufin_worker_api_c.h and a header-only
protobuf codec virtufin_worker_api_pb.h for worker authors. ABI version 2 is
the Worker Wire Protocol; ABI version 3 is version 2 plus the worker lifecycle
(Start/Stop exports and the host callbacks publish and report_status).
Both are supported.
Scenario: ABI struct layout
- WHEN a native worker is loaded
- THEN the engine SHALL pass a
VirtufinHost*whose first field is an opaqueengine_ptr, whose second field isabi_version(the host's version, 3), and whose remaining fields are C function pointerslog,gateway_call,free_response,publishandreport_status, the last two after every field an ABI v2 worker reads.
Scenario: ABI v3 lifecycle
- WHEN a worker's nuspec declares
virtufinAbiVersion3 - THEN the library SHALL export
Start(host, ctx_buf, ctx_len, &out_buf, &out_len)andStop(host, budget_ms), or the engine SHALL refuse to load it - AND the engine SHALL call
Startonce per activation before the firstProcess, withctx_bufthe UTF-8 JSON{"workerId", "config", "secrets"}; a1return SHALL fail the start with the UTF-8 message in*out_buf(released byFreeResult) - AND until
Stopreturns the worker MAY callhost->publish(batch, len)(a protobufCloudEventBatch, returning0published,1refused,-1failed) andhost->report_status(state, detail)from any thread;StopSHALL be called after the lastProcessand SHALL return withinbudget_ms
Scenario: ABI v2 worker without lifecycle
- WHEN a worker declares ABI version 2
- THEN the engine SHALL load and process it unchanged and call no lifecycle export
Scenario: CloudEvent input
- WHEN the engine invokes
Process(host, in_buf, in_len, &out_buf, &out_len)for an ABI v2 worker - THEN
in_bufSHALL hold a protobufio.cloudevents.v1.CloudEvent; thecorrelationidattribute SHALL carry the correlation id when present.
Scenario: CloudEventBatch output
- WHEN
Processreturns0 - THEN
*out_bufSHALL hold a protobufCloudEventBatch, and the engine SHALL publish every event in it, in order; an empty batch publishes nothing.
Scenario: Worker error
- WHEN
Processreturns1 - THEN
*out_bufSHALL hold a UTF-8 error message, and the engine SHALL fail the call with that message (the Runtime dead-letters the input, Worker Dead Letters; no response is published).
Scenario: Internal failure
- WHEN
Processreturns any other non-zero value - THEN the engine SHALL treat the call as failed (dead-lettered like a worker
error);
*out_bufMAY be NULL.
Scenario: FreeResult pairing
- WHEN
Processreturns a non-null*out_buf - THEN the engine SHALL call
FreeResult(out_buf)exactly once, including on the error path. The worker SHALL use the same allocator for the*out_bufallocation thatFreeResultreleases.
Scenario: Logging callback
- WHEN the native worker calls
host->log(level, message) - THEN the engine SHALL map
level(0=trace,1=debug,2=info,3=warn,4=error) to the correspondingMicrosoft.Extensions.Logging.LogLeveland forwardmessageto the engine's structured logger. The lifetime ofmessageSHALL be limited to the call.
Scenario: Gateway call indirection
- WHEN the native worker calls
host->gateway_call(service, method, request, request_len, &response, &response_len) - THEN the engine SHALL deserialize
requestas JSON into aDictionary<string, object?>, invokeApiClient.InvokeAsync(service, method, dict), JSON-serialize the returned dictionary into*response(allocated withMarshal.AllocCoTaskMem), and return0on success or non-zero on internal failure. The worker SHALL free*responseviahost->free_response(response).
Scenario: ABI version 1 refused
- WHEN a worker declares ABI version 1 (the former FlatBuffers wire format)
- THEN the engine SHALL refuse to load it, naming versions 2 and 3 as the supported ones.
Scenario: ABI version 1 worker rebuilt
- WHEN a worker built for ABI version 1 is rebuilt against the current headers
- THEN it SHALL return a protobuf
CloudEventBatchinstead of a FlatBuffersWorkerResponse.
Scenario: Manifest format
- WHEN the engine reads the worker's package metadata
- THEN the nuspec SHALL carry
virtufinAbiVersion(integer) andvirtufinLibrary(non-empty, matching[A-Za-z0-9_.-]+), and MAY carryvirtufinEntryPoint(defaultProcess) andvirtufinFreeResult(defaultFreeResult). A missing or empty library SHALL cause the engine to refuse the worker with a clear error.
Scenario: ABI version mismatch
- WHEN the worker nuspec declares a
virtufinAbiVersionthe engine does not support - THEN the engine SHALL refuse to load the worker with an error naming the supported versions.
Requirement: Native Worker Identity and Lifecycle
The NativeDllEngine SHALL communicate worker identity to the native
worker via process-level environment variables set at LoadCodeAsync time,
and SHALL release the loaded library and all engine-owned resources on
worker stop.
Scenario: Worker identity env vars
- WHEN the engine loads a native worker
- THEN it SHALL set
VIRTUFIN_WORKER_IDto the worker GUID andVIRTUFIN_WORKER_TOPICto the worker's input topic. The worker SHALL read these once at initialization.
Scenario: Correlation id from input
- WHEN the engine marshals the input CloudEvent for a
Processcall - THEN the protobuf
CloudEvent'scorrelationidattribute SHALL carry the input CloudEvent'scorrelationidextension attribute if present. Workers SHALL read it from the event, not from env vars.
Scenario: Concurrent Process calls
- WHEN multiple
Processinvocations are made against the same loaded library - THEN the engine SHALL allow them to proceed concurrently. The native worker SHALL be reentrant; any per-worker mutable state SHALL be protected by the worker's own synchronization primitives.
Scenario: Reload under load
- WHEN
LoadCodeAsyncis called while aProcessAsyncis in flight - THEN the engine SHALL block the reload until the in-flight call completes. The new code SHALL take effect for subsequent calls only.
Scenario: Cleanup on stop
- WHEN the engine is disposed or the worker is stopped
- THEN the engine SHALL call
NativeLibrary.Freeon the loaded library (except a NativeAOT library, see below), free theGCHandlefor the engine pointer, delete the temp file holding the DLL, and restore the prior values ofVIRTUFIN_WORKER_IDandVIRTUFIN_WORKER_TOPICif they were set.
Scenario: NativeAOT library stays loaded
- WHEN the library being released exports
DotNetRuntimeDebugHeader(a NativeAOT build with an embedded .NET runtime) - THEN the engine SHALL NOT call
NativeLibrary.Freeon it, because the embedded runtime keeps threads running in the library's code and unloading it crashes the hosting process; it SHALL still release every other resource.
Requirement: Native Worker Caveats
The NativeDllEngine SHALL document its isolation and timeout
characteristics in the engine README and the worker author guide.
Scenario: In-process crash isolation caveat
- WHEN a native worker raises an unrecoverable error (segmentation fault, illegal instruction, stack overflow from native code)
- THEN the Runtime process hosting it SHALL terminate because the error cannot be caught in a way that leaves the .NET runtime in a defined state. Workers SHALL be authored under the assumption that a crash terminates the entire Runtime pod. The Python and C# source engines isolate their workers in subprocesses; the DotNetDll engine runs them in-process too, but a managed exception is caught.
Scenario: Unmanaged timeout caveat
- WHEN the
MessageHandlingTimeoutSecondselapses while a nativeProcesscall is in flight - THEN the
CancellationTokenon the managed side SHALL fire and the call SHALL fail on the managed side, but the native call SHALL continue to run until it returns or the process crashes. Workers SHALL be authored to honor the timeout cooperatively (e.g. via periodic checks of a flag set from a watchdog thread, if at all) or SHALL be guaranteed to return within the timeout.
Scenario: ABI version policy
- WHEN the Runtime ships a new major ABI version
- THEN the engine SHALL support only that version while Virtufin is in beta: workers are rebuilt in the same release wave, and the engine SHALL refuse a worker declaring any other version.
Requirement: Native Worker Library Naming
The NativeDllEngine SHALL derive the per-platform shared library
filename from the manifest's library field by appending an OS-specific
prefix and suffix, matching the convention used by NuGet-distributed
native packages (runtimes/<rid>/native/). The engine SHALL pick the
filename based on RuntimeInformation.IsOSPlatform.
Scenario: Linux filename
- WHEN the running platform reports
OSPlatform.Linux - THEN the engine SHALL load
lib<library>.sofromruntimes/<rid>/native/.
Scenario: Windows filename
- WHEN the running platform reports
OSPlatform.Windows - THEN the engine SHALL load
<library>.dllfromruntimes/<rid>/native/.
Scenario: macOS filename
- WHEN the running platform reports
OSPlatform.OSX - THEN the engine SHALL load
lib<library>.dylibfromruntimes/<rid>/native/.
Scenario: Library name validation
- WHEN the manifest's
libraryfield is empty or contains characters outside[A-Za-z0-9_.-] - THEN the engine SHALL refuse the worker and throw a clear error naming the offending value.
Requirement: Worker Tags (Resource Metadata)
The WorkManager SHALL let clients attach arbitrary key/value tags to a
worker via CreateWorkerRequest.tags, SetTag, and SetTags, retrievable
via GetTag/GetTags/ListWorkers (WorkerInfo.tags). Tags SHALL be
resource metadata only — used for filtering
(ListWorkersRequest.filters: repeated TagFilter) and identification — and
SHALL NOT be injected into the worker's process environment or otherwise
exposed to the worker's code.
Scenario: Tags filter ListWorkers
- WHEN a client calls
ListWorkerswith one or moreTagFilter{key, value}entries - THEN only workers whose tags match every filter SHALL be returned
Scenario: Tags are not process-visible
- WHEN a worker is loaded by any engine
- THEN its tags SHALL NOT be set as environment variables and SHALL NOT
be passed to the worker's
ProcessAsync— a worker has no supported way to read its own tags
Scenario: SetTag/SetTags update stored metadata only
- WHEN a client calls
SetTagorSetTagson a running worker - THEN the WorkManager SHALL update the persisted tag metadata immediately; no engine reload is required since tags are never applied to the process
Requirement: Worker Creation-Time Config
The WorkManager SHALL let clients set worker-instance defaults via
CreateWorkerRequest.config (map<string, string>), retrievable via
WorkerInfo.config. Unlike the removed worker environment-variable
feature (which mutated process-wide Environment state and let co-loaded
workers silently stomp on each other's values), config SHALL be delivered
per-call, per-worker-instance: the Runtime SHALL stamp every
config entry onto the incoming CloudEvent as an extension attribute
before dispatching to the engine, using the same generic extension
mechanism already used for correlationid. Config SHALL be immutable
after creation — there SHALL be no SetConfig/GetConfig RPCs.
Scenario: Config delivered as CloudEvent extension attributes
- WHEN a worker created with
config: {"url": "https://example.com"}receives a triggering CloudEvent - THEN the CloudEvent SHALL carry a
urlextension attribute set to"https://example.com"by the time it reaches the worker's engine
Scenario: Config does not leak between co-loaded workers
- WHEN two workers, each created with a different
configvalue for the same key, are loaded in the same Runtime process - THEN each worker's incoming CloudEvents SHALL only ever carry that
worker's own
configvalue — never the other worker's
Scenario: Config is immutable after creation
- WHEN a client wants to change a worker's config
- THEN the WorkManager SHALL NOT provide an RPC to do so; the worker
MUST be deleted and recreated with the new
config
Scenario: Config survives WorkManager restart
- WHEN a worker is recovered after a restart (actor reactivation or
RecoverWorkers) - THEN the recovered worker SHALL retain its original
config
Scenario: A config key that can never reach the worker is refused
- WHEN
CreateWorkerRequest.confighas a key that is not a CloudEvent attribute name (lower-case letters and digits only, e.g.strategy_id,marketWorld,market.world), isdata, is a core attribute that takes no string (time,dataschema), or is an attribute every input already carries (id,source,specversion,type,datacontenttype) - THEN
CreateWorkerSHALL fail withINVALID_ARGUMENTnaming the key, and no worker SHALL be created - AND every host (the Runtime's WorkManager and the C#, Python and
TypeScript local WorkManagers) SHALL apply the same rule, pinned by the
conformance vectors'
config_validationcases (WorkerEventRules.ValidateConfig) - AND for a worker created before hosts validated config, the per-event rule SHALL still drop such an entry (logged) and apply the rest
Requirement: Worker Manager Horizontal Scalability
The WorkManager SHALL scale horizontally by spreading workers across processes, not by sharing one worker's load: every worker subscribed to a topic receives every message on it, wherever it runs (Cross-Instance Coordination), and a worker processes its topic's messages in order, one at a time. Throughput for a high-volume stream SHALL therefore come from splitting it across topics (e.g. one topic per currency pair) and workers, not from adding replicas of one worker.
Scenario: Multiple WM replicas on the same topic
- WHEN two Runtime pods each run a worker for topic
orders - THEN each message published to
ordersSHALL be delivered to both replicas, and both workers SHALL process it - AND this SHALL hold for distinct workers and for two deployments of the same package alike: deploying a second copy of a worker duplicates its processing (and its outputs), it does not share load
Scenario: Several workers on one topic
- WHEN several workers on the same Runtime pod subscribe to the same topic, e.g. several indicator workers on one candle topic
- THEN every message on the topic SHALL be delivered to each of them, and each SHALL process it with its own config and state
- AND one worker failing on a message SHALL NOT prevent its delivery to the others
- AND each worker SHALL receive the topic's messages in publish order
Scenario: Hot-swap within a topic
- WHEN a worker on
ordersis replaced by a new worker onorders - THEN the replacement SHALL be stop-then-start: stopping the old worker unsubscribes it, and from then on only the new worker receives messages on
orders - AND while both are subscribed, both SHALL receive every message (there is no "latest worker wins" dispatch)
Scenario: Scaling throughput
- WHEN operators want to scale throughput for a high-volume stream
- THEN they SHALL NOT deploy more copies of the same worker on the same topic expecting load sharing -- every copy processes every message
- AND they SHALL instead split the stream across topics (e.g. per-pair topics) with a worker per topic, which the WorkManager spreads across processes, or speed up the worker itself
- AND the Pub/Sub Topics spec "Subscription Implementation Pattern" requirement is the authoritative reference for delivery
Scenario: Observability for competing-consumer behavior
- WHEN an operator sees a worker not receiving messages on a topic
- THEN they SHALL check that the worker is RUNNING and subscribed to that exact topic, and that its process's subscription is open (messages published before a subscription opens are not delivered: Pub/Sub Topics spec, "New subscription starts at the stream tail")
- AND there is no competition between replicas to rule out: every subscribed process receives every message
- AND the
sys.workmanager.worker.lifecycleevents carryworker_idandtopicso operators can correlate
Requirement: Worker Source Authentication
URL-based code sources SHALL support an optional per-request credential, carried on the same request as the code source (CreateWorkerRequest/LoadCodeRequest), never persisted beyond the fetch it authorizes. A credential SHALL be one of: a directly-supplied Basic or Bearer secret, or a named reference resolved server-side. SSRF and host-allowlist validation SHALL apply regardless of whether a credential is present or which kind it is.
Scenario: Directly-supplied credential
- WHEN a code source URL is fetched with a Basic or Bearer credential attached to the request
- THEN the fetch SHALL use that credential's
Authorizationheader, and no other
Scenario: Named credential resolution
- WHEN a code source URL is fetched with a named credential reference
- THEN the service SHALL resolve the name to a server-held credential and validate the target host is permitted for that named credential before injecting it
Scenario: Unresolvable named credential
- WHEN a request references a named credential the service does not recognize, or whose target host is not permitted for that name
- THEN the fetch SHALL fail with a clear authorization error rather than proceeding unauthenticated
Scenario: No credential supplied
- WHEN a code source URL is fetched with no credential on the request
- THEN the fetch SHALL proceed without an
Authorizationheader
Scenario: Credentials are not persisted
- WHEN a code source fetch completes, successfully or not
- THEN the supplied credential SHALL NOT be logged or retained beyond that fetch
Requirement: Multi-Topic Worker Subscription
A worker MAY subscribe to more than one topic. CreateWorkerRequest SHALL accept a list of topics (minimum one); the WorkManager SHALL open a subscription for each distinct topic in the list and dispatch any triggering CloudEvent on any of them to the worker.
Scenario: Worker created with multiple topics
- WHEN a client creates a worker with more than one topic in
CreateWorkerRequest - THEN the WorkManager SHALL subscribe to every listed topic
- AND a CloudEvent arriving on any of the worker's topics SHALL be dispatched to that worker's
ProcessAsync
Scenario: Worker created with one topic
- WHEN a client creates a worker with exactly one topic
- THEN behavior SHALL be identical to the single-topic case that predates this requirement — this is a superset capability, not a new required pattern
Scenario: Shared topic across workers unaffected
- WHEN multiple workers (with possibly-overlapping topic lists) are registered
- THEN the existing per-topic dispatch rule (all local workers registered for a topic receive a matching message; see the Worker Lifecycle Events and Cross-Instance Coordination requirements) SHALL apply independently per topic, unchanged by a worker having additional topics
Scenario: Worker deletion unsubscribes all its topics
- WHEN a worker with multiple topics is deleted
- THEN the WorkManager SHALL unsubscribe/tear down the worker's registration on every one of its topics
- AND a topic SHALL only have its Dapr subscription fully torn down once no worker (in this or any other topic-sharing registration) still references it
Requirement: Worker Secrets by Reference
The WorkManager SHALL let clients give a worker secrets through CreateWorkerRequest.secrets (map<string, string>, attribute name -> secret key). It SHALL persist and return (WorkerInfo.secrets) only these references. The Runtime SHALL resolve them from the configured Dapr secret store, keep the values in memory only, and stamp each onto every incoming CloudEvent as the extension attribute of that name. Secrets SHALL be immutable after creation, like config.
Scenario: Secret delivered as an extension attribute
- WHEN a worker created with
secrets: {"alpacakey": "alpaca-paper-key"}receives a triggering CloudEvent - THEN the CloudEvent SHALL carry an
alpacakeyextension attribute set to the value ofalpaca-paper-keyin the secret store by the time it reaches the worker's engine, overriding analpacakeyattribute on the incoming event
Scenario: Only references are stored and listed
- WHEN a worker with secrets is persisted, listed, or recovered after a restart
- THEN its record and
WorkerInfoSHALL hold only the references, and the recovered worker SHALL resolve them again
Scenario: Unresolvable reference
- WHEN a reference does not resolve while the worker loads
- THEN the worker SHALL become
LOAD_FAILEDwith aload-failedlifecycle event naming the reference and not containing any secret value
Scenario: Invalid request
- WHEN a secret name is not a valid extension-attribute name, is reserved, is also a
configkey, or a reference does not match[A-Za-z0-9._-]{1,253}(Kubernetes Secret key rules) - THEN
CreateWorkerSHALL fail withINVALID_ARGUMENT
Scenario: Secrets do not reach outputs
- WHEN a worker returns a CloudEvent that carries one of its secret attributes (for example its own input)
- THEN the Runtime SHALL drop that attribute before the event is published
Scenario: Rotation
- WHEN a secret's value changes in the store
- THEN a loaded worker SHALL receive the new value within the Runtime's secret refresh interval (default 60 s) without being recreated
Scenario: Never logged
- WHEN the WorkManager or Runtime logs anything about a worker with secrets
- THEN the log SHALL NOT contain a secret value