Skip to content

Worker Management

Purpose

Event-driven worker management system enabling deployment and execution of workers written in multiple languages (Python, TypeScript, C#, .NET DLL, NativeAOT) that subscribe to Dapr pub/sub topics and process CloudEvents. Every process subscribed to a topic receives every message on it (one consumer group per sidecar), and each worker runs in exactly one process; see the Cross-Instance Coordination and horizontal scalability requirements below.

Topic and state-key naming for worker lifecycle events is defined in pubsub-topics/spec.md and built by the naming library from the virtufin.service.workmanager fragment. The lifecycle topic is sys.workmanager.worker.lifecycle; worker records live at sys.workmanager.worker.<worker_id>.

Requirements

Requirement: gRPC-only Control API

WorkManager SHALL expose business control operations only as the gRPC service virtufin.WorkManager; its HTTP port SHALL serve only health/readiness probes.

Scenario: Plain HTTP request

  • WHEN a client sends a plain (non-gRPC) HTTP request for anything other than a probe
  • THEN no worker-management operation SHALL execute
  • AND clients SHALL use gRPC directly or gateway invocation over gRPC

Requirement: Worker Lifecycle Management

The WorkManager SHALL support creating, starting, stopping, and deleting workers. Workers SHALL persist their configuration and restore automatically on service restart.

Scenario: Create and start a worker

  • WHEN a client creates a worker with a topic subscription and code payload, then starts it
  • THEN the worker SHALL be persisted to the state store, and once its code is loaded it SHALL be subscribed to the topic and set to Running status

Scenario: Hot-reload worker code

  • WHEN a client updates a running worker's code
  • THEN the worker SHALL switch to the new code without restarting the service: a started worker is stopped (stop token, StopAsync, lease revoked), reloaded and started again with a fresh context, never reloaded in place
  • AND a worker whose config says autoreload=false SHALL NOT be reloaded on a library release

Scenario: Auto-recovery on restart

  • WHEN a Runtime pod restarts, or a worker's actor was idle-deactivated
  • THEN the worker's next actor activation SHALL reload its code and resubscribe its topics from its Runtime record (sys.workmanager-runtime.worker.<worker_id>)
  • AND the WorkManager SHALL touch the actor of every running worker on start-up and periodically (RuntimeWorkerPokeService, default every 60 s), so a worker without inbound traffic is reactivated too, and a touch SHALL make sure the worker is started (a lapsed lease is claimed again)

Scenario: Worker version history

  • WHEN a worker's code is updated
  • THEN the previous code version SHALL be preserved in a versioned history

Scenario: Create returns before the code is loaded

  • WHEN a client creates a worker
  • THEN the WorkManager SHALL persist it and return with status Loading without waiting for the code fetch, dependency resolution or engine load
  • AND it SHALL load the code in the background and set the status to Stopped (or Running, see deferred start) once loaded, or LoadFailed on failure

Scenario: Start a worker that is still loading

  • WHEN a client starts a worker whose status is Loading
  • THEN the WorkManager SHALL accept the request and start the worker (subscribe it and set it to Running) as soon as loading completes

Scenario: Start a worker whose load failed

  • WHEN a client starts a worker whose status is LoadFailed
  • THEN the WorkManager SHALL reject the request with FAILED_PRECONDITION

Requirement: Service Architecture

The WorkManager and its Runtime SHALL use the virtufin-api gRPC for all pubsub and state operations. They SHALL NOT call any Dapr pubsub or state API directly. The Dapr sidecar SHALL remain in use for actors, the secret store (Worker Secrets by Reference), service-invocation mTLS, distributed tracing, and metrics.

Scenario: Service persists worker state

  • WHEN the WorkManager or the Runtime saves or retrieves a worker record
  • THEN it SHALL call the API's State gRPC service
  • AND it SHALL NOT call any Dapr state API directly

Scenario: Runtime subscribes to a worker topic

  • WHEN the Runtime subscribes to a worker's input topic
  • THEN it SHALL call Pubsub.Subscribe on the virtufin-api
  • AND it SHALL NOT call DaprPublishSubscribeClient.SubscribeAsync directly

Requirement: Worker Lifecycle Events

The WorkManager SHALL publish worker lifecycle events to the sys.workmanager.worker.lifecycle topic, with ce-type = com.virtufin.workmanager.worker.lifecycle and ce-subject = worker/<worker_id>; the payload's state field SHALL name the transition, and the payload SHALL carry worker_id and topic. Names come from the naming library (virtufin.service.workmanager fragment, pubsub-topics/spec.md). Each event SHALL be published once, when its transition happens; a call or an actor activation that changes no state SHALL publish nothing. The states degraded, stopping and unknown are reserved and SHALL NOT be published. The state status carries a worker's own report (ReportStatusAsync: payload status, detail) or the Runtime's lease loss (status = lease-lost); it is not a catalog transition and is also kept in the worker's Runtime record.

Scenario: Worker registered

  • WHEN CreateWorker has accepted and persisted a worker
  • THEN the WorkManager SHALL publish a lifecycle event with state = registered, before any event of its load

Scenario: Worker loaded

  • WHEN a worker's code and dependencies finish loading
  • THEN the WorkManager SHALL publish a lifecycle event with state = loaded with worker_id and the resolved Virtufin.* library versions (libraries) in the data payload

Scenario: Worker load failed

  • WHEN loading a worker's code or dependencies fails
  • THEN the WorkManager SHALL publish a lifecycle event with state = load-failed with error_type and error_message in the data payload

Scenario: Worker started

  • WHEN StartWorker starts a stopped worker, or a worker started while loading finishes loading
  • THEN the WorkManager SHALL publish state = starting, then state = online once the worker is subscribed

Scenario: Start fails

  • WHEN the Runtime fails to start a loaded worker
  • THEN the WorkManager SHALL publish state = starting, then state = failed with error_type and error_message, and the worker SHALL stay Stopped

Scenario: Start of a running worker

  • WHEN StartWorker is called on a Running worker, or on a Loading one (the start is only recorded)
  • THEN no lifecycle event SHALL be published for that call

Scenario: Worker stopped

  • WHEN StopWorker stops a Running worker, or DeleteWorker deletes one
  • THEN the WorkManager SHALL publish state = stopped; stopping a worker that is not running SHALL publish nothing

Scenario: Worker deleted

  • WHEN DeleteWorker has removed a worker
  • THEN the WorkManager SHALL publish state = deleted, after the stopped of a worker that ran
  • AND a DeleteWorker that fails SHALL publish nothing

Scenario: Worker updated

  • WHEN SetTag or SetTags has saved a worker's tags, or LoadCode has loaded new code into it
  • THEN the WorkManager SHALL publish state = updated with changed (["tags"] or ["code"]) and the new value as stored: tags (the whole tag map) or codeurl (null for inline code)

Scenario: Following the workers without polling

  • WHEN a client keeps a list of workers current
  • THEN every change of what ListWorkers returns SHALL have a lifecycle event (registered, deleted, updated and the state transitions), so the client reads ListWorkers once when its lifecycle subscription is live and then applies the events
  • AND because delivery is best effort, the client SHALL read ListWorkers again after a resubscription

Scenario: Self-heal fails

  • WHEN a Runtime actor activation cannot reload or resubscribe a Running worker
  • THEN the Runtime SHALL persist the worker as faulted, publish state = failed with error_type and error_message once, and fail the activation so that a later activation retries
  • AND a later activation that fails again SHALL publish nothing, and the one that succeeds SHALL publish state = online

Scenario: Healthy reactivation

  • WHEN a Runtime actor activation reloads and resubscribes a worker that was not faulted
  • THEN no lifecycle event SHALL be published

Scenario: Worker status report

  • WHEN a started worker calls context.ReportStatusAsync(state, detail)
  • THEN the Runtime SHALL publish a lifecycle event with state = status, status = <state> and detail, and keep both in the worker's Runtime record

Requirement: Worker Dead Letters

When a worker fails on a message (it throws, or a native worker returns an error status), the Runtime SHALL NOT retry it. It SHALL publish the original input CloudEvent exactly once to dlq.<topic> of the topic it arrived on (naming library DeadLetter), with the failure in extension attributes defined once in Virtufin.Worker.Host.WorkerAttributes, and SHALL then deliver the worker's next message.

Scenario: Worker throws on a message

  • WHEN a worker throws while processing a message from topic T
  • THEN exactly one dead letter SHALL be published to dlq.T: the input event as it arrived, plus dlqworkerid, dlqactortype (the worker's Runtime actor type), dlqerrortype, dlqerrormessage (at most 1024 characters, no stack trace) and dlqfailedtime
  • AND the message SHALL NOT be redelivered to the worker, and the worker SHALL receive the next message on its topics

Scenario: Native worker error status

  • WHEN a native worker's Process returns 1 or another non-zero status
  • THEN the message SHALL be dead-lettered exactly as if the worker had thrown

Scenario: Dead-letter publish fails

  • WHEN publishing the dead letter fails
  • THEN the Runtime SHALL log the failure at error level and continue with the next message, without retrying the publish

Scenario: No secrets in the dead letter

  • WHEN a worker created with secrets fails on a message
  • THEN the dead letter SHALL be the event as it arrived, before the worker's config and secret values were added

Requirement: Worker Wire Protocol

Every hop that carries a CloudEvent between the WorkManager, its Runtime and a worker SHALL use the CloudEvents Protobuf format (io.cloudevents.v1): a request SHALL be one CloudEvent message, a response one CloudEventBatch message whose events are published in order (an empty batch means no output). Binary data SHALL travel verbatim in binary_data. Errors SHALL be signalled by the transport, never inside a CloudEvent. One codec (CloudEventProtoCodec, Virtufin.Worker.Host) SHALL implement the mapping for every .NET hop.

Scenario: Binary data round trip

  • WHEN a worker returns a CloudEvent whose data is bytes (e.g. an order book image)
  • THEN every hop SHALL carry those bytes unchanged in binary_data, without base64

Scenario: Subprocess frames

  • WHEN the Runtime talks to a subprocess engine (Python, C# source, TypeScript)
  • THEN each frame SHALL be a 4-byte big-endian length, a 1-byte frame type and a payload: process carries a CloudEvent, result a CloudEventBatch, error a UTF-8 message, and ready, ping, pong and shutdown an empty payload

Scenario: Python workers see the same event

  • WHEN the Python runner decodes a process frame
  • THEN the worker's Process(event) SHALL receive the CloudEvents-JSON-shaped dict it received before (data parsed for JSON content, data_base64 for bytes), and the runner SHALL encode the returned dict(s) back into a CloudEventBatch using only the Python standard library

Scenario: TypeScript workers see the same event

  • WHEN the TypeScript engine's Node bridge decodes a process frame
  • THEN the worker's Process(event) SHALL receive the same dict form a Python worker receives (every attribute a top-level key, text data in data, binary data base64 in data_base64, timestamps as RFC 3339 text), and the bridge SHALL encode the returned event(s) back into a CloudEventBatch with no dependency beyond Node's standard library

Scenario: Subscription to worker actor

  • WHEN the Runtime's topic subscription hands a message to the actor hosting a worker
  • THEN it SHALL call the actor's ProcessAsync with the CloudEvent bytes and receive the CloudEventBatch bytes

Requirement: Polyglot Worker Engines

The WorkManager SHALL support executing worker code in multiple languages via a pluggable engine architecture. Built-in engines SHALL include Python (subprocess), TypeScript (a Node.js subprocess), C# source (Roslyn compilation in a CSharpSource.Bridge subprocess), pre-compiled .NET DLL (the default DotNetDllEngine via hostfxr + AssemblyLoadContext), and a native DLL engine (NativeDllEngine, see the Native DLL Engine / Native Worker ABI requirements below). Every engine SHALL be hosted by a WorkManager.Runtime pod through its per-engine actor; the WorkManager itself SHALL NOT run worker code.

Scenario: Python worker execution

  • WHEN a worker with a Python engine processes a message
  • THEN the worker code SHALL execute in a dedicated Python subprocess with a configurable timeout

Scenario: TypeScript worker execution

  • WHEN a worker with MIME type text/x-typescript (one ES module) or application/x-npm (an npm pack tarball whose main is the module) processes a message
  • THEN the module SHALL run in a dedicated node subprocess speaking the subprocess protocol, its TypeScript type-stripped by Node without a compile step (erasable syntax only), with the worker's Start(ctx)/Process(event)/Stop() exports or a default-exported worker object or class (start/process/stop) as its entry points, any of them optional and possibly asynchronous
  • AND a module that fails to load SHALL fail the load with Node's error, without waiting for a timeout

Scenario: TypeScript worker packages

  • WHEN a TypeScript worker is loaded
  • THEN the engine SHALL install its dependencies with npm into the worker's own directory: a module's bare import specifiers (Node built-ins and relative imports excluded) at their latest versions, a package's dependencies with every @virtufin/* one at its latest version
  • AND @virtufin/* SHALL resolve only from the private registry (npm scope mapping) with the Runtime's registry token passed through the environment and never written to disk; install scripts SHALL NOT run
  • AND the Runtime's registry credentials SHALL NOT be in the worker subprocess's environment

Scenario: C# source worker execution

  • WHEN a worker with a C# source engine processes a message
  • THEN the code SHALL be compiled at runtime via Roslyn and executed in the CSharpSource.Bridge subprocess

Scenario: C# source worker packages

  • WHEN a C# source worker starts with #r "nuget: <Id>", #r "nuget: <Id>, <version-or-range>" or <Id> [<version>] directives
  • THEN the engine SHALL resolve those packages and their dependencies with the same installer, feeds, credentials and version policy as DLL workers (Virtufin.* at the newest release in range), compile against them, and load host-shared assemblies (Virtufin.Worker.Host, CloudEvents, Virtufin.Api.Client) from the host so the worker sees the host's types
  • AND an unresolvable package, a non-NuGet #r, or a compile error (with its line number) SHALL fail the load

Scenario: C# source worker entry point

  • WHEN a C# source worker defines exactly one non-abstract class implementing IWorker with a public parameterless constructor
  • THEN the engine SHALL run it the way the DLL engine runs a DLL worker, publishing every output in order
  • AND a source without such a class MAY instead declare a CloudEvent ProcessAsync(CloudEvent) method; two such classes SHALL fail the load

Scenario: DLL worker execution (in-process, default)

  • WHEN a worker with MIME type application/x-dotnet-dll processes a message
  • THEN the worker DLL SHALL be loaded into an isolated AssemblyLoadContext and executed inside the hosting Runtime process

Scenario: Engine extension

  • WHEN a new engine implementation conforms to the IEngine interface
  • THEN it SHALL be added as a Runtime actor type and mapped from its MIME type (RuntimeActorTypeNames), without changing how the WorkManager routes workers

Requirement: Event-Driven Message Processing

Workers SHALL subscribe to Dapr pub/sub topics and process incoming CloudEvents. Each output CloudEvent's publishtopic extension, when present, SHALL determine its publish topic (the extension is removed before publishing); otherwise its type field SHALL.

Scenario: Message processing

  • WHEN a CloudEvent is published to a topic a worker subscribes to
  • THEN the worker's code SHALL be invoked with the event data and any output events SHALL be published to the topic named by their publishtopic extension, or else their type field

Scenario: Several outputs from one input

  • WHEN a multi-output worker returns several CloudEvents for one input
  • THEN every one of them SHALL be published, in the order returned, each to its own topic by the rule above
  • AND an empty result SHALL publish nothing

Scenario: Shared per-event rules and conformance vectors

  • WHEN any host runs a worker on an event (the WorkManager.Runtime with any engine, or an in-process local host such as the devkit's LocalWorkManager)
  • THEN it SHALL apply the per-event rules of Virtufin.Worker.Host.WorkerEventRules, not its own copy: PrepareInput (a missing datacontenttype becomes application/json, the worker's config fills only attributes the event leaves out, the worker's secrets are set over everything), FinishOutputs (a missing id becomes a fresh one, a missing source the input's, secret attributes are removed, the type has no default), Route (by the rule above; an output with neither is not published) and DeadLetter (the dead letter of the Worker Dead Letters requirement)
  • AND the host SHALL pass the conformance vectors conformance/worker-event-rules.v1.json shipped in the Virtufin.Worker.Host package (codec cases and rule vectors), using a vector's expect_python where the worker runs in the Python or the TypeScript engine (the dict-form wire path both share)

Scenario: Concurrent message handling

  • WHEN multiple messages arrive for the same worker
  • THEN they SHALL be processed concurrently (the engine's ProcessAsync is re-entrant; the worker author is responsible for internal synchronization)

Requirement: Worker-Addressed Topics

A CreateWorker topic MAY contain the placeholder {worker_id} (the naming library renders it for wrk.<worker_type>.command.<worker_id>). The WorkManager SHALL replace every occurrence with the id it mints for the worker, before the worker is registered anywhere, and SHALL persist and report the expanded topic. A topic names what its data is, not who happens to consume it: commands meant for one worker instance SHALL be addressed by that instance's id, never by a caller-side run or session tag.

Scenario: Command topic addressed by the worker

  • WHEN a client creates a worker with topic wrk.httpget.command.{worker_id}
  • THEN the worker SHALL be subscribed to wrk.httpget.command.<its id>, and GetWorker/ListWorkers SHALL report that expanded topic
  • AND the client SHALL derive the topic from the id CreateWorker returns (to target a trigger or publish commands)

Scenario: Orphans stay isolated

  • WHEN a client dies before deleting its workers and triggers, and a later client creates workers of the same kind
  • THEN the orphaned triggers SHALL only reach the orphaned workers, because each worker's command topic is its own

Scenario: Commands across a move

  • WHEN commands are published on a running worker's command topic while the worker moves to another Runtime pod (a rebalance, a pod replaced or killed, a reopened stream), when no pod is subscribed to it
  • THEN the activation the worker continues on SHALL receive each of them exactly once, in order, after the commands it had not yet handled: the Runtime SHALL persist a cursor per worker (sys.workmanager-runtime.worker.<id>.commands: since when, and the ids handled since), saved after each command it handles (processed or dead-lettered), and SHALL open a continuing worker's command topic with the API's replay_since from it (less a clock margin), skipping what is listed
  • AND a worker that is stopped and started again, or newly started, SHALL start at the tail (its cursor is deleted on stop and delete); its other input topics SHALL follow Scenario: Inputs across a move
  • AND a replay the API refuses SHALL be logged as an error naming the worker and the time, and the topic SHALL open at the tail
  • AND a command handled but not yet recorded when its pod died MAY be delivered again (at least once at a crash)

Scenario: Inputs across a move

  • WHEN events are published on a running worker's input topics (any topic but its command topic) while the worker moves to another Runtime pod (a rebalance, a pod replaced or killed, a reopened stream), and the worker's config does not say replay=off
  • THEN the activation the worker continues on SHALL receive each of them exactly once, in topic order, before the later live ones: the Runtime SHALL persist per worker the cursor of each input topic (sys.workmanager-runtime.worker.<id>.inputs: the id of the last event finished, and a time at or before its stream entry), batched (at most one State write per worker every few hundred milliseconds, never one per event) and settled when the worker stops on that pod while it still holds the lease; and SHALL catch a continuing worker up per topic from its cursor (the API's replay_since with replay_only, less a clock margin, skipping through the cursor's event), merged with the topic's shared live stream by CloudEvent id, without affecting other workers on the topic
  • AND a catch-up the API refuses (too many entries or bytes retained, no stream history), one that fails, or a replay that no longer holds the cursor's event SHALL be logged as a warning and published as a lifecycle event with state gap (topic, from, to, reason), and the worker SHALL continue live
  • AND a worker whose config says replay=off SHALL start every input topic at the tail after a move (workers that resynchronise themselves, such as the order-book workers); StopWorker and DeleteWorker SHALL delete the input cursors, so a stopped or new worker starts at the tail
  • AND an event finished but not yet recorded when its pod died MAY be delivered again (at least once at a crash); workers SHALL be written to tolerate a repeated input event or document that they do not

Requirement: Trigger Ownership

A trigger MAY name the worker it exists for (owner_worker_id on Trigger.ScheduleTrigger). When the WorkManager deletes a worker it SHALL delete every trigger owned by that worker, after the worker itself is deleted, so a trigger never outlives the worker it drives. A failure to delete owned triggers SHALL be logged and SHALL NOT fail the worker deletion. Triggers without an owner SHALL be unaffected.

Scenario: Deleting a worker deletes its triggers

  • WHEN a worker that owns triggers is deleted
  • THEN its owned triggers SHALL be deleted and SHALL stop firing
  • AND triggers owned by other workers, and unowned triggers, SHALL remain

Requirement: Cross-Instance Coordination

Messages SHALL reach Runtime processes by per-sidecar delivery, not competing consumers: the pub/sub component gives every sidecar its own consumer group (consumerID: {podName}), so every process that holds a subscription for a topic receives every message on it (Pub/Sub Topics spec, "Subscription Implementation Pattern"). A process subscribes to a topic only while it hosts a worker for it, and each worker is hosted by exactly one process (Ownership via Dapr Actor Placement), so each worker processes each message exactly once. A message that reaches a process with no worker registered for the topic (e.g. one that left between delivery and dispatch) SHALL be dropped, acked with no redelivery (an explicit business-logic skip per the Pub/Sub Topics spec's "Dead-Letter Policy on Subscribe Exceptions" requirement, not a processing failure).

Scenario: Message lands on a replica with no matching worker

  • WHEN a message reaches a Runtime process that has no worker registered for that topic
  • THEN the message SHALL be dropped and acked with no redelivery (an explicit business-logic skip, not a processing failure)
  • AND no re-routing SHALL occur: every process hosting a worker for the topic receives the message itself

Scenario: Every subscribed process receives every message

  • WHEN two Runtime pods each host a worker subscribed to topic orders
  • THEN each process SHALL receive every message published to orders, and each worker SHALL process it once
  • AND no message SHALL be load-balanced between the processes

Requirement: Ownership via Dapr Actor Placement

Each worker SHALL be a Dapr actor in the WorkManager.Runtime, of the actor type of its engine (RuntimeActorTypeNames, from the MIME type), keyed by the worker id. Placement keeps at most one activation per id, so exactly one Runtime process hosts a worker. The WorkManager is the catalog and router: it SHALL host no worker code and SHALL drive every worker operation (create, load, start, stop, delete, code reload) by calling the worker's actor. There SHALL be no instance directory, peer forwarding or reclaim; a worker record's InstanceId is informational.

Scenario: Operation on any WorkManager replica

  • WHEN a worker RPC (StartWorker, StopWorker, DeleteWorker, LoadCode) reaches any WorkManager replica
  • THEN the replica SHALL call the worker's actor, and placement SHALL deliver the call to the Runtime process hosting it

Scenario: Activation on another pod

  • WHEN a worker's actor is activated on a Runtime pod that never loaded it (failover, rebalance, idle reactivation)
  • THEN the activation SHALL load the worker from its Runtime record and resubscribe its topics if it was running

Scenario: Deactivation stops delivery on the old host

  • WHEN a worker's actor is deactivated on a Runtime pod (placement moved it, idle timeout, pod shutdown)
  • THEN that pod SHALL unsubscribe the worker's topics (closing a topic's stream when no other worker on the pod uses it) and release the worker's engine, without changing the worker's Runtime record
  • AND messages the pod still receives on those topics SHALL NOT be delivered to the worker's actor from that pod, so a worker moved by placement processes each message once, on its new host

Scenario: Graceful drain on Runtime shutdown

  • WHEN a Runtime pod is asked to stop (SIGTERM)
  • THEN it SHALL first stop every started worker (stop token, StopAsync, lease revoked) within a bounded budget, then close its topic streams, so no new message enters a worker's inbox
  • AND it SHALL then let each worker finish the messages it already holds, in order, and publish everything in its outbox (responses and dead letters), within a bounded drain budget
  • AND it SHALL then release every engine, and only then exit; what is still queued when the budget runs out is dropped and logged with a count per worker
  • AND the deployment SHALL give the pod a termination grace period that covers the drain and keep the sidecar available for it

Requirement: Worker Management Client Contract

There SHALL be one client contract for managing workers, IWorkManagerClient (Virtufin.WorkManager.Client), implemented by the cluster client (GatewayWorkManagerClient, through the api gateway) and by the in-process host (LocalWorkManager). Both SHALL accept the same WorkerSpec (CodeSource, mime type, topics, config, tags, secrets) and expose the same state model (Loading, Stopped, Running, LoadFailed) and operations (create, load code, start, stop, delete, list with tag filters, tags). CodeSource SHALL be a URL (with an optional credential), inline content, or an already constructed worker instance; the instance form SHALL be accepted by in-process hosts only. Local hosts SHALL load URL and content sources through pluggable IWorkerLoaders that mirror the Runtime's engines (Virtufin.WorkManager.Engines.Local: nupkg and C# source).

Scenario: Same spec on both hosts

  • WHEN a WorkerSpec with a nupkg URL or C# source content is deployed through the cluster client and through a local host with the engine loaders
  • THEN both SHALL load the same code, apply the same per-event rules and produce outputs identical by content

Scenario: Instance rejected by the cluster

  • WHEN a CodeSource.Instance is given to the cluster client
  • THEN it SHALL fail with a not-supported error before any RPC is sent

Scenario: Start while loading

  • WHEN StartWorkerAsync is called on a worker whose code is still loading, on either host
  • THEN the start SHALL take effect once the load finishes, and SHALL fail with a load-failed error if the load fails

Requirement: Worker Code Sources

Workers SHALL accept code from inline Base64-encoded content or from HTTP(S) URLs. URL-based sources SHALL be validated against SSRF and host allowlist constraints as defined in the cross-cutting security requirements. URL-based sources MAY carry a per-request credential (see Worker Source Authentication) used only for that fetch.

Scenario: Inline code worker

  • WHEN a worker is created with a Base64-encoded code string
  • THEN the decoded code SHALL be passed to the engine for execution

Scenario: URL code worker

  • WHEN a worker is created with a URL pointing to code
  • THEN the code SHALL be fetched, SHA-256 verified, and passed to the engine

Scenario: Blocked URL

  • WHEN a worker's code URL targets a private IP or a host not in the allowlist
  • THEN the worker creation SHALL fail with a security rejection

Requirement: Python Sandboxing

Python worker subprocesses running raw source code (text/x-python) SHALL run with restricted interpreter flags and a package allowlist; all non-allowed imports SHALL be blocked. A Python worker packaged as a wheel (application/x-python) is NOT subject to this blocklist/allowlist: its dependencies are declared and vetted via its build-time Requires-Dist manifest instead of being restricted at runtime.

Scenario: Allowed import

  • WHEN a raw-source Python worker imports a package listed in PYTHON_ALLOWED_PACKAGES
  • THEN the import SHALL succeed

Scenario: Blocked import

  • WHEN a raw-source Python worker imports a package not in the allowlist
  • THEN the import SHALL fail with an error

Scenario: Wheel-packaged worker import

  • WHEN a Python worker packaged as a wheel imports any module
  • THEN the import SHALL succeed regardless of the blocklist/allowlist, since its dependencies were vetted at publish time via the wheel's declared Requires-Dist manifest, not restricted at runtime

Requirement: Worker Contract (IWorker)

The IWorker interface (Virtufin.Worker.Host) SHALL define StartAsync(IWorkerContext context, CancellationToken stop), ProcessAsync(CloudEvent input, CancellationToken ct) returning Task<IReadOnlyList<CloudEvent>> (zero or more outputs for one input, in publish order) and StopAsync(CancellationToken deadline). There SHALL be no separate single- or multi-output contract, so no host dispatches on the worker's type. Every host (the Runtime's engines and bridges, the devkit's local WorkManager, the test hosts) SHALL call StartAsync once before the first ProcessAsync of an activation and StopAsync after the last. The context SHALL be handed over once, at StartAsync, and be the activation's: the worker's id, an IVirtufinApi (state, pubsub, triggers, gateway) whose publishes route through the host, a logger, the worker's Config and Secrets snapshots, State (the worker's own keys under sys.workmanager.worker.<id>.state.<key>, refused when a value contains a secret value, deleted with the worker), PublishAsync (through the host's outbox under the output rules, gated by ownership) and ReportStatusAsync. Nothing SHALL be ambient: a worker that needs the platform reads it from the context, never from the environment, so the same worker runs under the Runtime (the configured gateway) and under a local host (an in-memory api). The stop token SHALL be cancelled when the host loses ownership of the worker (lease lost, stop, reload, delete, deactivation, drain); from then on the host SHALL publish nothing of the worker's, SHALL never call ProcessAsync again, and SHALL call StopAsync with a bounded deadline.

Scenario: The context reaches the worker

  • WHEN any host starts a worker
  • THEN context.WorkerId SHALL be the worker's id in that host, context.Api the host's api, context.Config and context.Secrets the values the host stamps on every input
  • AND a C# source worker SHALL compile against the context's types (logging abstractions, Virtufin.Api.Client) without declaring them

Scenario: Start before the first Process

  • WHEN a host has an input for a worker it has not started
  • THEN it SHALL start the worker first (the Runtime refuses the input with FAILED_PRECONDITION when the worker is not running)

Scenario: A host publish is routed like an output

  • WHEN a worker calls context.PublishAsync (or its context.Api publish methods)
  • THEN the host SHALL apply the output rules (a fresh id, secret attributes stripped), route the event to its publishtopic else its ce-type, drop one with neither, and publish it in order with the worker's responses through its outbox

Scenario: Ownership lost

  • WHEN the host loses ownership of a started worker
  • THEN it SHALL cancel the stop token, refuse every later publish of the worker's, call StopAsync with a deadline, and never call ProcessAsync on that activation again

Scenario: Async worker process

  • WHEN the Runtime invokes a worker's ProcessAsync method
  • THEN the call SHALL be awaited asynchronously without blocking threads

Scenario: Zero, one or several outputs

  • WHEN the engine hosts a worker and calls its ProcessAsync
  • THEN it SHALL publish every returned CloudEvent in order
  • AND an empty list SHALL publish nothing

Scenario: Every engine carries the lifecycle

  • WHEN a C# source, Python, TypeScript or native (ABI v3) worker is started or stopped by the Runtime
  • THEN its engine SHALL hand it the activation's context and call its start and stop exactly as the DotNetDll engine does: the subprocess engines over the subprocess protocol (start/stop frames, the start acknowledged within the engine's startup timeout and never under the per-message timeout), the native engine through the Start/Stop exports
  • AND the worker's publishes, status reports (and for subprocess workers its state and fresh secrets) SHALL go back to the host as host calls, from any of the worker's threads and at any time between its start and the return of its stop, through the host's outbox and ownership gate
  • AND from the moment the host's stop token fires, the engine SHALL refuse the worker's publishes
  • AND the lifecycle conformance vectors SHALL pass through the Runtime on each engine

Scenario: Python worker lifecycle

  • WHEN a Python worker module defines Start(ctx) and/or Stop() next to Process(event)
  • THEN Start SHALL be called once per activation before the first event with a ctx carrying worker_id, config, secrets, a stopping event, publish, report_status, state (get/save/delete/keys) and get_secrets, and Stop after the last event with ctx.stopping already set
  • AND a refused host call SHALL raise HostRefused and a stale etag StateConflict; the devkit's LocalWorkManager SHALL offer the same ctx

Scenario: A host publish without a route

  • WHEN a worker publishes through its context an event with neither publishtopic nor type
  • THEN every engine SHALL drop it (logged) and publish the rest

Requirement: Worker Lease

A started worker SHALL hold a lease: on every start and every self-heal the Runtime SHALL claim the API's publication fence keyed by the worker id (a fresh token superseding the persisted one), persist the token in the worker's Runtime record, start the worker only after the claim, renew the lease periodically (default every 1 s, lease 10 s) and check it locally before every outbox batch and host publish. A rejected renewal (another activation claimed the worker) SHALL stop the worker on this activation (gate closed, stop token, StopAsync), leave its topics and publish a status lifecycle event (status = lease-lost), the record being the successor's. It SHALL start the worker there again only once nobody holds the lease: a poke or call reaching it SHALL, at most once per renewal interval, reload the persisted record and, if it still says running, claim under its own superseded token, which the fence refuses while the successor's lease is live. Renewals that fail until the lease lapses SHALL stop the worker the same way and mark it faulted (failed); the next poke or call SHALL claim and start it again (online). The lease SHALL be revoked on stop, delete, deactivation, reload and the shutdown drain.

Scenario: Takeover

  • WHEN a successor activation claims a worker's lease while the old activation still runs
  • THEN the old activation SHALL publish nothing after its next renewal, SHALL call the worker's StopAsync with the stop token cancelled, and SHALL not touch the Runtime record

Scenario: Reactivation claims with the persisted token

  • WHEN a worker's actor activates on a Runtime pod and its record says running
  • THEN the activation SHALL claim the lease with the record's token, start the worker and resubscribe; a refused claim SHALL fault the worker and fail the activation

Scenario: Superseded activation after the successor is gone

  • WHEN an activation whose lease was taken over receives a poke or call after the successor's lease was revoked or expired, and the persisted record says running
  • THEN it SHALL claim the lease, start the worker and resubscribe, so the worker never stays listed running while not started; while the successor's lease is live, or once the successor stopped the worker, it SHALL not start it

Requirement: Worker Commands

The WorkManager SHALL expose InvokeWorkerCommand(id, command, payload, content_type, attributes, timeout_ms): the command SHALL become a CloudEvent of type com.virtufin.workmanager.worker.command (source urn:com.virtufin:workmanager, subject worker/<id>, the command name in the command extension, the attributes as string extensions, the payload as data) that goes through the worker's actor ProcessAsync like any input, serialized with its inbox; the outputs SHALL be returned flattened and SHALL NOT be published. The local WorkManager SHALL implement the same (IWorkManagerClient.InvokeCommandAsync).

Scenario: Command to a running worker

  • WHEN a client invokes a command on a running worker
  • THEN the worker's ProcessAsync SHALL see the command event with its config and secrets stamped, and the client SHALL receive every output in order, none of them published

Scenario: Command to a worker that is not running

  • WHEN a client invokes a command on a stopped, loading or load-failed worker
  • THEN the WorkManager SHALL answer FAILED_PRECONDITION without calling the actor

Requirement: Topic Dependencies

A worker's input topics (CreateWorker topics) SHALL be its whole dependency: it subscribes to them and SHALL NOT name the worker that produces them. The WorkManager SHALL NOT match topics to producers. An orchestrator outside the platform (a notebook, a script, the TUI) SHALL open the producers of the topics it needs (for venue data the WebSocketFeed workers that serve them, WebSocket Feed, Feed Orchestration); workers and producers MAY start in any order. The devkit's former requires=<type>/<name> attach/detach hook is removed (worker-devkit 0.40.0): a config key of that name is ordinary config.

Scenario: A worker and its feed start in any order

  • WHEN a market-data worker is created with its input topics and an orchestrator applies the feeds that serve them, in either order
  • THEN the worker SHALL receive the frames once both run, and neither SHALL send the other anything

Requirement: Worker Base Classes

Virtufin.Worker.DevKit SHALL provide the abstract WorkerCore<TInput, TOutput> (an IWorker constructed with a worker type and an IPayloadCodec<TInput, TOutput>, owning the envelope helpers BuildResponse and Error) and, on it, the abstract WorkerBase<TInput, TOutput>: it decodes the input's data with the codec, calls the typed ProcessAsync(CloudEvent, TInput) override, and returns its one result in the response envelope (BuildEnvelope), or no output for null. Virtufin.Behavior.DevKit (built in the worker-devkit repo) SHALL provide BehaviorWorkerBase, a WorkerCore<JsonNode, JsonObject> that returns every output of a bound behaviour process (behaviour spec, Worker Translation). There SHALL be no command-dispatch base class in C#: a worker that takes commands reads them from its typed input.

Scenario: Worker named by its worker type

  • WHEN a worker of type <type> (e.g. httpget) builds a response or an error
  • THEN its source SHALL be urn:com.virtufin:worker:<type>, a response SHALL have ce-type com.virtufin.<type>.response on topic wrk.<type>.response, and an error ce-type com.virtufin.<type>.error on topic wrk.<type>.error (WorkerTypeNames, from the naming library)
  • AND an incoming replytopic SHALL change the topic of both, never the ce-type

Scenario: Response envelope

  • WHEN WorkerBase builds a response
  • THEN it SHALL give the response a new id (never the input's), set source, type, time and correlationid, and set the publishtopic extension to the response topic

Scenario: Unhandled exception

  • WHEN the typed ProcessAsync override throws
  • THEN WorkerBase SHALL return an error response with message Internal error: <exception message>

Scenario: API client

  • WHEN a worker needs the API gateway
  • THEN it SHALL use context.Api (the IVirtufinApi of its IWorkerContext: the host's configured gateway under the Runtime, a local host's in-memory API otherwise)
  • AND only an explicit other endpoint (the apihost config) SHALL be dialled through WorkerApiEndpoint.CreateClient(endpoint), which attaches the host's key only to the host's own configured gateway

Requirement: Python Worker Base Classes

The Python worker devkit (virtufin.worker_devkit, in the workmanager repo) SHALL provide WorkerBase(worker_type=...) with an abstract handle(event) returning none, one or a list of CloudEvent dicts, named by the worker type like the C# WorkerBase. It SHALL also provide CommandWorker, ApiWorker and ApiCommandWorker.

Scenario: Command dispatch

  • WHEN a CommandWorker receives an event whose data is a JSON object with a command field
  • THEN it SHALL call handle_command(input, command, data)

Scenario: Command errors

  • WHEN the event data is not valid JSON, has no command field, or handle_command returns None
  • THEN the worker SHALL return an error response (Invalid JSON in event data, Missing 'command' field, Unknown command: <command>)

Scenario: API workers

  • WHEN an ApiWorker or ApiCommandWorker is created with an optional api_host/api_port
  • THEN it SHALL hold an API client for that endpoint, else for the host's configured gateway (self.api); ApiWorker passes it to process_async(input, api)

Scenario: Unhandled exception

  • WHEN handle raises
  • THEN process SHALL return an error response with message Internal error: <exception>

Requirement: Correlation ID Propagation

Worker responses SHALL propagate the correlationid extension attribute from the input CloudEvent to the output CloudEvent. The CloudEvent.WithCorrelationId(input) extension method SHALL handle this as a no-op when absent.

Scenario: Response with correlation ID

  • WHEN a worker creates a response or an error via the base class helpers (BuildResponse/Error in C#, response/error in Python)
  • THEN the output CloudEvent SHALL carry the same correlationid as the input CloudEvent if present

Scenario: Input without correlation ID

  • WHEN a worker processes an input CloudEvent without a correlationid extension
  • THEN the output CloudEvent SHALL NOT have a correlationid extension

Requirement: In-Process DotNet DLL Engine

The WorkManager SHALL support an in-process DotNetDllEngine (default for MIME type application/x-dotnet-dll) that loads pre-compiled .NET worker DLLs directly into the AOT-compiled Runtime process that hosts the worker. The engine SHALL embed the .NET runtime (CoreCLR) into the host process on first use via the hostfxr C API and SHALL load the worker DLL into a per-worker, collectible AssemblyLoadContext. The engine SHALL be AOT-compatible: the engine code itself is compiled with <IsAotCompatible>true</IsAotCompatible> and SHALL NOT use reflection on its own types; the only reflection SHALL be on the JIT-loaded worker assembly.

Scenario: First LoadCodeAsync initializes CoreCLR

  • WHEN the first LoadCodeAsync is called for MIME type application/x-dotnet-dll
  • THEN the engine SHALL load libhostfxr via NativeLibrary.Load and call hostfxr_initialize_for_runtime_config to initialize CoreCLR in the host process; the cost SHALL be paid once per Runtime process

Scenario: Subsequent LoadCodeAsync in the same process

  • WHEN a second LoadCodeAsync is called after CoreCLR is already initialized
  • THEN the engine SHALL skip the hostfxr initialization and load the new worker DLL directly

Scenario: Per-worker AssemblyLoadContext

  • WHEN a LoadCodeAsync is called with a worker nupkg
  • THEN the engine SHALL extract the worker DLL (and its sibling dependencies) from the nupkg's lib/<tfm>/ entries, create a new collectible AssemblyLoadContext, and load the worker assembly into it

Scenario: Host resolution

  • WHEN the JIT-loaded worker assembly references Virtufin.Worker.Host
  • THEN the engine SHALL resolve the reference to the Virtufin.Worker.Host instance already loaded in the Runtime process (so the worker and the engine see the same type identity for IWorker, IWorkerContext, IVirtufinApi and CloudEvents)
  • AND Virtufin.Worker.DevKit (the authoring layer: WorkerBase, codecs, envelope helpers) SHALL load into the worker's own assembly load context from the worker's dependencies, so it may be newer than the deployed runtime as long as its Host requirement is satisfied

Scenario: Direct in-process dispatch

  • WHEN ProcessAsync is called on the engine with a CloudEvent
  • THEN the engine SHALL call the worker's IWorker.ProcessAsync directly, with no socket, no JSON serialization, and no subprocess scheduling; the per-call latency SHALL be sub-microsecond

Scenario: Runtime discovery on target

  • WHEN the Runtime runs on a target that does not have a compatible .NET runtime installed
  • THEN the first LoadCodeAsync SHALL fail with a clear error message that names the missing runtime and the discovery paths (DOTNET_ROOT, system install, side-by-side host/fxr/<version>/)

Scenario: Roll-forward to a newer runtime

  • WHEN the Runtime is built for .NET 10 and the target has a .NET 11+ runtime installed
  • THEN the engine SHALL use the newer runtime; the generated runtimeconfig.json SHALL declare "rollForward": "Major"

Requirement: Native DLL Engine

The WorkManager SHALL support a NativeDllEngine for executing workers shipped as a per-architecture native shared library (.so / .dll / .dylib). The engine SHALL run in-process inside the (NativeAOT-compiled) Runtime and SHALL NOT require a separate worker-host subprocess or a JIT build stage. The engine SHALL be registered for MIME type application/x-native-dll. Per-architecture entries SHALL live under runtimes/<rid>/native/ inside the worker zip, matching the NuGet convention for native packages.

Scenario: Native worker execution

  • WHEN a worker with MIME type application/x-native-dll processes a CloudEvent
  • THEN the engine SHALL NativeLibrary.Load the matching runtimes/<rid>/native/<file> entry from the worker zip, resolve the entry_point and free_result exports declared in manifest.json, encode the input CloudEvent as a protobuf CloudEvent, invoke the native Process function, and decode the returned CloudEventBatch.

Scenario: Per-architecture entry selection

  • WHEN a worker zip contains entries under runtimes/<rid>/native/ for at least one supported RID (linux-x64, linux-arm64)
  • THEN the engine SHALL select the entry whose <rid> directory matches RuntimeInformation.RuntimeIdentifier of the running process and SHALL load the corresponding lib<library>.so, <library>.dll, or lib<library>.dylib per the platform (see Native Worker Library Naming).

Scenario: Unsupported architecture

  • WHEN the worker zip does not contain a runtimes/<rid>/native/ entry for the running architecture
  • THEN the engine SHALL throw NotSupportedException listing the supported archs (linux-x64, linux-arm64).

Scenario: Missing manifest

  • WHEN the worker zip does not contain manifest.json at the root
  • THEN the engine SHALL refuse to load the worker and throw a clear error naming the missing file.

Scenario: ABI version mismatch

  • WHEN the manifest.json declares an abi_version that is not in the engine's supported set
  • THEN the engine SHALL refuse to load the worker and throw a clear error naming the supported versions.

Requirement: Native Worker ABI

The NativeDllEngine SHALL define and enforce a stable C ABI between the managed engine and the native worker. The ABI SHALL consist of: (a) the Worker Wire Protocol messages (CloudEvent in, CloudEventBatch out); (b) two C function exports Process and FreeResult; (c) a VirtufinHost struct carrying host callbacks; (d) the worker nuspec's virtufin* elements declaring the ABI version (virtufinAbiVersion), the library basename and the export names. The engine SHALL vendor virtufin_worker_api_c.h and a header-only protobuf codec virtufin_worker_api_pb.h for worker authors. ABI version 2 is the Worker Wire Protocol; ABI version 3 is version 2 plus the worker lifecycle (Start/Stop exports and the host callbacks publish and report_status). Both are supported.

Scenario: ABI struct layout

  • WHEN a native worker is loaded
  • THEN the engine SHALL pass a VirtufinHost* whose first field is an opaque engine_ptr, whose second field is abi_version (the host's version, 3), and whose remaining fields are C function pointers log, gateway_call, free_response, publish and report_status, the last two after every field an ABI v2 worker reads.

Scenario: ABI v3 lifecycle

  • WHEN a worker's nuspec declares virtufinAbiVersion 3
  • THEN the library SHALL export Start(host, ctx_buf, ctx_len, &out_buf, &out_len) and Stop(host, budget_ms), or the engine SHALL refuse to load it
  • AND the engine SHALL call Start once per activation before the first Process, with ctx_buf the UTF-8 JSON {"workerId", "config", "secrets"}; a 1 return SHALL fail the start with the UTF-8 message in *out_buf (released by FreeResult)
  • AND until Stop returns the worker MAY call host->publish(batch, len) (a protobuf CloudEventBatch, returning 0 published, 1 refused, -1 failed) and host->report_status(state, detail) from any thread; Stop SHALL be called after the last Process and SHALL return within budget_ms

Scenario: ABI v2 worker without lifecycle

  • WHEN a worker declares ABI version 2
  • THEN the engine SHALL load and process it unchanged and call no lifecycle export

Scenario: CloudEvent input

  • WHEN the engine invokes Process(host, in_buf, in_len, &out_buf, &out_len) for an ABI v2 worker
  • THEN in_buf SHALL hold a protobuf io.cloudevents.v1.CloudEvent; the correlationid attribute SHALL carry the correlation id when present.

Scenario: CloudEventBatch output

  • WHEN Process returns 0
  • THEN *out_buf SHALL hold a protobuf CloudEventBatch, and the engine SHALL publish every event in it, in order; an empty batch publishes nothing.

Scenario: Worker error

  • WHEN Process returns 1
  • THEN *out_buf SHALL hold a UTF-8 error message, and the engine SHALL fail the call with that message (the Runtime dead-letters the input, Worker Dead Letters; no response is published).

Scenario: Internal failure

  • WHEN Process returns any other non-zero value
  • THEN the engine SHALL treat the call as failed (dead-lettered like a worker error); *out_buf MAY be NULL.

Scenario: FreeResult pairing

  • WHEN Process returns a non-null *out_buf
  • THEN the engine SHALL call FreeResult(out_buf) exactly once, including on the error path. The worker SHALL use the same allocator for the *out_buf allocation that FreeResult releases.

Scenario: Logging callback

  • WHEN the native worker calls host->log(level, message)
  • THEN the engine SHALL map level (0=trace, 1=debug, 2=info, 3=warn, 4=error) to the corresponding Microsoft.Extensions.Logging.LogLevel and forward message to the engine's structured logger. The lifetime of message SHALL be limited to the call.

Scenario: Gateway call indirection

  • WHEN the native worker calls host->gateway_call(service, method, request, request_len, &response, &response_len)
  • THEN the engine SHALL deserialize request as JSON into a Dictionary<string, object?>, invoke ApiClient.InvokeAsync(service, method, dict), JSON-serialize the returned dictionary into *response (allocated with Marshal.AllocCoTaskMem), and return 0 on success or non-zero on internal failure. The worker SHALL free *response via host->free_response(response).

Scenario: ABI version 1 refused

  • WHEN a worker declares ABI version 1 (the former FlatBuffers wire format)
  • THEN the engine SHALL refuse to load it, naming versions 2 and 3 as the supported ones.

Scenario: ABI version 1 worker rebuilt

  • WHEN a worker built for ABI version 1 is rebuilt against the current headers
  • THEN it SHALL return a protobuf CloudEventBatch instead of a FlatBuffers WorkerResponse.

Scenario: Manifest format

  • WHEN the engine reads the worker's package metadata
  • THEN the nuspec SHALL carry virtufinAbiVersion (integer) and virtufinLibrary (non-empty, matching [A-Za-z0-9_.-]+), and MAY carry virtufinEntryPoint (default Process) and virtufinFreeResult (default FreeResult). A missing or empty library SHALL cause the engine to refuse the worker with a clear error.

Scenario: ABI version mismatch

  • WHEN the worker nuspec declares a virtufinAbiVersion the engine does not support
  • THEN the engine SHALL refuse to load the worker with an error naming the supported versions.

Requirement: Native Worker Identity and Lifecycle

The NativeDllEngine SHALL communicate worker identity to the native worker via process-level environment variables set at LoadCodeAsync time, and SHALL release the loaded library and all engine-owned resources on worker stop.

Scenario: Worker identity env vars

  • WHEN the engine loads a native worker
  • THEN it SHALL set VIRTUFIN_WORKER_ID to the worker GUID and VIRTUFIN_WORKER_TOPIC to the worker's input topic. The worker SHALL read these once at initialization.

Scenario: Correlation id from input

  • WHEN the engine marshals the input CloudEvent for a Process call
  • THEN the protobuf CloudEvent's correlationid attribute SHALL carry the input CloudEvent's correlationid extension attribute if present. Workers SHALL read it from the event, not from env vars.

Scenario: Concurrent Process calls

  • WHEN multiple Process invocations are made against the same loaded library
  • THEN the engine SHALL allow them to proceed concurrently. The native worker SHALL be reentrant; any per-worker mutable state SHALL be protected by the worker's own synchronization primitives.

Scenario: Reload under load

  • WHEN LoadCodeAsync is called while a ProcessAsync is in flight
  • THEN the engine SHALL block the reload until the in-flight call completes. The new code SHALL take effect for subsequent calls only.

Scenario: Cleanup on stop

  • WHEN the engine is disposed or the worker is stopped
  • THEN the engine SHALL call NativeLibrary.Free on the loaded library (except a NativeAOT library, see below), free the GCHandle for the engine pointer, delete the temp file holding the DLL, and restore the prior values of VIRTUFIN_WORKER_ID and VIRTUFIN_WORKER_TOPIC if they were set.

Scenario: NativeAOT library stays loaded

  • WHEN the library being released exports DotNetRuntimeDebugHeader (a NativeAOT build with an embedded .NET runtime)
  • THEN the engine SHALL NOT call NativeLibrary.Free on it, because the embedded runtime keeps threads running in the library's code and unloading it crashes the hosting process; it SHALL still release every other resource.

Requirement: Native Worker Caveats

The NativeDllEngine SHALL document its isolation and timeout characteristics in the engine README and the worker author guide.

Scenario: In-process crash isolation caveat

  • WHEN a native worker raises an unrecoverable error (segmentation fault, illegal instruction, stack overflow from native code)
  • THEN the Runtime process hosting it SHALL terminate because the error cannot be caught in a way that leaves the .NET runtime in a defined state. Workers SHALL be authored under the assumption that a crash terminates the entire Runtime pod. The Python and C# source engines isolate their workers in subprocesses; the DotNetDll engine runs them in-process too, but a managed exception is caught.

Scenario: Unmanaged timeout caveat

  • WHEN the MessageHandlingTimeoutSeconds elapses while a native Process call is in flight
  • THEN the CancellationToken on the managed side SHALL fire and the call SHALL fail on the managed side, but the native call SHALL continue to run until it returns or the process crashes. Workers SHALL be authored to honor the timeout cooperatively (e.g. via periodic checks of a flag set from a watchdog thread, if at all) or SHALL be guaranteed to return within the timeout.

Scenario: ABI version policy

  • WHEN the Runtime ships a new major ABI version
  • THEN the engine SHALL support only that version while Virtufin is in beta: workers are rebuilt in the same release wave, and the engine SHALL refuse a worker declaring any other version.

Requirement: Native Worker Library Naming

The NativeDllEngine SHALL derive the per-platform shared library filename from the manifest's library field by appending an OS-specific prefix and suffix, matching the convention used by NuGet-distributed native packages (runtimes/<rid>/native/). The engine SHALL pick the filename based on RuntimeInformation.IsOSPlatform.

Scenario: Linux filename

  • WHEN the running platform reports OSPlatform.Linux
  • THEN the engine SHALL load lib<library>.so from runtimes/<rid>/native/.

Scenario: Windows filename

  • WHEN the running platform reports OSPlatform.Windows
  • THEN the engine SHALL load <library>.dll from runtimes/<rid>/native/.

Scenario: macOS filename

  • WHEN the running platform reports OSPlatform.OSX
  • THEN the engine SHALL load lib<library>.dylib from runtimes/<rid>/native/.

Scenario: Library name validation

  • WHEN the manifest's library field is empty or contains characters outside [A-Za-z0-9_.-]
  • THEN the engine SHALL refuse the worker and throw a clear error naming the offending value.

Requirement: Worker Tags (Resource Metadata)

The WorkManager SHALL let clients attach arbitrary key/value tags to a worker via CreateWorkerRequest.tags, SetTag, and SetTags, retrievable via GetTag/GetTags/ListWorkers (WorkerInfo.tags). Tags SHALL be resource metadata only — used for filtering (ListWorkersRequest.filters: repeated TagFilter) and identification — and SHALL NOT be injected into the worker's process environment or otherwise exposed to the worker's code.

Scenario: Tags filter ListWorkers

  • WHEN a client calls ListWorkers with one or more TagFilter{key, value} entries
  • THEN only workers whose tags match every filter SHALL be returned

Scenario: Tags are not process-visible

  • WHEN a worker is loaded by any engine
  • THEN its tags SHALL NOT be set as environment variables and SHALL NOT be passed to the worker's ProcessAsync — a worker has no supported way to read its own tags

Scenario: SetTag/SetTags update stored metadata only

  • WHEN a client calls SetTag or SetTags on a running worker
  • THEN the WorkManager SHALL update the persisted tag metadata immediately; no engine reload is required since tags are never applied to the process

Requirement: Worker Creation-Time Config

The WorkManager SHALL let clients set worker-instance defaults via CreateWorkerRequest.config (map<string, string>), retrievable via WorkerInfo.config. Unlike the removed worker environment-variable feature (which mutated process-wide Environment state and let co-loaded workers silently stomp on each other's values), config SHALL be delivered per-call, per-worker-instance: the Runtime SHALL stamp every config entry onto the incoming CloudEvent as an extension attribute before dispatching to the engine, using the same generic extension mechanism already used for correlationid. Config SHALL be immutable after creation — there SHALL be no SetConfig/GetConfig RPCs.

Scenario: Config delivered as CloudEvent extension attributes

  • WHEN a worker created with config: {"url": "https://example.com"} receives a triggering CloudEvent
  • THEN the CloudEvent SHALL carry a url extension attribute set to "https://example.com" by the time it reaches the worker's engine

Scenario: Config does not leak between co-loaded workers

  • WHEN two workers, each created with a different config value for the same key, are loaded in the same Runtime process
  • THEN each worker's incoming CloudEvents SHALL only ever carry that worker's own config value — never the other worker's

Scenario: Config is immutable after creation

  • WHEN a client wants to change a worker's config
  • THEN the WorkManager SHALL NOT provide an RPC to do so; the worker MUST be deleted and recreated with the new config

Scenario: Config survives WorkManager restart

  • WHEN a worker is recovered after a restart (actor reactivation or RecoverWorkers)
  • THEN the recovered worker SHALL retain its original config

Scenario: A config key that can never reach the worker is refused

  • WHEN CreateWorkerRequest.config has a key that is not a CloudEvent attribute name (lower-case letters and digits only, e.g. strategy_id, marketWorld, market.world), is data, is a core attribute that takes no string (time, dataschema), or is an attribute every input already carries (id, source, specversion, type, datacontenttype)
  • THEN CreateWorker SHALL fail with INVALID_ARGUMENT naming the key, and no worker SHALL be created
  • AND every host (the Runtime's WorkManager and the C#, Python and TypeScript local WorkManagers) SHALL apply the same rule, pinned by the conformance vectors' config_validation cases (WorkerEventRules.ValidateConfig)
  • AND for a worker created before hosts validated config, the per-event rule SHALL still drop such an entry (logged) and apply the rest

Requirement: Worker Manager Horizontal Scalability

The WorkManager SHALL scale horizontally by spreading workers across processes, not by sharing one worker's load: every worker subscribed to a topic receives every message on it, wherever it runs (Cross-Instance Coordination), and a worker processes its topic's messages in order, one at a time. Throughput for a high-volume stream SHALL therefore come from splitting it across topics (e.g. one topic per currency pair) and workers, not from adding replicas of one worker.

Scenario: Multiple WM replicas on the same topic

  • WHEN two Runtime pods each run a worker for topic orders
  • THEN each message published to orders SHALL be delivered to both replicas, and both workers SHALL process it
  • AND this SHALL hold for distinct workers and for two deployments of the same package alike: deploying a second copy of a worker duplicates its processing (and its outputs), it does not share load

Scenario: Several workers on one topic

  • WHEN several workers on the same Runtime pod subscribe to the same topic, e.g. several indicator workers on one candle topic
  • THEN every message on the topic SHALL be delivered to each of them, and each SHALL process it with its own config and state
  • AND one worker failing on a message SHALL NOT prevent its delivery to the others
  • AND each worker SHALL receive the topic's messages in publish order

Scenario: Hot-swap within a topic

  • WHEN a worker on orders is replaced by a new worker on orders
  • THEN the replacement SHALL be stop-then-start: stopping the old worker unsubscribes it, and from then on only the new worker receives messages on orders
  • AND while both are subscribed, both SHALL receive every message (there is no "latest worker wins" dispatch)

Scenario: Scaling throughput

  • WHEN operators want to scale throughput for a high-volume stream
  • THEN they SHALL NOT deploy more copies of the same worker on the same topic expecting load sharing -- every copy processes every message
  • AND they SHALL instead split the stream across topics (e.g. per-pair topics) with a worker per topic, which the WorkManager spreads across processes, or speed up the worker itself
  • AND the Pub/Sub Topics spec "Subscription Implementation Pattern" requirement is the authoritative reference for delivery

Scenario: Observability for competing-consumer behavior

  • WHEN an operator sees a worker not receiving messages on a topic
  • THEN they SHALL check that the worker is RUNNING and subscribed to that exact topic, and that its process's subscription is open (messages published before a subscription opens are not delivered: Pub/Sub Topics spec, "New subscription starts at the stream tail")
  • AND there is no competition between replicas to rule out: every subscribed process receives every message
  • AND the sys.workmanager.worker.lifecycle events carry worker_id and topic so operators can correlate

Requirement: Worker Source Authentication

URL-based code sources SHALL support an optional per-request credential, carried on the same request as the code source (CreateWorkerRequest/LoadCodeRequest), never persisted beyond the fetch it authorizes. A credential SHALL be one of: a directly-supplied Basic or Bearer secret, or a named reference resolved server-side. SSRF and host-allowlist validation SHALL apply regardless of whether a credential is present or which kind it is.

Scenario: Directly-supplied credential

  • WHEN a code source URL is fetched with a Basic or Bearer credential attached to the request
  • THEN the fetch SHALL use that credential's Authorization header, and no other

Scenario: Named credential resolution

  • WHEN a code source URL is fetched with a named credential reference
  • THEN the service SHALL resolve the name to a server-held credential and validate the target host is permitted for that named credential before injecting it

Scenario: Unresolvable named credential

  • WHEN a request references a named credential the service does not recognize, or whose target host is not permitted for that name
  • THEN the fetch SHALL fail with a clear authorization error rather than proceeding unauthenticated

Scenario: No credential supplied

  • WHEN a code source URL is fetched with no credential on the request
  • THEN the fetch SHALL proceed without an Authorization header

Scenario: Credentials are not persisted

  • WHEN a code source fetch completes, successfully or not
  • THEN the supplied credential SHALL NOT be logged or retained beyond that fetch

Requirement: Multi-Topic Worker Subscription

A worker MAY subscribe to more than one topic. CreateWorkerRequest SHALL accept a list of topics (minimum one); the WorkManager SHALL open a subscription for each distinct topic in the list and dispatch any triggering CloudEvent on any of them to the worker.

Scenario: Worker created with multiple topics

  • WHEN a client creates a worker with more than one topic in CreateWorkerRequest
  • THEN the WorkManager SHALL subscribe to every listed topic
  • AND a CloudEvent arriving on any of the worker's topics SHALL be dispatched to that worker's ProcessAsync

Scenario: Worker created with one topic

  • WHEN a client creates a worker with exactly one topic
  • THEN behavior SHALL be identical to the single-topic case that predates this requirement — this is a superset capability, not a new required pattern

Scenario: Shared topic across workers unaffected

  • WHEN multiple workers (with possibly-overlapping topic lists) are registered
  • THEN the existing per-topic dispatch rule (all local workers registered for a topic receive a matching message; see the Worker Lifecycle Events and Cross-Instance Coordination requirements) SHALL apply independently per topic, unchanged by a worker having additional topics

Scenario: Worker deletion unsubscribes all its topics

  • WHEN a worker with multiple topics is deleted
  • THEN the WorkManager SHALL unsubscribe/tear down the worker's registration on every one of its topics
  • AND a topic SHALL only have its Dapr subscription fully torn down once no worker (in this or any other topic-sharing registration) still references it

Requirement: Worker Secrets by Reference

The WorkManager SHALL let clients give a worker secrets through CreateWorkerRequest.secrets (map<string, string>, attribute name -> secret key). It SHALL persist and return (WorkerInfo.secrets) only these references. The Runtime SHALL resolve them from the configured Dapr secret store, keep the values in memory only, and stamp each onto every incoming CloudEvent as the extension attribute of that name. Secrets SHALL be immutable after creation, like config.

Scenario: Secret delivered as an extension attribute

  • WHEN a worker created with secrets: {"alpacakey": "alpaca-paper-key"} receives a triggering CloudEvent
  • THEN the CloudEvent SHALL carry an alpacakey extension attribute set to the value of alpaca-paper-key in the secret store by the time it reaches the worker's engine, overriding an alpacakey attribute on the incoming event

Scenario: Only references are stored and listed

  • WHEN a worker with secrets is persisted, listed, or recovered after a restart
  • THEN its record and WorkerInfo SHALL hold only the references, and the recovered worker SHALL resolve them again

Scenario: Unresolvable reference

  • WHEN a reference does not resolve while the worker loads
  • THEN the worker SHALL become LOAD_FAILED with a load-failed lifecycle event naming the reference and not containing any secret value

Scenario: Invalid request

  • WHEN a secret name is not a valid extension-attribute name, is reserved, is also a config key, or a reference does not match [A-Za-z0-9._-]{1,253} (Kubernetes Secret key rules)
  • THEN CreateWorker SHALL fail with INVALID_ARGUMENT

Scenario: Secrets do not reach outputs

  • WHEN a worker returns a CloudEvent that carries one of its secret attributes (for example its own input)
  • THEN the Runtime SHALL drop that attribute before the event is published

Scenario: Rotation

  • WHEN a secret's value changes in the store
  • THEN a loaded worker SHALL receive the new value within the Runtime's secret refresh interval (default 60 s) without being recreated

Scenario: Never logged

  • WHEN the WorkManager or Runtime logs anything about a worker with secrets
  • THEN the log SHALL NOT contain a secret value