Client Testing
Purpose¶
Standardized client test requirements across all 3 repos (api, websocketmanager, workmanager) and all 3 languages (Python, TypeScript, .NET). Every repo SHALL have the same set of client tests in every language, ensuring SDK parity and preventing regressions.
Requirements¶
Requirement: Cross-Language Client Test Parity¶
Every test case listed in the per-service tables below SHALL exist in every language marked ✓ for that row. The tables are the source of truth for which tests are expected where.
common/scripts/check-client-test-parity.sh parses those tables directly (via awk, keyed on the ## <service> Client Test Cases headings and the | <ID> | rows) and compares them against ID comments found in each repo's test tree. It previously duplicated the expected IDs in a hand-maintained bash case statement, which silently drifted from the spec and hid a real coverage gap — hence parsing rather than restating.
Scenario: A test is added in one language¶
- WHEN a hand-written client test is added to any of the three repos
- THEN a corresponding row SHALL be added to the per-service table in this spec
- AND the test SHALL be added to every other language the row marks
✓
Scenario: A parity gap exists¶
- WHEN the checker finds an ID expected by the tables but absent from a language's test tree
- THEN it SHALL exit non-zero and name the missing IDs
- AND the scheduled
test-parity-commonworkflow (weekly, plusworkflow_dispatch) SHALL report the failure — note this is a periodic check, not a per-merge gate, so a gap can land and be caught later
Scenario: The tables and the checker disagree¶
- WHEN the checker's expectations diverge from these tables
- THEN the tables win, and the checker SHALL be corrected to parse them rather than restate them
Requirement: Test-Case ID Comments¶
Every test case ID in the per-service tables SHALL appear as a comment in each language's implementation of that test. This is what makes automated matching possible: the checker scans for the IDs, not for test names.
Scenario: Marking a test with its ID¶
- WHEN a hand-written test implements a spec test case
- THEN the ID SHALL appear as a comment immediately above the test, in that language's comment syntax (
# W18,// W18) - AND it SHALL be present in every language implementing that case, not only the first
Requirement: Test Naming Conventions¶
Test names SHALL follow the per-language convention below, so that a human reading a gap report can locate the test the ID refers to.
| Language | Pattern | Example |
|---|---|---|
| Python | test_<snake_case_name> |
test_connect_delegates_to_stub |
| TypeScript | it("<human readable name>") |
it("connect delegates to gRPC client") |
| .NET | CamelCaseName |
ConnectDelegatesToStub |
Scenario: Naming a new test¶
- WHEN a test is added in any language
- THEN its name SHALL match that language's pattern above
Requirement: Separate ID Namespaces for Generated and Hand-Written Tests¶
Generated structural tests SHALL use a G-prefixed ID namespace distinct from the hand-written spec IDs, so the two cannot collide and the checker validates only what the tables actually track.
| Service | Hand-written prefix | Generated prefix |
|---|---|---|
| virtufin-api | A |
GA |
| virtufin-websocketmanager | W |
GW |
| virtufin-workmanager | M |
GM |
Scenario: Generated tests are excluded from parity expectations¶
- WHEN the checker scans a repo containing generated test files
- THEN their
GA*/GW*/GM*IDs SHALL NOT match the hand-written\b(A|W|M)[0-9]+\bpattern - AND only hand-written IDs SHALL be validated against the tables
Scenario: Regenerating structural tests¶
- WHEN a
.protofile changes and the generator is re-run - THEN generated test files SHALL be overwritten wholesale and SHALL NOT be hand-edited
- AND hand-written tests SHALL be unaffected, because they live in separate files
Workflow¶
How test parity is maintained¶
proto file ──► generator ──► checker ──► CI gate
- Generator (
common/scripts/generate-client-tests.py): Reads a.protofile and emits test files in Python, TypeScript, and .NET with test-case ID comments. Run after proto changes. - Checker (
common/scripts/check-client-test-parity.sh): Scans all 3 repos for ID comments and reports gaps against this spec. Run locally or in CI. - CI (
common/.github/workflows/test-parity-common.yaml): Weekly + on-demand parity gate.
Running the generator¶
# The repos are NOT a monorepo, so the script is fetched
# directly from the virtufin-common Gitea repo at runtime.
cd virtufin-<service>
python3 <(curl -L https://git.haenerconsulting.com/virtufin/common/raw/branch/master/scripts/generate-client-tests.py) \
src/Virtufin.<Service>.Protos/proto/<service>.proto \
--prefix <X>
The generator is idempotent. Run it whenever the .proto file changes.
Running the checker¶
cd virtufin # directory containing all service repos
VIRTUFIN_ROOT=. bash common/scripts/check-client-test-parity.sh
Exits 0 if all IDs are present, 1 if gaps exist.
Generated vs hand-written tests¶
- Generated (
tests/python/test_generated.py,tests/typescript/generated.test.ts,tests/...Client.Tests/GeneratedTests.cs): Structural coverage from.proto. Do not edit — regenerated on proto changes. - Hand-written (
tests/python/test_*.py,tests/typescript/*.test.ts,tests/...Client.Tests/*Tests.cs): Business logic, patterns, integration. ID-commented for parity.
Both carry test-case IDs. Generated tests provide baseline coverage; hand-written extend it.
Test ID namespaces¶
The spec test IDs (e.g. A11, W18, M30) are only for hand-written tests. They correspond to the rows in the per-service tables below. The generator emits its own test IDs in a separate namespace so that generated structural/stub tests don't collide with the spec IDs:
| Service | Hand-written prefix | Generated prefix |
|---|---|---|
| virtufin-api | A |
GA |
| virtufin-websocketmanager | W |
GW |
| virtufin-workmanager | M |
GM |
The generator defaults --gen-prefix to G + --prefix (so --prefix A → --gen-prefix GA). Override with --gen-prefix <X> if needed.
The parity checker (check-client-test-parity.sh) only validates the hand-written IDs against the spec — generated test files are scanned but their GA* / GW* / GM* IDs don't match the \b(A|W|M)[0-9]+\b pattern, so they're correctly excluded.
virtufin-api Client Test Cases¶
The previously specified A1-A10 and A12-A14 (constructor/package/method/integration tests) are deferred. They described an aspirational set that was never implemented in hand-written code; the generator now provides structural coverage via the GA-prefix test IDs in test_generated.py / generated.test.ts / GeneratedTests.cs (see "Test ID namespaces" above). Only A11 is currently hand-written.
| # | Category | Test Case | Python | TypeScript | .NET |
|---|---|---|---|---|---|
| A11 | Integration | Connectivity check | ✓ | -- | -- |
virtufin-websocketmanager Client Test Cases¶
| # | Category | Test Case | Python | TypeScript | .NET |
|---|---|---|---|---|---|
| W1 | Constructor | Default host/port | ✓ | ✓ | ✓ |
| W2 | Constructor | Custom host/port | ✓ | -- | -- |
| W3 | Constructor | All expected methods present | ✓ | ✓ | ✓ |
| W4 | Package | Proto types importable | ✓ | ✓ | ✓ |
| W5 | Package | gRPC client stub accessible | ✓ | ✓ | ✓ |
| W6 | Package | Transport dependency available | -- | ✓ | ✓ |
| W7 | Package | ConnectRequest schema | -- | ✓ | ✓ |
| W8 | Package | ConnectResponse schema | -- | ✓ | ✓ |
| W9 | Package | ListRequest schema | -- | ✓ | ✓ |
| W10 | Package | ListResponse schema | -- | ✓ | ✓ |
| W11 | Package | DisconnectRequest schema | -- | ✓ | ✓ |
| W12 | Package | SendRequest schema | -- | ✓ | ✓ |
| W13 | Package | SendRequest fields (id, message, timeout) | -- | -- | ✓ |
| W14 | Package | SendRawRequest schema | -- | ✓ | ✓ |
| W15 | Package | StartPublishRequest schema | -- | ✓ | ✓ |
| W16 | Package | StopPublishRequest schema | -- | ✓ | ✓ |
| W17 | Package | WebSocketConnection schema | -- | ✓ | ✓ |
| W18 | Method | Connect delegates to stub | ✓ | ✓ | -- |
| W19 | Method | List delegates to stub | ✓ | ✓ | -- |
| W20 | Method | Disconnect delegates to stub | ✓ | ✓ | -- |
| W21 | Method | Send delegates to stub | ✓ | ✓ | -- |
| W22 | Method | Send with default timeout | ✓ | -- | -- |
| W23 | Method | SendRaw delegates to stub | ✓ | ✓ | -- |
| W24 | Method | StartPublish delegates to stub | ✓ | ✓ | -- |
| W25 | Method | StopPublish delegates to stub | ✓ | ✓ | -- |
| W26 | Method | Close releases channel | ✓ | -- | -- |
| W27 | Error | Connect error propagates | ✓ | ✓ | ✓ |
| W28 | Error | Send error propagates | ✓ | ✓ | -- |
| W29 | Error | List error propagates | ✓ | ✓ | -- |
| W30 | Lifecycle | GrpcChannel construction | -- | -- | ✓ |
| W31 | Lifecycle | Transport creates successfully | -- | ✓ | -- |
| W32 | Lifecycle | Package exports index entry | -- | ✓ | -- |
| W33 | Method | SetTag delegates to stub | ✓ | ✓ | -- |
| W34 | Method | GetTag delegates to stub | ✓ | ✓ | -- |
| W35 | Method | SetTags delegates to stub | ✓ | ✓ | -- |
| W36 | Method | GetTags delegates to stub | ✓ | ✓ | -- |
virtufin-workmanager Client Test Cases¶
| # | Category | Test Case | Python | TypeScript | .NET |
|---|---|---|---|---|---|
| M1 | Constructor | Default host/port | ✓ | ✓ | ✓ |
| M2 | Constructor | Custom host/port | ✓ | -- | -- |
| M3 | Constructor | All expected methods present | ✓ | ✓ | ✓ |
| M4 | Package | Proto types importable | ✓ | ✓ | ✓ |
| M5 | Package | Proto enums accessible | -- | ✓ | ✓ |
| M6 | Package | gRPC client stub accessible | ✓ | ✓ | ✓ |
| M7 | Package | Transport dependency available | -- | ✓ | ✓ |
| M8 | Package | CreateWorkerRequest schema | -- | ✓ | ✓ |
| M9 | Package | ListWorkersRequest schema | -- | ✓ | ✓ |
| M10 | Package | DeleteWorkerRequest schema | -- | ✓ | ✓ |
| M11 | Package | StartWorkerRequest schema | -- | ✓ | ✓ |
| M12 | Package | StopWorkerRequest schema | -- | ✓ | ✓ |
| M13 | Package | RecoverWorkersRequest schema | -- | ✓ | ✓ |
| M14 | Package | LoadCodeRequest has an Id property | -- | ✓ | ✓ |
| M15 | Package | LoadCodeRequest has a CodeSource property | -- | ✓ | ✓ |
| M16 | Package | WorkerInfo schema | -- | ✓ | ✓ |
| M17 | Package | CodeSource schema | -- | ✓ | ✓ |
| M18 | Package | WorkerStatus enum | -- | ✓ | ✓ |
| M19 | Method | CreateWorker with URL | ✓ | ✓ | -- |
| M20 | Method | CreateWorker with content | ✓ | -- | -- |
| M21 | Method | CreateWorker without connect raises | ✓ | -- | -- |
| M22 | Method | ListWorkers delegates to stub | ✓ | ✓ | -- |
| M23 | Method | DeleteWorker delegates to stub | ✓ | ✓ | -- |
| M24 | Method | StartWorker delegates to stub | ✓ | ✓ | -- |
| M25 | Method | StopWorker delegates to stub | ✓ | ✓ | -- |
| M26 | Method | RecoverWorkers delegates to stub | ✓ | ✓ | -- |
| M27 | Method | GetWorkerHistory delegates to stub | ✓ | ✓ | -- |
| M28 | Obsolete | ~~LoadCodeFromContent delegates to stub~~ — LoadCode consolidates both variants behind one RPC (see M3); no separate LoadCodeFromContent stub method exists to delegate to. |
-- | -- | -- |
| M29 | Obsolete | ~~LoadCodeFromUrl delegates to stub~~ — same consolidation as M28. | -- | -- | -- |
| M30 | Error | CreateWorker error propagates | ✓ | ✓ | ✓ |
| M31 | Error | ListWorkers error propagates | -- | ✓ | -- |
| M32 | Error | DeleteWorker error propagates | -- | ✓ | -- |
| M33 | Lifecycle | GrpcChannel construction | -- | -- | ✓ |
| M34 | Lifecycle | Transport creates successfully | -- | ✓ | -- |
| M35 | Lifecycle | Package exports index entry | -- | ✓ | -- |
| M36 | Obsolete | ~~SetEnvironmentVariable delegates to stub~~ — the custom worker environment-variable RPC was removed entirely (process-wide state, unsafe across co-loaded workers); config now flows through the CloudEvent payload instead. | -- | -- | -- |
| M37 | Obsolete | ~~GetEnvironmentVariable delegates to stub~~ — same removal as M36. | -- | -- | -- |
| M38 | Obsolete | ~~SetEnvironmentVariables delegates to stub~~ — same removal as M36. | -- | -- | -- |
| M39 | Obsolete | ~~GetEnvironmentVariables delegates to stub~~ — same removal as M36. | -- | -- | -- |
Rules for Adding Tests¶
Moved to Requirements above, so the rules and the tables they govern cannot drift apart. See Cross-Language Client Test Parity, Test-Case ID Comments, Test Naming Conventions, and Separate ID Namespaces for Generated and Hand-Written Tests.