Testing

The project uses focused regression tests in tests/async/, plus schema/migration checks at tests/ root. The root Makefile provides a single gate while retaining fast commands for focused work.

Layout

tests/
├── async/                       # focused regression + source-contract tests
│   ├── test_*.py                # plain python3 regressions; no framework
│   ├── run_*.sh                 # special-purpose and legacy thin wrappers
│   └── run_*_mysql.sh           # MySQL-backed schema-contract tests (need a live DB)
├── run_regression_tests.py      # discovery, bounded parallelism, deadlines, failure summary
├── compare_bootstrap_mud_schema.sh   # diff live schema vs bootstrap baseline
├── test_migration_replay_safety.sh   # migration re-run safety
└── test_run_migration_persistence_schema.sh

Test styles

Behavioral tests compile the production source into a native harness, or boot the server in a journey, and check what the code does. A new test is behavioral whenever a harness or a journey can reach the code: prefer more assertions in an existing harness for the same source, which adds no compile.

Source-contract tests read the sources as text. One fails on a harmless rename or a reshaped block, and passes when the behavior is wrong but the text still matches, so a text check stays only for what text alone can show: a call site that must not come back, or an order that must hold. Before keeping one, break the code it guards; when another test then fails, the text check is redundant. Example (tests/async/test_sql_pool_shutdown.py): slices out sql_pool_acquire/sql_pool_shutdown from src/sql/sql_pool.c and asserts the closing-pool checks are present.

Schema tests run migrations and repositories against a disposable MariaDB or MySQL. Every one is a line of tests/run_db_tests.sh, so make test-db runs it: a test no gate runs stops passing unnoticed. A leg that needs the full schema runs under tests/async/with_disposable_mariadb.sh and sources tests/async/disposable_schema.sh; none reads the checkout's .env.

Counts and hashes. Check the property a count stands for (every entry has an owner and a retention, names are unique, one array per class), not the count, unless the count itself is the contract, such as the sealed migrations' ids or the shops' durable ids. A count that grows with content breaks the test when nothing is wrong. Hashes stay where they guard a generated file against hand edits.

Harness stubs. A native harness compiles tests/async/harness_stubs.cpp (_paths.HARNESS_STUBS from Python) for the server functions it does not link. Its definitions are weak, so the real source or a harness's own definition replaces them at link time; write a local stub only where it must behave differently. A harness built on its own types, without the server's headers, keeps its stubs.

Running

# Complete gate: builds all maintained binaries and tools,
# generates world data, then runs Python and native regression tests.
make test-all

# Regression tests without rebuilding the server or area editor:
make test

# Limit concurrency, filter by filename, or inspect discovery:
make test TEST_JOBS=1
make test-python TEST_MATCH=wear
make test-list

# Single test (preferred while iterating):
python3 tests/async/test_wear_all_regression.py
# or via its wrapper:
tests/async/run_sql_pool_shutdown.sh

# Isolated Docker database legs and MariaDB journeys (Docker is an optional prerequisite):
make test-db

# Full historical legacy upgrade, replay, bootstrap equivalence, and compatibility:
tests/async/run_legacy_migration_mysql.sh

# Runtime schema compatibility on the supported variants (MySQL 8.0, MariaDB 10.11 and
# 11.8), and the migration runner on all three:
tests/async/run_runtime_compatibility_mysql.sh
RUNTIME_DB_IMAGE=mariadb:10.11 tests/async/run_runtime_compatibility_mysql.sh
RUNTIME_DB_IMAGE=mariadb:11.8 tests/async/run_runtime_compatibility_mysql.sh
python3 tests/async/run_migration_runner_engines.py

TEST_JOBS=0 is the default and selects up to eight workers based on available CPUs. Test output is buffered per process so parallel failures remain readable, and a failing test's output is printed as soon as it fails. A test still running 900 s after it started, or a journey 1800 s after, is ended with everything it started and reported as TIMEOUT; a test ended by a signal is reported by the signal's name. Both are failures, the end of the run lists every failure with its kind, the summary counts timeouts and signals apart, and every 60 s a line names the tests still running. However a test ends, whatever it started and left running ends with it. Stopping the runner, by Ctrl-C or a SIGTERM, ends the tests it is running with everything they started, and starts no others. The runner executes every discovered test_*.py in a separate process and returns nonzero if any test fails. The journeys that boot a real server, and the large sanitizer harness, start together once the parallel phase is done. They wait on game time, not on the CPU, and they share one server build.

make test-all deliberately excludes Docker and externally provisioned database checks. make test-db builds the server and generates the world first, then runs the isolated database legs and the MariaDB journeys listed in tests/run_db_tests.sh, 12 at a time (TEST_DB_JOBS=N changes that). Every test creates and destroys its own MySQL or MariaDB container, prints PASS or FAIL with its seconds, and writes its log under bin/tests/db/. A new leg or journey is one more line in that list; tests/async/with_disposable_mariadb.sh CMD... gives a journey its own MariaDB. The three scripts at the root of tests/ require explicitly named disposable or read-only databases and are manual migration-verification tools. Never point them at production.

The MariaDB playtime journey runs in make test-db. To run it alone after a MariaDB server build, use tests/async/with_disposable_mariadb.sh python3 tests/async/test_mysql_playtime_journey.py --server bin/server/dms_new. The journey creates and removes its own schema and invokes test_playtime_mysql_repository.py with that schema's environment. Both scripts are excluded from argument-free discovery; the database-independent playtime capture, checkpoint, legacy SQL, and flat-file tests remain in make test-all.

The game loop budget journey (test_mysql_game_loop_budget_journey.py, in make test-db) boots the full world on MariaDB, has scripted players camp and rent while the first hourly event runs, and fails if a pulse runs past its 250 ms. It prints what the loop measured (tick, event and command times, deferred events, the costliest callbacks), so it is also the load to measure a build with: tests/async/with_disposable_mariadb.sh python3 tests/async/test_mysql_game_loop_budget_journey.py --server <dms_new> --players 30 --hours 2.

The world capture journey (test_mysql_world_capture_journey.py, in make test-db) boots the full world on MariaDB and Redis with world recovery on and 8,000 more objects on the ground, and has scripted players play while the first capture runs. It fails if the generation is not above the old 64 MiB ceiling, if its capture takes a third of its 300 s, or if a pulse runs past its 250 ms; it then kills the server, and fails unless the next boot restores the generation. It prints the generation's size, the capture's time and what the restore brought back; --players N scales the load. Its first boot is on an empty database, so it is also the boot test for SQL queued before the writer starts: it fails unless every artifact the boot created has its artifacts and artifact_domain_state rows, each outpost's stored hit points are set and the frag list cache is filled. Both boots must cache the six artifact lists (the restored one reads them after the restore), and neither may log a sql job line (not queued, or failed at boot). run_world_recovery_journey.py, also in make test-db, covers the switch on the mini world: off, the alert of an outage, the writer coming back, a crash's restore and the defaults. run_world_restart_journey.py <server> <scenario> covers what a restart leaves, one scenario a run and each its own line in make test-db: crash (the lease is renewed, the boot after a crash consumes the generation it restored and publishes its own, and a second crash restores that one), copyover (the new image holds the lease at once), restart (a clean stop takes a last capture and the boot restores it), taken (an object a character took and saved after the capture is not restored to the floor, and one it saved and dropped before the capture is; the boot reaps the dropped object's ownership record with the dissolved starter kit's, so the login after it counts no missing_payload_rows and the saves wrote no unowned_object line, both without DURIS_PERSISTENCE_TRACE), handover (a character takes the banana after the capture and is checkpointed, gives it to a second one, who saves, drops it and saves, and the server crashes before the giver's next checkpoint: the boot keeps the record, which is what makes the giver's older copy load as stale, so the floor copy comes back and only one character holds the banana), midcapture (an object that leaves a floor container while a capture runs, and whose drop is journaled, is restored once) and slowread (a boot whose reads of Redis miss their deadline asks again and restores).

MySQL fixtures that reuse a table within a statement must account for MySQL's Can't reopen table restriction on connection-local temporary tables; MariaDB may accept the same fixture. Use isolated ordinary tables with explicit cleanup when needed to execute the actual production query shape. Do not weaken queries or narrow the backend matrix merely to make a temporary-table fixture pass.

Shared journey server artifacts

The Python runner defaults DURIS_REGRESSION_BUILD_CACHE to bin/regression-artifacts. Account recovery, flat-file boot preflight, Chaos kit, combat and full-world boot acquire one compatible flat-file server from tests/async/server_build_artifacts.py. Each journey retains its own temporary authority, journals, logs, listeners and process cleanup. Executables are shared read-only; runtime state is never stored in the artifact directory. The item-prompt ASan/UBSan harness remains a separate build with its existing flags and the server build's 600-second ceiling. That one server build uses every core, and the journeys start together when it is done.

The artifact key covers all files under src/ (including untracked files), test headers, the helper contract, the flat-file backend, and the inherited build environment, including profile, feature flags and Make overrides. It also hashes the effective compiler, Make, assembler, linker, compiler internals, installed headers and libraries. Standard GNU compiler search directories and explicit CPATH, CPLUS_INCLUDE_PATH, C_INCLUDE_PATH and LIBRARY_PATH directories, plus explicit include/library paths in Make's effective flags, are fingerprinted by content. Keep external build inputs in these declared paths; use DURIS_REGRESSION_BUILD_CACHE=off for toolchains with hidden inputs or wrappers that do not support GNU compiler discovery. Runtime-only full-world diagnostics and shell working-directory variables do not invalidate a build.

On every acquisition, the helper verifies executable and build-log SHA-256 hashes and rechecks the client-free compilation assertions. Missing or malformed metadata, altered binaries/logs and changed inputs trigger a fresh build. Inputs are checked again after compilation; a build spanning an input change is rejected. A per-key lock prevents duplicate concurrent builds, and atomic manifests publish only successful artifacts. Replacement builds use new directories so an existing process keeps its executable. Old generations remain until the cache is removed when no journeys are using it; make clean-all also removes bin/.

Direct test invocations build privately unless the cache variable is set. To measure the complete Python gate on the same host and worker count:

DURIS_REGRESSION_BUILD_CACHE=off python3 tests/run_regression_tests.py --jobs 4
DURIS_REGRESSION_BUILD_CACHE="$PWD/bin/benchmark-fresh" python3 tests/run_regression_tests.py --jobs 4
DURIS_REGRESSION_BUILD_CACHE="$PWD/bin/benchmark-fresh" python3 tests/run_regression_tests.py --jobs 4

Choose an unused directory for the clean-cache run. Per-journey output reports server build time, artifact validation time, and remaining journey/fixture time. The last category includes inspector compilation and process startup; it is not solely gameplay time. Cache keys hash environment values without storing their plaintext in manifests. Never check cache artifacts into Git.

Before a merge

Run the gate, one command after the other, on a quiet machine. The performance gate (test_telemetry_capacity_272.py) and the latency journeys are CPU-sensitive, and a busy machine fails their time budgets.

./scripts/format.sh --all --check
make test-all -j16 TEST_JOBS=16
make test-db

Runtime depends on the host, selected tests, and build cache. Run the following additional checks when the change touches what they check:

Do not replay the flatfile-build, quality or build.yml workflow jobs step by step. Their tests are the ones make test-all just ran on the same host.

When a change fixes a defect and adds or changes its regression test, run scripts/check_tests_catch.sh BASE HEAD (BASE is the commit before the fix). For each tests/async/test_*.py the range adds or changes, it runs the test in two fresh worktrees under bin/analysis/, each after make world: HEAD's, where it must pass, and BASE's sources with HEAD's whole tests/ tree. It reports "catches" when the test passes on HEAD and fails on BASE, "does not catch" when it passes on both, and "cannot judge" when it fails on HEAD's tree too, for something a fresh worktree lacks. A test that passes on the code before its fix proves nothing; one reshaped by a refactor may pass and still be right. It reports and does not block: the reviewer reads it, and the landing commit message records it. It does nothing when the range changes no src/. A journey builds a server in each worktree first, about three minutes. The last line of each failure shows how the test failed; a test that extracts a function the old code does not have fails without saying anything about behaviour.

Each run of tests/run_regression_tests.py (so each make test and make test-all) writes bin/test-history/<UTC time>-<short sha>-<pid>.json: the commit, whether the tree was dirty (any changed or untracked file), when it finished, its --match filter and worker count, and each test's path, status and seconds. python3 scripts/test_history.py reads them and lists the tests that both passed and failed on one clean commit, those whose last time rose more than half over their median of the ten runs before that were made like the last (same --match and workers; among tests taking a second or more), and the twenty slowest of the last run. bin/ is ignored, so the history is the checkout's own; a gate run in another worktree keeps its files there.

Coverage

make coverage (or scripts/coverage.sh [--db] [COMMIT]) measures which lines of src/ the suite runs. Nothing else runs it, and its percentage is never a target: it shows what no test reaches. It works in a detached worktree of the commit (HEAD by default) under bin/analysis/coverage-<sha>/, so the checkout and any running gate are untouched, with scripts/coverage/g++ first on PATH. That wrapper gives every compile, the server's, the journeys' flat-file builds and the harnesses', --coverage and its own directory of notes and counts under bin/coverage/<sha>/data/, so a harness built in a temporary directory leaves its counts behind. It runs make -k test-all, and make test-db too with --db (COVERAGE_ARGS=--db make coverage), then writes bin/coverage/<sha>/index.html, summary.txt (per file), directories.txt (per directory) and failed.txt (the tests that failed under instrumentation: their counts are missing, the report is not).

It cannot see a process ended with SIGKILL (the crash probes write no counts; SIGTERM shuts the server down in order, so most journeys count), a test that patches a copy of a server source and compiles the copy (only src/ is reported), or a test that calls c++ or clang++ instead of g++. The wrapper drops -Werror (at -O2 the instrumentation provokes false warnings inside libstdc++), so a test that needs a compile to fail on a warning fails under coverage and is listed in failed.txt.

Mutation testing

python3 scripts/mutate.py [--jobs N] [--journeys] SRC_FILE... measures whether the tests that name a source notice when it is wrong. It writes each mutant (a relational operator flipped, ==/!= or &&/|| swapped, a ! dropped, a returned true, false, 0 or 1 replaced; never in a comment, string or preprocessor line) into a worktree of HEAD under bin/analysis/mutate/, and runs the tests there that name the file, the quickest first (by bin/test-history/), until one fails. It refuses a checkout with uncommitted changes under src/ or tests/, so commit a new test before scoring with it. Each file's tests run once unmutated first; one that fails there is left out of the score and listed. The report, a score per file and every surviving mutant with its line, is printed and kept in bin/analysis/mutate-report.txt. A survivor is a missing test, dead code, or a mutant no caller can tell apart (an equivalent mutant: say why in the commit that leaves it).

scripts/mutate/g++, first on the tests' PATH, splits each harness's one-command build into one ccache compile per source and a link, so a mutant rebuilds only its own file; without ccache it compiles uncached, and mutate.py says so at the start. The journeys run only with --journeys; they name these files to build a helper and take minutes. It never runs in a gate, and its score counts only the tests that name the file.

Fuzzing

Each tests/fuzz/<target>.cpp feeds LLVMFuzzerTestOneInput to code that reads outside input: websocket (the upgrade request and frames), gmcp (a client's Core.Hello and Client.Info) and player_snapshot (save decoding, with an encode-and-decode round trip). Its // fuzz-sources: and // fuzz-libs: lines say what it links, and // fuzz-max-len: its largest input, a little past the sizes its code checks; the shared harness stubs fill in the rest.

make fuzz FUZZ_TARGET=websocket FUZZ_SECONDS=3600 (or scripts/fuzz.sh TARGET [SECONDS]) builds the target with clang's libFuzzer and the address and undefined-behaviour sanitizers under bin/fuzz/<target>/ and fuzzes it, starting from tests/fuzz/corpus/<target>/. It writes what it finds to bin/fuzz/<target>/corpus/ and a crash to bin/fuzz/<target>/crash-*. Copy a crash, and inputs worth keeping, into the committed corpus (bin/fuzz/<target>/fuzzer -merge=1 tests/fuzz/corpus/<target> DIR... keeps the ones that add coverage). Long runs are by hand; nothing schedules them.

tests/async/test_fuzz_corpus.py, in every make test, builds each target with g++, the same sanitizers and a main that feeds it every committed corpus file once, so a fixed crash stays fixed.

Full-world save diagnostics

test_flatfile_full_world_boot.py supports opt-in synthetic failure/recovery experiments without using the configured database:

Normal runs keep their original deadlines and successful-fixture cleanup. Classify failures by boot, manual save, camp, recovery, or shutdown; distinguish an observation timeout from a rejected save. Retain the revision/worker timeline before diagnosing storage or scheduler latency. A passing rerun does not explain a failure whose timeline was discarded.

The separate combat journey reconnects after creation to hydrate account-bank revision and disables optional boons. It proves that stated fixture scope; it does not prove first-session currency or boon-enabled death/reward behavior.

Validation matrix

No single command proves release readiness. Use the narrowest applicable row while iterating, then run every row required by the session or release gate.

Load, recovery, and fault contracts

These focused tests use source inspection, compiled local harnesses, or synthetic state. They do not connect to the configured database:

python3 tests/async/test_player_load_pipeline.py
python3 tests/async/test_player_load_items.py
python3 tests/async/test_player_load_pets.py
python3 tests/async/test_player_save_pipeline.py
python3 tests/async/test_critical_command_coordinator.py
python3 tests/async/test_critical_transaction_contract.py
python3 tests/async/test_world_recovery_pipeline.py
python3 tests/async/test_maintenance_scheduler.py
python3 tests/async/test_worker_fault_injection.py
python3 tests/async/test_phase01_recovery_gate.py
python3 tests/async/test_phase02_capacity_and_crash_gate.py

The Phase 01 and Phase 02 gates include 25/50/100/200 logical-client waves and bounded codec/crash coverage. They are precursor evidence only. They do not perform the eight representative workload profiles, 30-minute 200-player holds, production-clone query measurements, or complete fault/reconciliation matrix the 200-player gate requires.

Account recovery contracts

Password reset by email has a core harness (injected mail sender and clock, no network), a wire-level SMTP harness against a fake relay on 127.0.0.1, a source-contract pin set, and a telnet journey that boots the client-free server with a fake relay. None of them needs a real relay or a database; the compiled harnesses need g++ plus the libcurl and OpenSSL development headers from the dependency metapackage:

python3 tests/async/test_account_recovery.py
python3 tests/async/test_account_recovery_smtp_live.py
python3 tests/async/test_account_recovery_contract.py
python3 tests/async/test_account_recovery_journey.py

The harnesses never print a reset code or an address; a captured code reaches the driver only through a temporary file. TLS/STARTTLS and SMTP AUTH are not exercised (the fake relay is plaintext on loopback, which the configuration rules permit), so those fixtures do not qualify a real relay.

Lifecycle and privacy contracts

Run the source/synthetic controls first:

python3 scripts/validate_data_lifecycle.py --json
python3 tests/async/test_data_lifecycle_manifest.py
python3 tests/async/test_season_reset_manifest.py
python3 tests/async/test_lifecycle_archive_execution.py
python3 tests/async/test_personal_data_export.py
python3 tests/async/test_account_erasure.py
python3 tests/async/test_immutable_migration_runner.py
python3 tests/async/test_runtime_boot_compatibility.py

Then run schema behavior only through the self-contained disposable wrappers:

tests/async/run_lifecycle_archive_schema_mysql.sh
tests/async/run_personal_data_export_schema_mysql.sh
tests/async/run_account_erasure_schema_mysql.sh
tests/async/run_legacy_migration_mysql.sh
tests/async/run_immutable_migration_ledger_mysql.sh
tests/async/run_runtime_compatibility_mysql.sh
RUNTIME_DB_IMAGE=mariadb:10.11 tests/async/run_runtime_compatibility_mysql.sh
RUNTIME_DB_IMAGE=mariadb:11.8 tests/async/run_runtime_compatibility_mysql.sh
python3 tests/async/run_migration_runner_engines.py

Each wrapper must create and destroy its own isolated container. Never replace its target with .env values or use a configured, shared, restored, or production database. The checked-in lifecycle policy keeps canonical archive, export, and erasure mutation disabled; passing tests prove the guard and synthetic contract, not policy approval.

200-player readiness

Neither make test-all, make test-db, the Phase 01/02 logical-client gates, nor a successful server boot is 200-player readiness evidence. Only the complete gate in PHASE03_READINESS.md is: eight profiles at 25/50/100/200 clients with 30-minute 200-client holds, the fault matrix, reconciliation after every run, and privacy/restore validation on isolated representative data. That document holds the procedure and the gate's current state. UNQUALIFIED is a safe refusal, not a failed workload, and QUALIFIED is not readiness evidence either.

Conventions for new tests

• One concern per file; name it after the feature/regression (test_<feature>.py). The root runner discovers it automatically. Add a run_<feature>.sh wrapper only when the test needs special environment setup or is useful as a standalone workflow.
• Keep them fast and deterministic, and behavioral (see "Test styles"); a text check only for what text alone can show.
• When you change behavior, add or update the focused regression test next to it — this is a stated repo convention (see AGENTS.md).
• Schema-related changes should extend or add a _schema_mysql variant, listed in tests/run_db_tests.sh, so the contract is checked against a real database.

What exists today (samples)

The four untrusted-input suites are behavioral rather than contract-style: they exercise the live decoders that handle network and player-visible text — UTF-8 widths and malformed/overlong/surrogate encodings, ANSI colour parsing and gradients, JSON and GMCP escaping, and RFC 1091 terminal-type negotiation including MTTS capability parsing. That is the shape to copy when the code under test is a parser: every case is a real call, and every defect found while writing them (unbounded continuation runs, invalid UTF-8 re-encoded as U+FFFD and accepted as a map glyph, MTTS 4JUNK silently enabling capabilities, gradient application dereferencing end() on an empty string) stayed as a regression.

test_compiler_warning_profile.py is what keeps the -Werror guarantee in BUILDING.md real: it fails if any of the six formerly-excepted categories is suppressed again, by flag, by a reintroduced exception variable, or by pragma.