Testing
Test projects
| Project | Covers |
|---|---|
NetPrints.Core.Tests | The model, serialization, translation and the repository rules (SourceHygieneTests, golden fixtures). |
NetPrints.Analyzers.Tests | Every Roslyn analyzer, code fix and generator NetPrints owns (ADR-0024): the analyzer assemblies load into a CompilationWithAnalyzers host and report their supported diagnostics. |
NetPrints.Catalog.Tests | The type catalog and its annotations. |
NetPrints.Cli.Tests | The netprints command line, in process (CliTestHost); some tests run the real tool or the real project system as child processes. |
NetPrints.Editor.Tests | Editor view models and hosting services, over fakes (no window), and the checks of the editor guides (below). |
NetPrints.Editor.UITests | The editor's views, headless (Avalonia headless platform), through page objects and AutomationIds. |
NetPrints.Desktop.E2ETests | The real desktop editor on a private X11 display (Linux only). |
NetPrints.Testing, NetPrints.Testing.Ui | Shared fixtures, the driver abstraction, page objects and the Screenplay layer; not tests themselves. |
Running them
Build quietly, then run without building again (the solution selects the Microsoft.Testing.Platform runner):
dotnet build -v q -tl:off --nologo
dotnet test --no-build --no-progress --no-ansi
Run one class with the test assembly itself (--treenode-filter is not accepted):
tests/NetPrints.Editor.Tests/bin/Debug/net10.0/NetPrints.Editor.Tests -class '*RunStateTrackerTests'
The full suite is what CI's test job runs, and the format check is part of the gate:
dotnet test --solution NetPrints.slnx -c Release --no-build --no-progress --no-ansi -- --ignore-exit-code 8
dotnet format NetPrints.slnx --verify-no-changes
Never run the whole NetPrints.Editor.UITests project under Xvfb (it runs out of memory); the headless tests need no display.
Desktop E2E tests
They need Linux with Xvfb, openbox, xdotool, ImageMagick and GTK 3, and run only with NETPRINTS_E2E=1; without it the scenarios skip and only the always-on tests (failure capture, entry point, worker pool) run; the exit code is 0.
Each test rents a worker from a pool (a private Xvfb display with openbox) and starts a fresh editor on it
(ADR-0006). The pool size is NETPRINTS_E2E_WORKERS (default min(cores / 2, 4)).
If :100 is taken, set NETPRINTS_E2E_DISPLAY_START. Never use DISPLAY=:1.
dotnet build tests/NetPrints.Desktop.E2ETests -c Release
NETPRINTS_E2E=1 dotnet test --project tests/NetPrints.Desktop.E2ETests -c Release --no-build --no-progress --no-ansi -- --fail-skips on
Diagnostics of a failed scenario
When a scenario fails (an exception, an assertion, the test's own time budget running out, or the editor exiting
while the test holds it), the harness captures the state before it reports. The test then fails with an
E2EStepFailureException: its message starts with [step '<name>' running for <n> s] and its InnerException is the
original failure, unchanged.
The files are written to TestResults/e2e-diagnostics/<TestClass>/ and, on CI, uploaded with the e2e-results artifact:
| File | Content |
|---|---|
summary.md | The failure kind, the test, the step that was running and for how long, the worker display, the editor PID and the UTC time. |
timings.md | Every step with its duration; the step that was open is marked (running). |
ui-tree.json | For each editor window, every control with an automation id: automationId, name, type, bounds, isVisible, isEnabled, hasFocus; the top-level focused names the focused control. If the editor is gone it holds its exit code instead. |
display.png | The whole display (root window), including platform dialogs. |
editor.log | The last 400 lines the editor logged, then the tail of its stderr. |
process.txt | running, or exited <code>. |
run-state.json | The last launched program: phase (notStarted, building, running, exited), exitCode (present only once the program has exited), and the last 200 lines of stdout and stderr. It tells a slow run from lost output. |
capture-errors.txt | Only when a part could not be captured: which one and why. |
Each part has 10 seconds and the whole capture 30, so a hung editor cannot block it.
Forcing a timeout
E2EDiagnosticsTests proves the capture by holding the open project step until its 20 s budget runs out, through a
hook of its own class (ForcedTimeoutStep); it never sets an environment variable, so parallel workers are unaffected.
To see the diagnostics of any scenario locally, name the step in NETPRINTS_E2E_FORCE_TIMEOUT for a run of one class:
NETPRINTS_E2E=1 NETPRINTS_E2E_FORCE_TIMEOUT='run' tests/NetPrints.Desktop.E2ETests/bin/Release/net10.0/NetPrints.Desktop.E2ETests -class '*EditCompileAndRunTests'
The step names are the using (Step("...")) blocks of the scenarios. The step is held at its next checkpoint, so name
one that records a checkpoint (start, open project, edit graph, run, create project, add references). compile and the nested
wait for worker (timed inside start) have none, and forcing a step that was never held fails the scenario loudly. The variable is for manual runs only: set for a whole
run, it would hold that step in every test.
Documentation checks and guide screenshots
The editor guides live in docs/guide/editor/ and their screenshots in website/static/img/guide/editor/. Two tests in
NetPrints.Editor.Tests keep them honest:
-
ShortcutGuideTestsgenerates the shortcut table (menu, command, shortcut and where it works) from the command registry and compares it with the table between theshortcuts:beginandshortcuts:endmarkers ofdocs/guide/editor/commands-and-shortcuts.md. After a command or a shortcut changes, rewrite the table instead of editing it by hand:NETPRINTS_UPDATE_SNAPSHOTS=1 dotnet test --project tests/NetPrints.Editor.Tests -c Release --no-build --no-progress --no-ansi -- --filter-class '*ShortcutGuideTests' -
GuidePagesTestschecks that each of the six guide pages exists, shows at least one screenshot, and that every PNG a page references is in the images folder.
The screenshots come from GuideScreenshotTests, a scenario of the Desktop E2E project that drives the real editor on a
private X11 display with the HelloWorld sample copied to a neutral folder under the system temporary folder, so no
screenshot shows your own paths. It is an explicit test ([Fact(Explicit = true)]): a normal run, and CI's
--fail-skips on run, neither run it nor fail for it. Regenerate the images with the script, which builds the project,
picks a free display from :100 up and lists the files it wrote:
scripts/guide-screenshots.sh
It needs Xvfb, openbox, xdotool and ImageMagick like the other desktop tests. Look at every image that changed before you commit it: a guide may only describe what the screenshot run showed. Add a capture to the class, not a hand-made image.
scripts/build-docs.sh builds the whole site (the pages, the DocFX API reference at /api/ and the JSON Schemas) into
website/build; the Docs workflow runs the same script, and a broken link fails the build.
Continuous integration
Two workflows run tests (ADR-0019).
CI (.github/workflows/ci.yml)
Runs on every pull request to master, every push to master and on demand.
| Job | Check name | What it does |
|---|---|---|
checks | Repository checks (Linux) | Builds NetPrints.slnx in Release, runs the graph checks (regen --check, format --check), dotnet format --verify-no-changes, the Desktop E2E smoke step (no display, always-on tests only), the CLI smoke and the sample compile and run, and fails on any changed generated file. |
test | Test (Core), Test (Catalog), Test (CLI), Test (Editor), Test (Editor UI (headless) 1/4) to 4/4 | One matrix leg per test project, run in parallel (fail-fast: false); the headless UI project runs in four shards split by namespace (shard 4 is the complement of the --filter-namespace lists of shards 1 to 3). Each leg builds only its own project in Release and runs it with --report-xunit-trx and static code coverage (ADR-0008). |
build-test | Build and test (Linux) | The aggregate. It runs even when a job it needs failed and fails unless checks, every test leg, godot-extension and godot-smoke succeeded. |
e2e | Desktop E2E (Linux, Xvfb) | The Desktop E2E project on Xvfb with NETPRINTS_E2E=1 and --fail-skips on. |
packages, desktop-publish, jsonschema | Packages (local feed), Self-contained editor smoke (linux-x64), JSON Schema (metaschema, lint, validate) | Packaging and schema checks; they run no tests. |
godot-extension, godot-smoke | Godot extension (packed packages), Godot extension loads in the editor host | The sample extension built, tested, packed and verified against the packed packages through netprints-extension.yml, then loaded by the editor host (GodotExtensionHostTests). Both are in the aggregate. |
Branch protection requires two checks, Build and test (Linux) and Desktop E2E (Linux, Xvfb). Do not rename them.
A new test leg is covered by the aggregate without touching branch protection.
Artifacts, uploaded even when a job fails:
| Artifact | From | Content |
|---|---|---|
test-results-<leg> | each test leg (core, catalog, cli, editor, editor-ui-1 to editor-ui-4) | The .trx results of that project. |
coverage-<leg> | each test leg | Cobertura coverage of that project. |
ui-headless-<leg> | each editor-ui-* shard | Snapshot actual and diff images, flow screenshots and per-test diagnostics. |
e2e-results | e2e | The .trx results, ui/, e2e-timings-*.md and e2e-diagnostics/ (see Diagnostics of a failed scenario). |
Performance budgets (non-blocking) (the perf job in ci.yml)
Runs the tests with [Trait("Category", "Performance")] of tests/NetPrints.Core.Tests, NetPrints.Catalog.Tests,
NetPrints.Editor.Tests and NetPrints.Editor.UITests with
NETPRINTS_PERF_STRICT=1, which switches ProjectOpenPerformanceTests from its sanity bound to the SC-005 strict gate
(3x the spec target, best of three restored runs). It is not a required check and is not in the build-test aggregate.
The blocking Editor leg runs the same tests with the sanity bound only, so a slow shared runner cannot fail a pull
request, while an order-of-magnitude regression still does.
The job then judges SC-005's M1 (shell start) and M2 (graph scope) against R5-0, the commit before the composition
moved to MS.DI, on the same runner (ADR-0031 amendment 1):
eng/perf/prepare-r5-0.sh checks R5-0 out as a worktree, applies eng/perf/r5-0.patch and copies this tree's
tests/NetPrints.Testing/PerformanceGate.cs into it, and builds its two test projects; eng/perf/ab.sh runs the two
trees in alternation, one fresh test process per run, and fails when the head's time (GC pauses left out) or allocated
bytes is more than 10% over R5-0's. It prints every run, both trees' figures, the ratios and the CPU model, and the
per-run lines are uploaded as TestResults/perf-ab.txt. To run it locally, build the head's
tests/NetPrints.Editor.Tests and tests/NetPrints.Editor.UITests in Release, then:
eng/perf/prepare-r5-0.sh ../r5-0 # once; it resets the worktree on later calls
eng/perf/ab.sh ../r5-0 . # --rounds N (default 8), --metrics "M1", --results FILE
git worktree remove --force ../r5-0 # when done
A change to how the M1 or M2 test times a sample (TimeOneStartAsync, TimeOneBuild) is made in the test and in
eng/perf/r5-0.patch, so both trees keep measuring the same thing.
CLI (Windows) (.github/workflows/cli-windows.yml)
Builds tests/NetPrints.Cli.Tests and runs it on windows-latest, and uploads test-results-cli-windows. It runs on
pull requests and pushes to master that touch the CLI, a project in its transitive ProjectReference closure, the
test fixtures or the build files (the path filter), and on demand. It is not a required check.
CiWorkflowTests.CliWindowsPathFilterCoversTheCliTestsBuildInputs fails when a project joins that closure without a
path entry.
A CLI test that cannot run on Windows calls Assert.Skip("<what is missing>") behind an OS check (for example a POSIX
file mode). ShowTextconvEncodingTests never skips: it pins show --textconv to UTF-8 without a byte order mark,
which the git diff driver needs.
Adding a test project
- Create
tests/<Name>.Tests(or.UITests) and add it toNetPrints.slnx. - Add a row to the
testmatrix inci.ymlwith thename(the check name), theleg(the artifact slug) and theproject.CiWorkflowTestsfails while a project undertests/has no row, has two, or a row names a missing project. The Desktop E2E project is the one exception: it runs ine2e. - If the project joins the CLI tests' closure, add its folder to the
pathsof both triggers incli-windows.yml. - Add the project to the table at the top of this page.