Skip to main content

Testing

Test projects​

ProjectCovers
NetPrints.Core.TestsThe model, serialization, translation and the repository rules (SourceHygieneTests, golden fixtures).
NetPrints.Analyzers.TestsEvery Roslyn analyzer, code fix and generator NetPrints owns (ADR-0024): the analyzer assemblies load into a CompilationWithAnalyzers host and report their supported diagnostics.
NetPrints.Catalog.TestsThe type catalog and its annotations.
NetPrints.Cli.TestsThe netprints command line, in process (CliTestHost); some tests run the real tool or the real project system as child processes.
NetPrints.Editor.TestsEditor view models and hosting services, over fakes (no window), and the checks of the editor guides (below).
NetPrints.Editor.UITestsThe editor's views, headless (Avalonia headless platform), through page objects and AutomationIds.
NetPrints.Desktop.E2ETestsThe real desktop editor on a private X11 display (Linux only).
NetPrints.Testing, NetPrints.Testing.UiShared fixtures, the driver abstraction, page objects and the Screenplay layer; not tests themselves.

Running them​

Build quietly, then run without building again (the solution selects the Microsoft.Testing.Platform runner):

dotnet build -v q -tl:off --nologo
dotnet test --no-build --no-progress --no-ansi

Run one class with the test assembly itself (--treenode-filter is not accepted):

tests/NetPrints.Editor.Tests/bin/Debug/net10.0/NetPrints.Editor.Tests -class '*RunStateTrackerTests'

The full suite is what CI's test job runs, and the format check is part of the gate:

dotnet test --solution NetPrints.slnx -c Release --no-build --no-progress --no-ansi -- --ignore-exit-code 8
dotnet format NetPrints.slnx --verify-no-changes

Never run the whole NetPrints.Editor.UITests project under Xvfb (it runs out of memory); the headless tests need no display.

Desktop E2E tests​

They need Linux with Xvfb, openbox, xdotool, ImageMagick and GTK 3, and run only with NETPRINTS_E2E=1; without it the scenarios skip and only the always-on tests (failure capture, entry point, worker pool) run; the exit code is 0. Each test rents a worker from a pool (a private Xvfb display with openbox) and starts a fresh editor on it (ADR-0006). The pool size is NETPRINTS_E2E_WORKERS (default min(cores / 2, 4)). If :100 is taken, set NETPRINTS_E2E_DISPLAY_START. Never use DISPLAY=:1.

dotnet build tests/NetPrints.Desktop.E2ETests -c Release
NETPRINTS_E2E=1 dotnet test --project tests/NetPrints.Desktop.E2ETests -c Release --no-build --no-progress --no-ansi -- --fail-skips on

Diagnostics of a failed scenario​

When a scenario fails (an exception, an assertion, the test's own time budget running out, or the editor exiting while the test holds it), the harness captures the state before it reports. The test then fails with an E2EStepFailureException: its message starts with [step '<name>' running for <n> s] and its InnerException is the original failure, unchanged.

The files are written to TestResults/e2e-diagnostics/<TestClass>/ and, on CI, uploaded with the e2e-results artifact:

FileContent
summary.mdThe failure kind, the test, the step that was running and for how long, the worker display, the editor PID and the UTC time.
timings.mdEvery step with its duration; the step that was open is marked (running).
ui-tree.jsonFor each editor window, every control with an automation id: automationId, name, type, bounds, isVisible, isEnabled, hasFocus; the top-level focused names the focused control. If the editor is gone it holds its exit code instead.
display.pngThe whole display (root window), including platform dialogs.
editor.logThe last 400 lines the editor logged, then the tail of its stderr.
process.txtrunning, or exited <code>.
run-state.jsonThe last launched program: phase (notStarted, building, running, exited), exitCode (present only once the program has exited), and the last 200 lines of stdout and stderr. It tells a slow run from lost output.
capture-errors.txtOnly when a part could not be captured: which one and why.

Each part has 10 seconds and the whole capture 30, so a hung editor cannot block it.

Forcing a timeout​

E2EDiagnosticsTests proves the capture by holding the open project step until its 20 s budget runs out, through a hook of its own class (ForcedTimeoutStep); it never sets an environment variable, so parallel workers are unaffected. To see the diagnostics of any scenario locally, name the step in NETPRINTS_E2E_FORCE_TIMEOUT for a run of one class:

NETPRINTS_E2E=1 NETPRINTS_E2E_FORCE_TIMEOUT='run' tests/NetPrints.Desktop.E2ETests/bin/Release/net10.0/NetPrints.Desktop.E2ETests -class '*EditCompileAndRunTests'

The step names are the using (Step("...")) blocks of the scenarios. The step is held at its next checkpoint, so name one that records a checkpoint (start, open project, edit graph, run, create project, add references). compile and the nested wait for worker (timed inside start) have none, and forcing a step that was never held fails the scenario loudly. The variable is for manual runs only: set for a whole run, it would hold that step in every test.

Documentation checks and guide screenshots​

The editor guides live in docs/guide/editor/ and their screenshots in website/static/img/guide/editor/. Two tests in NetPrints.Editor.Tests keep them honest:

  • ShortcutGuideTests generates the shortcut table (menu, command, shortcut and where it works) from the command registry and compares it with the table between the shortcuts:begin and shortcuts:end markers of docs/guide/editor/commands-and-shortcuts.md. After a command or a shortcut changes, rewrite the table instead of editing it by hand:

    NETPRINTS_UPDATE_SNAPSHOTS=1 dotnet test --project tests/NetPrints.Editor.Tests -c Release --no-build --no-progress --no-ansi -- --filter-class '*ShortcutGuideTests'
  • GuidePagesTests checks that each of the six guide pages exists, shows at least one screenshot, and that every PNG a page references is in the images folder.

The screenshots come from GuideScreenshotTests, a scenario of the Desktop E2E project that drives the real editor on a private X11 display with the HelloWorld sample copied to a neutral folder under the system temporary folder, so no screenshot shows your own paths. It is an explicit test ([Fact(Explicit = true)]): a normal run, and CI's --fail-skips on run, neither run it nor fail for it. Regenerate the images with the script, which builds the project, picks a free display from :100 up and lists the files it wrote:

scripts/guide-screenshots.sh

It needs Xvfb, openbox, xdotool and ImageMagick like the other desktop tests. Look at every image that changed before you commit it: a guide may only describe what the screenshot run showed. Add a capture to the class, not a hand-made image.

scripts/build-docs.sh builds the whole site (the pages, the DocFX API reference at /api/ and the JSON Schemas) into website/build; the Docs workflow runs the same script, and a broken link fails the build.

Continuous integration​

Two workflows run tests (ADR-0019).

CI (.github/workflows/ci.yml)​

Runs on every pull request to master, every push to master and on demand.

JobCheck nameWhat it does
checksRepository checks (Linux)Builds NetPrints.slnx in Release, runs the graph checks (regen --check, format --check), dotnet format --verify-no-changes, the Desktop E2E smoke step (no display, always-on tests only), the CLI smoke and the sample compile and run, and fails on any changed generated file.
testTest (Core), Test (Catalog), Test (CLI), Test (Editor), Test (Editor UI (headless) 1/4) to 4/4One matrix leg per test project, run in parallel (fail-fast: false); the headless UI project runs in four shards split by namespace (shard 4 is the complement of the --filter-namespace lists of shards 1 to 3). Each leg builds only its own project in Release and runs it with --report-xunit-trx and static code coverage (ADR-0008).
build-testBuild and test (Linux)The aggregate. It runs even when a job it needs failed and fails unless checks, every test leg, godot-extension and godot-smoke succeeded.
e2eDesktop E2E (Linux, Xvfb)The Desktop E2E project on Xvfb with NETPRINTS_E2E=1 and --fail-skips on.
packages, desktop-publish, jsonschemaPackages (local feed), Self-contained editor smoke (linux-x64), JSON Schema (metaschema, lint, validate)Packaging and schema checks; they run no tests.
godot-extension, godot-smokeGodot extension (packed packages), Godot extension loads in the editor hostThe sample extension built, tested, packed and verified against the packed packages through netprints-extension.yml, then loaded by the editor host (GodotExtensionHostTests). Both are in the aggregate.

Branch protection requires two checks, Build and test (Linux) and Desktop E2E (Linux, Xvfb). Do not rename them. A new test leg is covered by the aggregate without touching branch protection.

Artifacts, uploaded even when a job fails:

ArtifactFromContent
test-results-<leg>each test leg (core, catalog, cli, editor, editor-ui-1 to editor-ui-4)The .trx results of that project.
coverage-<leg>each test legCobertura coverage of that project.
ui-headless-<leg>each editor-ui-* shardSnapshot actual and diff images, flow screenshots and per-test diagnostics.
e2e-resultse2eThe .trx results, ui/, e2e-timings-*.md and e2e-diagnostics/ (see Diagnostics of a failed scenario).

Performance budgets (non-blocking) (the perf job in ci.yml)​

Runs the tests with [Trait("Category", "Performance")] of tests/NetPrints.Core.Tests, NetPrints.Catalog.Tests, NetPrints.Editor.Tests and NetPrints.Editor.UITests with NETPRINTS_PERF_STRICT=1, which switches ProjectOpenPerformanceTests from its sanity bound to the SC-005 strict gate (3x the spec target, best of three restored runs). It is not a required check and is not in the build-test aggregate. The blocking Editor leg runs the same tests with the sanity bound only, so a slow shared runner cannot fail a pull request, while an order-of-magnitude regression still does.

The job then judges SC-005's M1 (shell start) and M2 (graph scope) against R5-0, the commit before the composition moved to MS.DI, on the same runner (ADR-0031 amendment 1): eng/perf/prepare-r5-0.sh checks R5-0 out as a worktree, applies eng/perf/r5-0.patch and copies this tree's tests/NetPrints.Testing/PerformanceGate.cs into it, and builds its two test projects; eng/perf/ab.sh runs the two trees in alternation, one fresh test process per run, and fails when the head's time (GC pauses left out) or allocated bytes is more than 10% over R5-0's. It prints every run, both trees' figures, the ratios and the CPU model, and the per-run lines are uploaded as TestResults/perf-ab.txt. To run it locally, build the head's tests/NetPrints.Editor.Tests and tests/NetPrints.Editor.UITests in Release, then:

eng/perf/prepare-r5-0.sh ../r5-0 # once; it resets the worktree on later calls
eng/perf/ab.sh ../r5-0 . # --rounds N (default 8), --metrics "M1", --results FILE
git worktree remove --force ../r5-0 # when done

A change to how the M1 or M2 test times a sample (TimeOneStartAsync, TimeOneBuild) is made in the test and in eng/perf/r5-0.patch, so both trees keep measuring the same thing.

CLI (Windows) (.github/workflows/cli-windows.yml)​

Builds tests/NetPrints.Cli.Tests and runs it on windows-latest, and uploads test-results-cli-windows. It runs on pull requests and pushes to master that touch the CLI, a project in its transitive ProjectReference closure, the test fixtures or the build files (the path filter), and on demand. It is not a required check. CiWorkflowTests.CliWindowsPathFilterCoversTheCliTestsBuildInputs fails when a project joins that closure without a path entry.

A CLI test that cannot run on Windows calls Assert.Skip("<what is missing>") behind an OS check (for example a POSIX file mode). ShowTextconvEncodingTests never skips: it pins show --textconv to UTF-8 without a byte order mark, which the git diff driver needs.

Adding a test project​

  1. Create tests/<Name>.Tests (or .UITests) and add it to NetPrints.slnx.
  2. Add a row to the test matrix in ci.yml with the name (the check name), the leg (the artifact slug) and the project. CiWorkflowTests fails while a project under tests/ has no row, has two, or a row names a missing project. The Desktop E2E project is the one exception: it runs in e2e.
  3. If the project joins the CLI tests' closure, add its folder to the paths of both triggers in cli-windows.yml.
  4. Add the project to the table at the top of this page.