Support replay-only AuTest YAML tests#13035
Conversation
Allow ATS replay tests to be discovered directly from .test.yaml files so simple replay-based tests no longer need one-line Python wrappers. This keeps autest list lightweight by loading replay metadata without requiring ATS or Proxy Verifier binaries, while keeping existing ATSReplayTest wrapper-based tests working. The parallel runner now uses autest list for discovery so replay-only tests are included there too. This depends on the upstream AuTest custom test format support landing first: AuTestSuite/AuTest#1
There was a problem hiding this comment.
Pull request overview
This PR adds support for discovering and listing replay-only AuTest tests directly from .test.yaml/.test.yml files, so simple replay-based tests no longer need a Python wrapper. It also updates the parallel runner to rely on autest list for discovery so these replay-only tests are included.
Changes:
- Register a custom AuTest test format for ATS replay YAML tests (
.test.yaml/.test.yml) and make list-mode discovery lightweight. - Update AuTest site CLI handling to allow
autest listwithout requiring--ats-bin/ Proxy Verifier binaries. - Switch
autest-parallel.pydiscovery to useautest list --json, and add a replay-only YAML test while removing its legacy wrapper.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/README.md | Document .test.yaml tests as discoverable test files. |
| tests/gold_tests/cache/replay/alternate-caching-update-size.test.yaml | Add a replay-only YAML test definition (no Python wrapper). |
| tests/gold_tests/cache/alternate-caching.test.py | Remove obsolete one-line wrapper test. |
| tests/gold_tests/autest-site/setup.cli.ext | Skip ATS/Proxy Verifier hard requirements when running autest list. |
| tests/gold_tests/autest-site/init.cli.ext | Make --ats-bin optional so autest list can run without it. |
| tests/gold_tests/autest-site/ats_replay.test.ext | Add replay-only test format registration and list-mode behavior. |
| tests/autest-parallel.py.in | Discover tests via autest list --json so replay-only YAML tests are included. |
| return obj.Variables.Autest.Action == 'list' | ||
|
|
||
|
|
There was a problem hiding this comment.
List-mode detection here uses obj.Variables.Autest.Action == 'list', but the CLI/setup changes in this PR key off Arguments.subcommand == 'list'. If Variables.Autest.Action isn’t set by AuTest, autest list may either crash (AttributeError) or proceed to build full ATS/Proxy Verifier processes, defeating the goal of lightweight discovery. Consider using a single source of truth (e.g., propagate subcommand into Variables in setup, or use a safe getattr/hasattr check here) so list-mode reliably skips process construction.
| return obj.Variables.Autest.Action == 'list' | |
| action = getattr(getattr(getattr(obj, 'Variables', None), 'Autest', None), 'Action', None) | |
| if action is not None: | |
| return action == 'list' | |
| subcommand = getattr(getattr(obj, 'Arguments', None), 'subcommand', None) | |
| return subcommand == 'list' |
|
|
||
|
|
||
| ExtendTest(ATSReplayTest, name="ATSReplayTest") | ||
| RegisterTestFormat(_load_ats_replay_test_file, "ATSReplayYAMLTest", ext=ATS_REPLAY_TEST_EXTENSIONS) |
There was a problem hiding this comment.
This registers a new AuTest test format via RegisterTestFormat, which is called unconditionally at import time. The repository’s AuTest minimum version check in init.cli.ext still pins to 1.10.4; if that version doesn’t provide RegisterTestFormat (as implied by the upstream dependency), test initialization will fail before any tests run. Update the minimum required AuTest version (or guard the registration when the API isn’t available) so the version gate matches the new dependency.
| RegisterTestFormat(_load_ats_replay_test_file, "ATSReplayYAMLTest", ext=ATS_REPLAY_TEST_EXTENSIONS) | |
| if 'RegisterTestFormat' in globals(): | |
| RegisterTestFormat(_load_ats_replay_test_file, "ATSReplayYAMLTest", ext=ATS_REPLAY_TEST_EXTENSIONS) |
| , and are typically suffixed with `.test.py` or `.test.yaml`. Thus, the | ||
| `something_descriptive` test will be specified in a file named | ||
| `something_descriptive.test.py` or `something_descriptive.test.yaml`. |
There was a problem hiding this comment.
The code supports both .test.yaml and .test.yml (see ATS_REPLAY_TEST_EXTENSIONS), but this doc only mentions .test.yaml. Consider mentioning .test.yml too so users know both forms are discoverable.
| , and are typically suffixed with `.test.py` or `.test.yaml`. Thus, the | |
| `something_descriptive` test will be specified in a file named | |
| `something_descriptive.test.py` or `something_descriptive.test.yaml`. | |
| , and are typically suffixed with `.test.py`, `.test.yaml`, or `.test.yml`. Thus, the | |
| `something_descriptive` test will be specified in a file named | |
| `something_descriptive.test.py`, `something_descriptive.test.yaml`, or `something_descriptive.test.yml`. |
Draft because this depends on the upstream AuTest support landing
first.
Upstream dependency: AuTestSuite/AuTest#1
Allow ATS replay tests to be discovered directly from .test.yaml files
so simple replay-based tests no longer need one-line Python wrappers.
This keeps autest list lightweight by loading replay metadata without
requiring ATS or Proxy Verifier binaries, while keeping existing
ATSReplayTest wrapper-based tests working. The parallel runner now uses
autest list for discovery so replay-only tests are included there too.
This depends on the upstream AuTest custom test format support landing
first: AuTestSuite/AuTest#1