Skip to content

Support replay-only AuTest YAML tests#13035

Draft
bneradt wants to merge 1 commit intoapache:masterfrom
bneradt:autest-test-yaml-format
Draft

Support replay-only AuTest YAML tests#13035
bneradt wants to merge 1 commit intoapache:masterfrom
bneradt:autest-test-yaml-format

Conversation

@bneradt
Copy link
Copy Markdown
Contributor

@bneradt bneradt commented Mar 28, 2026

Draft because this depends on the upstream AuTest support landing
first.

Upstream dependency: AuTestSuite/AuTest#1

Allow ATS replay tests to be discovered directly from .test.yaml files
so simple replay-based tests no longer need one-line Python wrappers.

This keeps autest list lightweight by loading replay metadata without
requiring ATS or Proxy Verifier binaries, while keeping existing
ATSReplayTest wrapper-based tests working. The parallel runner now uses
autest list for discovery so replay-only tests are included there too.

This depends on the upstream AuTest custom test format support landing
first: AuTestSuite/AuTest#1

Allow ATS replay tests to be discovered directly from .test.yaml files
so simple replay-based tests no longer need one-line Python wrappers.

This keeps autest list lightweight by loading replay metadata without
requiring ATS or Proxy Verifier binaries, while keeping existing
ATSReplayTest wrapper-based tests working. The parallel runner now uses
autest list for discovery so replay-only tests are included there too.

This depends on the upstream AuTest custom test format support landing
first: AuTestSuite/AuTest#1
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds support for discovering and listing replay-only AuTest tests directly from .test.yaml/.test.yml files, so simple replay-based tests no longer need a Python wrapper. It also updates the parallel runner to rely on autest list for discovery so these replay-only tests are included.

Changes:

  • Register a custom AuTest test format for ATS replay YAML tests (.test.yaml / .test.yml) and make list-mode discovery lightweight.
  • Update AuTest site CLI handling to allow autest list without requiring --ats-bin / Proxy Verifier binaries.
  • Switch autest-parallel.py discovery to use autest list --json, and add a replay-only YAML test while removing its legacy wrapper.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/README.md Document .test.yaml tests as discoverable test files.
tests/gold_tests/cache/replay/alternate-caching-update-size.test.yaml Add a replay-only YAML test definition (no Python wrapper).
tests/gold_tests/cache/alternate-caching.test.py Remove obsolete one-line wrapper test.
tests/gold_tests/autest-site/setup.cli.ext Skip ATS/Proxy Verifier hard requirements when running autest list.
tests/gold_tests/autest-site/init.cli.ext Make --ats-bin optional so autest list can run without it.
tests/gold_tests/autest-site/ats_replay.test.ext Add replay-only test format registration and list-mode behavior.
tests/autest-parallel.py.in Discover tests via autest list --json so replay-only YAML tests are included.

Comment on lines +211 to +213
return obj.Variables.Autest.Action == 'list'


Copy link

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

List-mode detection here uses obj.Variables.Autest.Action == 'list', but the CLI/setup changes in this PR key off Arguments.subcommand == 'list'. If Variables.Autest.Action isn’t set by AuTest, autest list may either crash (AttributeError) or proceed to build full ATS/Proxy Verifier processes, defeating the goal of lightweight discovery. Consider using a single source of truth (e.g., propagate subcommand into Variables in setup, or use a safe getattr/hasattr check here) so list-mode reliably skips process construction.

Suggested change
return obj.Variables.Autest.Action == 'list'
action = getattr(getattr(getattr(obj, 'Variables', None), 'Autest', None), 'Action', None)
if action is not None:
return action == 'list'
subcommand = getattr(getattr(obj, 'Arguments', None), 'subcommand', None)
return subcommand == 'list'

Copilot uses AI. Check for mistakes.


ExtendTest(ATSReplayTest, name="ATSReplayTest")
RegisterTestFormat(_load_ats_replay_test_file, "ATSReplayYAMLTest", ext=ATS_REPLAY_TEST_EXTENSIONS)
Copy link

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This registers a new AuTest test format via RegisterTestFormat, which is called unconditionally at import time. The repository’s AuTest minimum version check in init.cli.ext still pins to 1.10.4; if that version doesn’t provide RegisterTestFormat (as implied by the upstream dependency), test initialization will fail before any tests run. Update the minimum required AuTest version (or guard the registration when the API isn’t available) so the version gate matches the new dependency.

Suggested change
RegisterTestFormat(_load_ats_replay_test_file, "ATSReplayYAMLTest", ext=ATS_REPLAY_TEST_EXTENSIONS)
if 'RegisterTestFormat' in globals():
RegisterTestFormat(_load_ats_replay_test_file, "ATSReplayYAMLTest", ext=ATS_REPLAY_TEST_EXTENSIONS)

Copilot uses AI. Check for mistakes.
Comment on lines +37 to +39
, and are typically suffixed with `.test.py` or `.test.yaml`. Thus, the
`something_descriptive` test will be specified in a file named
`something_descriptive.test.py` or `something_descriptive.test.yaml`.
Copy link

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code supports both .test.yaml and .test.yml (see ATS_REPLAY_TEST_EXTENSIONS), but this doc only mentions .test.yaml. Consider mentioning .test.yml too so users know both forms are discoverable.

Suggested change
, and are typically suffixed with `.test.py` or `.test.yaml`. Thus, the
`something_descriptive` test will be specified in a file named
`something_descriptive.test.py` or `something_descriptive.test.yaml`.
, and are typically suffixed with `.test.py`, `.test.yaml`, or `.test.yml`. Thus, the
`something_descriptive` test will be specified in a file named
`something_descriptive.test.py`, `something_descriptive.test.yaml`, or `something_descriptive.test.yml`.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants