tests: speed up unit tests#5980
Merged
Merged
Conversation
Bobronium
force-pushed
the
arseny/concurrent-tests
branch
2 times, most recently
from
a8f9f03 to
d151c7d
Compare
Bobronium
force-pushed
the
arseny/concurrent-tests
branch
2 times, most recently
from
cccb2b0 to
927a04f
Compare
Run a module's async tests together on one event loop via asyncio.gather, with per-test output/log/caplog capture and leaked-task attribution keyed on a contextvar, a live progress line, and fd-level capture so a spawned child's stdout can't corrupt it. Opt in per test with @pytest.mark.concurrent (and out with @pytest.mark.no_concurrent); --concurrent / --no-concurrent / LK_TEST_CONCURRENCY flip the default. Wired in via pyproject (-p tests.concurrency) and conftest. Co-authored-by: Claude <noreply@anthropic.com> https://claude.ai/code/session_01UFjbSTeVMYyRhLjZmcVRGd
Bobronium
force-pushed
the
arseny/concurrent-tests
branch
from
927a04f to
396bce8
Compare
@pytest.mark.virtual_time runs a test on a SelectorEventLoop whose clock jumps to the next scheduled timer instead of sleeping, so timing-coupled voice tests finish in ~0 wall time and deterministically. time.time()/perf_counter() and captured default_factory clocks are scoped to the running loop for the marked test only; off-loop callers keep the real clock. real_time opts a test out. Co-authored-by: Claude <noreply@anthropic.com> https://claude.ai/code/session_01UFjbSTeVMYyRhLjZmcVRGd
Mark each async unit module concurrent or no_concurrent, and the timing-coupled voice modules virtual_time + no_concurrent, so they run on the deterministic loop sequentially. Sync-only modules stay plain unit. Co-authored-by: Claude <noreply@anthropic.com> https://claude.ai/code/session_01UFjbSTeVMYyRhLjZmcVRGd
Trim fixed waits now that these run fast: test_ipc shutdown/aclose guardrails (_SESSION_ACLOSE_TIMEOUT 2->0.5s, simulated shutdown 1->0.3s) and its channel teardown sleep; test_audio_decoder reader warm-up 1->0.1s; test_room reconnect audio-continuity windows 1.5->1.0s. Co-authored-by: Claude <noreply@anthropic.com> https://claude.ai/code/session_01UFjbSTeVMYyRhLjZmcVRGd
Run the blocking proc.join() in an executor so it can't stall the shared loop, and inject the all-spawns-fail case (#5868) through a real init fn that raises instead of monkeypatching ProcJobExecutor process-wide. Co-authored-by: Claude <noreply@anthropic.com> https://claude.ai/code/session_01UFjbSTeVMYyRhLjZmcVRGd
Bobronium
force-pushed
the
arseny/concurrent-tests
branch
from
396bce8 to
0973861
Compare
Bobronium
changed the title
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running
pytest --unitwith these changes goes from 3 minutes 44 seconds to just 30 seconds.How does it work
Virtual time
Takes advantage of the fact that many of the tests rely on
asyncio.sleep()to synchronize events in the loop. It unties it from the real time so thatawait asyncio.sleep(60)returns instantly withloop.time()reporting 60 seconds passed.time.time()is then patched to report loop time. This brings downtest_agent_session.pytest time form 90 seconds to just 1 second without any modifications to the test itself.--real-timeoption can be used.pytest.mark.virtual_time/.real_timecan be used.Concurrent tests
Tests with real I/O are groupped by their module and executed in a shared event loop. This shrinks wall time of the whole module to wall time of the slowest test + raw compute.
contextvarsare used to preserve input capture and track leaked tasks.--no-concurrentoption can be used.pytest.mark.no_concurrent/.no_concurrentcan be used.Changes
pytest-asyncio-concurrent— gives the speedup for I/O bound teststests/concurrent.pythat extends onpytest-asyncio-concurrentand adds crucial features:pytest)fail_on_leaked_taskscontinues to work)--concurrent/--no-concurrentoptionstests/virtual_time.pythat provides instant and deterministicasyncioclock for time-bound executionChanges to the tests sources
concurrent/virtual_timemarkers to tests modulestest_audio_decoder.py,test_ipc.ptandtest_room.pytest_async_channelandtest_proc_pool_launch_job_raises_when_all_spawns_failintest_ipc.pysafe to run concurrently