Skip to content

Cancel stream on abort or timeout to prevent CRT HTTP client connection pool exhaustion#6919

Merged
joviegas merged 13 commits into
masterfrom
joviegas/crt_close_stream_on_abort
Jun 4, 2026
Merged

Cancel stream on abort or timeout to prevent CRT HTTP client connection pool exhaustion#6919
joviegas merged 13 commits into
masterfrom
joviegas/crt_close_stream_on_abort

Conversation

@joviegas
Copy link
Copy Markdown
Contributor

Motivation and Context

When API call timeout fires, the SDK does not abort the in-flight CRT HTTP request, causing connections to remain permanently leased and eventually exhausting the connection pool.

Modifications

  • In CrtRequestExecutor (sync) and CrtAsyncRequestExecutor (async), register a
    requestFuture.whenComplete hook that calls HttpStreamBase.cancel() when the future
    completes exceptionally.

Testing

  • Did one off test in a service client where a proxy server was added to simulate a low responsive server.
  • Added Junit test cases

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

License

  • I confirm that this pull request can be released under the Apache 2 license

@joviegas joviegas requested a review from a team as a code owner April 29, 2026 16:56
@joviegas joviegas changed the title fix(aws-crt-http-client): cancel stream on abort/timeout to evict connection from pool Cancel stream on abort or timeout to prevent CRT HTTP client connection pool exhaustion Apr 29, 2026
@joviegas joviegas force-pushed the joviegas/crt_close_stream_on_abort branch from 55f6b1d to af112fb Compare April 29, 2026 17:15
@joviegas joviegas force-pushed the joviegas/crt_close_stream_on_abort branch from 494bbca to df131c1 Compare April 29, 2026 22:08
long finalAcquireStartTime = acquireStartTime;

streamFuture.whenComplete((streamBase, throwable) -> {
if (shouldPublishMetrics) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like we can invoke onAcquireStream here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


try (SdkHttpClient client = AwsCrtHttpClient.builder().maxConcurrency(3).build()) {
stubUnresponsiveServer();
executeAndAbort(client, uri, 3);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, why do we need to send 3 requests?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use 3 to match maxConcurrency(3) and verify all 3 dead connections are evicted and the pool can accept 3 new requests.

@joviegas joviegas force-pushed the joviegas/crt_close_stream_on_abort branch from 5efcffe to 603b69d Compare May 6, 2026 00:10
@joviegas joviegas force-pushed the joviegas/crt_close_stream_on_abort branch from 603b69d to 6a13a35 Compare May 6, 2026 00:22
joviegas added 4 commits May 26, 2026 07:33
…treamFuture.whenComplete in both sync and async executors, removing the redundant thenAccept callback and Also fix a connection pool leak race where an abort/timeout before stream acquisition left the late-acquired stream unclosed; closeConnection() now records intent so onAcquireStream() cancels and closes it
@joviegas joviegas force-pushed the joviegas/crt_close_stream_on_abort branch from 469b8e8 to 484e62c Compare May 29, 2026 22:43
@joviegas joviegas enabled auto-merge June 4, 2026 15:55
@joviegas joviegas added this pull request to the merge queue Jun 4, 2026
Merged via the queue into master with commit 4a9fe92 Jun 4, 2026
12 of 13 checks passed
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 4, 2026

This pull request has been closed and the conversation has been locked. Comments on closed PRs are hard for our team to see. If you need more assistance, please open a new issue that references this one.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Jun 4, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants