Close SocketChannel on pre-registration failure in TlsChannelStream

[rozza]

Resolve address before opening SocketChannel to avoid FD leak on DNS failure. Close the channel in catch blocks if opened but not yet registered with the selector monitor. Cancel pending registration action to prevent races with the timeout scheduler.

JAVA-6216

[rozza]

This is #1972 reworked slightly and assigned to a new ticket.

[Copilot]

Pull request overview

This PR updates TlsChannelStream connection establishment to prevent file descriptor leaks and avoid races when failures occur before a SocketChannel is registered with the selector monitor (e.g., DNS resolution failures, connect-time failures).

Changes:

• Resolve the target InetSocketAddress before opening the SocketChannel to avoid leaking an FD when name resolution fails.
• Ensure an opened (but not yet registered) SocketChannel is closed on failures and cancel the pending registration action to prevent timeout/registration races.
• Add functional tests covering DNS-resolution failure (no channel opened) and connect failure (channel closed), plus minor agent-guidance updates.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
driver-core/src/test/functional/com/mongodb/internal/connection/TlsChannelStreamFunctionalTest.java	Adds functional tests for “no open on DNS failure” and “close on pre-registration connect failure”.
driver-core/src/main/com/mongodb/internal/connection/TlsChannelStreamFactoryFactory.java	Refactors openAsync to resolve before opening, and closes/cancels on pre-registration failures.
driver-core/AGENTS.md	Adds a driver-core-specific reminder about handling async errors via callbacks/handlers.
AGENTS.md	Adds a repo-wide style guideline preferring lambdas over SAM anonymous classes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

driver-core/src/main/com/mongodb/internal/connection/TlsChannelStreamFactoryFactory.java:281

• scheduleTimeoutInterruption can close the SocketChannel before the SelectorMonitor thread has processed pendingRegistrations. In that case the selector thread will later attempt socketChannel.register(...) on a closed channel, which throws (and because the exception occurs before iter.remove(), the registration can remain in pendingRegistrations indefinitely, potentially blocking/ delaying later registrations and producing repeated selector-loop warnings on subsequent wakeups). Consider making the selector-monitor registration loop resilient by removing a pending registration when it has been canceled (afterConnectAction == null / tryCancelPendingConnection already happened) and/or ensuring failed register attempts still remove the entry.

        private void scheduleTimeoutInterruption(final AsyncCompletionHandler<Void> handler,
                                                 final SelectorMonitor.SocketRegistration socketRegistration,
                                                 final int connectTimeoutMs) {
            group.getTimeoutExecutor().schedule(() -> {
                if (socketRegistration.tryCancelPendingConnection()) {
                    closeAndTimeout(handler, socketRegistration.socketChannel);
                }
            }, connectTimeoutMs, TimeUnit.MILLISECONDS);
        }