[FLINK-39367][connector-http] Add configurable error logging with detailed context#35
Open
featzhang wants to merge 6 commits into
Open
[FLINK-39367][connector-http] Add configurable error logging with detailed context#35featzhang wants to merge 6 commits into
featzhang wants to merge 6 commits into
Conversation
davidradl
reviewed
Mar 17, 2026
davidradl
reviewed
Mar 17, 2026
featzhang
changed the title
Introduce HttpErrorLogger to surface HTTP errors at a configurable SLF4J severity level (ERROR/WARN/INFO), independently of the existing http.logging.level mechanism which controls content verbosity at DEBUG. New configuration options: - http.error.log.level : log level for errors (default: ERROR) - http.error.log.include.body : include request/response body (default: false) - http.error.log.body.max.size: max body characters logged (default: 1024) Sensitive headers (authorization, cookie, api-key, x-api-key) are automatically masked in all HttpErrorLogger output. Behavior respects continue-on-error: - continue-on-error=false: errors are thrown and logged at configured level - continue-on-error=true : errors are tolerated and logged at DEBUG only, since ERROR/WARN would be misleading when the connector is functioning as configured Also adds a dedicated logSinkError overload for status-code-only failures to avoid constructing synthetic Exception objects when no exception occurred.
featzhang
force-pushed
the
feature/improve-error-logging
branch
from
97a703b to
ca2e622
Compare
davidradl
reviewed
Apr 21, 2026
|
|
||
| // -------------- Error logging configuration -------------- | ||
|
|
||
| public static final String ERROR_LOG_LEVEL = FLINK_CONNECTOR_HTTP + "error.log.level"; |
Contributor
There was a problem hiding this comment.
I think we should fit in with the existing configuration based logging . We added this so that potentially sensitive information in the http messages could be obfuscated. We agreed that it would be possible to log out the clear content for development debugging. I suggest we use the existing controls and scope and extend them to to cover sink
Per reviewer feedback, removed HttpErrorLogger class and integrated error logging functionality directly into the existing HttpLogger. Changes: - Extended HttpLogger with http.error.log.severity config (OFF/ERROR/WARN/INFO) - Enhanced logLookupError() and logSinkError() to respect error severity level - Maintained http.logging.level (MIN/REQ_RESP/MAX) for content verbosity - Two orthogonal dimensions: content detail × error severity - Backward compatible: default behavior unchanged (errors at DEBUG) - Production-friendly: set http.error.log.severity=ERROR for monitoring Removed: - HttpErrorLogger class and related files - Duplicate error logging configuration (ERROR_LOG_LEVEL, ERROR_LOG_INCLUDE_BODY, ERROR_LOG_BODY_MAX_SIZE) Kept: - Existing HttpLogger infrastructure - Sensitive header masking - Continue-on-error awareness (DEBUG when tolerated) The refactored design provides: 1. http.logging.level (MIN/REQ_RESP/MAX) controls content verbosity 2. http.error.log.severity (OFF/INFO/WARN/ERROR) controls log level for actual errors 3. Errors with continue-on-error=true always log at DEBUG (tolerated errors) 4. Errors with continue-on-error=false respect http.error.log.severity setting
- Remove 9 trailing whitespace violations detected by Checkstyle - Lines fixed: 53, 125, 131, 212, 215, 237, 257, 260, 319 - No functional changes, formatting only
- Fix Javadoc line length in HttpConnectorConfigConstants - Reformat multiline method calls to match Google Java Style - Fix indentation in log method calls (8 spaces for continuation) - Files updated: * HttpConnectorConfigConstants.java * HttpLogger.java * JavaNetSinkHttpClient.java * JavaNetHttpPollingClient.java
- Consolidate method signatures to single line where possible - Reformat nested String.format() calls with proper indentation - Apply consistent 8-space continuation indent - StringBuilder initialization formatting
- Consolidate String.format calls to single line - Applies to two 'Request Headers' append operations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR improves HTTP request error logging in the HTTP connector by adding configurable error logging with detailed context. Previously, HTTP errors were logged with minimal information, making it difficult to diagnose issues in production environments. This enhancement provides more comprehensive error details including HTTP status codes, response bodies (configurable), and configurable log levels.
Change log
HttpErrorLogLevelenum for configurable error log levels (ERROR/WARN/INFO)HttpErrorLoggerclass for enhanced error logging with detailed contexthttp.error.log.level: Controls error log level (default: ERROR)http.error.log.include.body: Whether to include response body in logs (default: false)http.error.log.body.max.size: Maximum body size to log in bytes (default: 1024)HttpErrorLoggerinJavaNetHttpPollingClientandJavaNetSinkHttpClientVerifying
HttpErrorLogger(12 tests, all passing)Affected Components
flink-connector-http-core: Core HTTP client implementationDocumentation
Configuration documentation should be updated to include the new error logging parameters:
http.error.log.levelhttp.error.log.include.bodyhttp.error.log.body.max.sizeBackward Compatibility
This change is fully backward compatible. All new configuration parameters have sensible defaults, and existing code will continue to work without modifications.