Skip to content

OpenTelemetry logging#1733

Open
mzacho wants to merge 2 commits intoistio:masterfrom
mzacho:master
Open

OpenTelemetry logging#1733
mzacho wants to merge 2 commits intoistio:masterfrom
mzacho:master

Conversation

@mzacho
Copy link

@mzacho mzacho commented Jan 26, 2026

This PR adds as the ability to send tracing log Events to an OpenTelemetry collector via gRPC. When OTLP_LOGGING=true it uses opentelemetry-tracing-appender to add a OTEL tracing bridge as a layer to the current tracing subscriber registry. The current plain/json logging to stdout is not modified, so with this change ztunnel can either log:

  1. to stdout using the plain format
  2. to stdout using the plain format AND to otel collector in OTLP format
  3. to stdout using the json format
  4. to stdout using the json format AND to otel collector in OTLP format

The collector endpoint can be configured with OTEL_COLLECTOR_ENDPOINT.

cc @ilrudie, you mentioned in #1625 you'd be curios to see how intrusive OpenTelemetry support is.

@mzacho mzacho requested a review from a team as a code owner January 26, 2026 09:42
@istio-policy-bot
Copy link

😊 Welcome @mzacho! This is either your first contribution to the Istio ztunnel repo, or it's been
a while since you've been here.

You can learn more about the Istio working groups, Code of Conduct, and contribution guidelines
by referring to Contributing to Istio.

Thanks for contributing!

Courtesy of your friendly welcome wagon.

@istio-testing istio-testing added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-ok-to-test labels Jan 26, 2026
@istio-testing
Copy link
Contributor

Hi @mzacho. Thanks for your PR.

I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

use opentelemetry_sdk::logs::SdkLoggerProvider;

let mut builder = LogExporter::builder()
.with_tonic()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A very common use case is going to be wanting to connect to an in-cluster collector endpoint using mesh mtls. using the out-of-the-box client will not support this case.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense. I don't know how to implement it though.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any pointers/ ideas?

//
// Note: This filtering will also drop logs from these components even when
// they are used outside of the OTLP Exporter.
let filter_otel = EnvFilter::new("trace")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally I don't think that we want to just blindly send (even a subset) logs to otel. We probably only want access logs, and we want those to follow otel semantic conventions. This requires specialized calls specifically for otel. I wouldn't be surprised if it ends up not even using tracing at all.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would want the same logs that are currently logged to stdout to be available in OTEL. It seems weird to require that people run a file log exporter in order to get logs.

But I agree with the OTEL semantic conventions.

@Stevenjin8
Copy link
Contributor

/ok-to-test

@istio-testing istio-testing added ok-to-test Set this label allow normal testing to take place for a PR not submitted by an Istio org member. needs-rebase Indicates a PR needs to be rebased before being merged and removed needs-ok-to-test labels Jan 29, 2026
@istio-testing istio-testing removed the needs-rebase Indicates a PR needs to be rebased before being merged label Feb 7, 2026
@mzacho
Copy link
Author

mzacho commented Feb 7, 2026

/retest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-test Set this label allow normal testing to take place for a PR not submitted by an Istio org member. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants