Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal for Log and Event capturing with auto instrumentation #4329

Open
aabmass opened this issue Nov 27, 2024 · 10 comments · May be fixed by #4340
Open

Proposal for Log and Event capturing with auto instrumentation #4329

aabmass opened this issue Nov 27, 2024 · 10 comments · May be fixed by #4340
Assignees
Labels
auto-instrumentation related to auto-instrumentation of the sdk logging sdk Affects the SDK package.

Comments

@aabmass
Copy link
Member

aabmass commented Nov 27, 2024

Maybe rehashing #4269 a little bit here.

Current behavior

The current behavior of auto instrumentation is that Logs and Events are NOOP by default. That means any logs created through the Events API will be dropped.

However, if you set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true code, then we:

  1. set up real global LoggerProvider, EventLoggerProvider
  2. set the OTel LoggingHandler on Python's root logger

Number 2 causes python's builtin logging logs to be captured only in the logging SDK. If they were previously being printed, they no longer will be.

Proposal

By default:

We could also consider teeing logs so they are both captured in the SDK and get printed out.

@aabmass aabmass added sdk Affects the SDK package. auto-instrumentation related to auto-instrumentation of the sdk labels Nov 27, 2024
@xrmx
Copy link
Contributor

xrmx commented Nov 28, 2024

I'm in favor of moving the setup of LoggerProvider and EventLoggerProvider out of OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED guard.

@lzchen
Copy link
Contributor

lzchen commented Dec 2, 2024

One thing of note is that users have to explicitly set what exporter is needed when using autoinstrumentation (or elses otlp/grpc will be used by default) -> opentelemetry-instrument --traces_exporter console --metrics_exporter None --logs_exporter None python main.py.

Those not interested in logging before will now have to explicitly set this (instead of OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED ). Also I've always been a little in favor of making the default behavior console but this might be too big of a breaking change for traces and metrics if we do alltogether.

@aabmass
Copy link
Member Author

aabmass commented Dec 2, 2024

Those not interested in logging before will now have to explicitly set this (instead of OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED ).

I didn't follow your concern @lzchen. This proposal shouldn't have any affect on users who don't want OTel to mess with stdlib logging.

@lzchen
Copy link
Contributor

lzchen commented Dec 2, 2024

If we enable setting up events/log by default (without relying on OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED, users will now need to explicitly set logs_exporter to None if they are using autoinstrumentation and NOT interested in logging (because by default, an otlp pipeline is setup for each signal if nothing is set for those parameters) otherwise they would get erroneous collector endpoint missing exceptions. Before when users don't have the env var set, it defaults to not setting up any logging pipeline at all.

Unless if I am misunderstanding @xrmx 's message.

@aabmass
Copy link
Member Author

aabmass commented Dec 2, 2024

If we enable setting up events/log by default (without relying on OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED, users will now need to explicitly set logs_exporter to None if they are using autoinstrumentation and NOT interested in logging (because by default, an otlp pipeline is setup for each signal if nothing is set for those parameters) otherwise they would get erroneous collector endpoint missing exceptions.

What do you mean by not interesting in logging here?

The only thing that would be sending logs would be Event API instrumentation. In which case, any collector endpoint missing issues are not really erroneous

@lzchen
Copy link
Contributor

lzchen commented Dec 3, 2024

What do you mean by not interesting in logging here?

If I want to use auto instrumentation but I don't want logging/metrics telemetry, perhaps due to cost reasons.

@xrmx
Copy link
Contributor

xrmx commented Dec 4, 2024

One thing of note is that users have to explicitly set what exporter is needed when using autoinstrumentation (or elses otlp/grpc will be used by default) -> opentelemetry-instrument --traces_exporter console --metrics_exporter None --logs_exporter None python main.py.

I think this may already be the case since we changed the default for OTEL_LOGS_EXPORTER to otlp in the distro.

@lzchen
Copy link
Contributor

lzchen commented Dec 4, 2024

I think this may already be the case since we changed the default for OTEL_LOGS_EXPORTER to otlp in the distro.

I don't think this will happen if users don't explicitly set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED

@xrmx
Copy link
Contributor

xrmx commented Dec 5, 2024

I think this may already be the case since we changed the default for OTEL_LOGS_EXPORTER to otlp in the distro.

I don't think this will happen if users don't explicitly set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED

I think I hit this by chance yesterday when I tested a git checkout and I didn't have that environment variable set. AFAIR opentelemetry-instrument looks at the env vars set and doesn't care if things are enabled or not.

@xrmx xrmx self-assigned this Dec 5, 2024
@lzchen lzchen added the logging label Dec 5, 2024
@xrmx
Copy link
Contributor

xrmx commented Dec 5, 2024

Tested this again and confirm behavior already changed:

OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=false opentelemetry-instrument --traces_exporter=none --metrics_exporter=none python3 app.py
...
    raise RuntimeError(
RuntimeError: Requested component 'otlp_proto_grpc' not found in entry point 'opentelemetry_logs_exporter'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-instrumentation related to auto-instrumentation of the sdk logging sdk Affects the SDK package.
Projects
None yet
3 participants