Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for OpenTelemetry logs with HTTP JSON/binary format #5562

Open
spetz opened this issue Nov 23, 2024 · 0 comments
Open

Support for OpenTelemetry logs with HTTP JSON/binary format #5562

spetz opened this issue Nov 23, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@spetz
Copy link

spetz commented Nov 23, 2024

Is your feature request related to a problem? Please describe.
The Quickwit only supports gRPC transport to send logs. The docs stated that this would be an easy feature to add :)

Describe the solution you'd like
It would be great to also see the support for HTTP JSON and/or binary (protobuf) format (so it can be easily used with e.g. opentelemetry_sdk in Rust).

Describe alternatives you've considered
No alternatives.

Additional context
What I've been experimenting with recently, is a way of having an additional proxy (such as HTTP API) solely responsible for the logs ingestion. Imagine the following, multi-tenant scenario:

  • Some process which produces the logs is deployed across the multiple distinct consumers (tenants).
  • The logs should be collected in a single place (Quickwit server), but each tenant, should have its own, fully isolated index compatible with Otel logs specification (thus ingesting all the logs into existing /v1/logs endpoint is not enough).
  • There's an intermediary HTTP API proxy (custom application) which receives the logs from the mentioned process.
  • The custom proxy sends further the logs to the Quickwit API depending on their source (unique tenant index).

My understanding is, that it could be only achieved by creating the index per tenant via API in the first place, and then invoking the ingest endpoint. Ideally, when receiving the Otel compatible logs by the custom proxy (in binary format, for example), it could be as simple as just invoking QW ingest API and passing the raw bytes to it (similar to having gRPC connection to /v1/logs as it's working currently).

@spetz spetz added the enhancement New feature or request label Nov 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant