Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manual tracing for auto-instrumentation-enabled Lambda fuctions #672

Open
nicognaW opened this issue Sep 13, 2023 · 7 comments
Open

Manual tracing for auto-instrumentation-enabled Lambda fuctions #672

nicognaW opened this issue Sep 13, 2023 · 7 comments

Comments

@nicognaW
Copy link

nicognaW commented Sep 13, 2023

I enabled the auto-instrumentation support by following the instructions on ADOT Lambda Support For JavaScript in my TypeScript Lambda functions. The auto-instrumentation is enabled successfully as follow:

image

However, when I manually create spans following instructions from here, the information seems not reported to the CloudWatch X-Ray.

My operations exactly

  1. I added this layer to my Lambda function: arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-nodejs-amd64-ver-1-15-0:2
  2. I enabled the active tracing in my AWS Lambda function
  3. I added some manual instrumentations, code:
    const sdk = new NodeSDK({
      resource: new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: 'echo-handler',
        [SemanticResourceAttributes.SERVICE_VERSION]: '1.0',
      }),
      traceExporter: new OTLPTraceExporter(),
      }),
    });
    
    sdk.start();
    const tracer = api.trace.getTracer('echo-handler-tracer');
    
    module.exports.handler = async (
      event: APIGatewayProxyEvent,
      context: Context,
      callback: Callback<APIGatewayProxyResult>,
    ): Promise<APIGatewayProxyResult> => {
      return tracer.startActiveSpan('echo API handler', async (span) => {
        ... some logics
        span.end();
        return {
          statusCode: 200,
          body: JSON.stringify({
            message: requestBody.message,
            tables: tables,
          }),
        };
      });
    };

Question

I'm unable to find any information about this, is manually instrumenting not supported when the auto-instrumentation is enabled?

@nicognaW
Copy link
Author

This is the code I tried later, still doesn't work:

const _traceExporter = new OTLPTraceExporter();
const _spanProcessor = new BatchSpanProcessor(_traceExporter);
const sdk = new NodeSDK({
  textMapPropagator: new AWSXRayPropagator(),
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'echo-handler',
    [SemanticResourceAttributes.SERVICE_VERSION]: '1.0',
  }),
  instrumentations: [
    new AwsLambdaInstrumentation({
      requestHook: (span, { event, context }) => {
        span.setAttribute('faas.name', context.functionName);
        span.setAttribute('event', event);
      },
      responseHook: (span, { err, res }) => {
        if (err instanceof Error) span.setAttribute('faas.error', err.message);
        if (res) span.setAttribute('faas.res', res);
      },
    }),
    new HttpInstrumentation(),
    new AwsInstrumentation({
      suppressInternalInstrumentation: true,
    }),
  ],
  traceExporter: _traceExporter,
  spanProcessor: _spanProcessor,
  idGenerator: new AWSXRayIdGenerator(),
});

sdk.start();

const tracer = api.trace.getTracer('echo-handler-tracer');

module.exports.handler = async (
  event: APIGatewayProxyEvent,
  context: Context,
  callback: Callback<APIGatewayProxyResult>,
): Promise<APIGatewayProxyResult> => {
  return tracer.startActiveSpan('echo API handler', async (span) => {
    ... some logics
    span.end();
    return {
      statusCode: 200,
      body: JSON.stringify({
        message: requestBody.message,
        tables: tables,
      }),
    };
  });
};

When I turn the activate trace option on, only traces provided by the layer (from my perspective) is exported to the X-Ray, and the traces from the code is not.

@nicognaW
Copy link
Author

I took a log to check if the default URL is correct:

import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const _traceExporter = new OTLPTraceExporter({});
console.log(`using default OTLP TraceExporter URL: ${_traceExporter.getDefaultUrl({})}`);

The log:
image

From ADOT document

Also note that, the OTEL_EXPORTER_OTLP_ENDPOINT environment variable does not need to be set. The default value is http://localhost:4318, as the ADOT Node.JS lambda layer only supports OTLP over HTTP.

This seems to be right.

@ferdy-roz
Copy link

were you able to get this working? I am running into the same issue

@nicognaW
Copy link
Author

nicognaW commented Oct 8, 2023

were you able to get this working? I am running into the same issue

Are you also using the layer from the ADOT document?

@ferdy-roz
Copy link

were you able to get this working? I am running into the same issue

Are you also using the layer from the ADOT document?

Yes -- I was able to get manual instrumentation working for other backends like New Relic, so been working with that. Could not get xray to pick up any of my spans/attributes

@nicognaW
Copy link
Author

I turned to use PowerTools instead of the ADOT layer FYI @ferdy-roz

@gshpychka
Copy link

Could it be because you're not awaiting your async callback?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants