[Issue] Not sending a User-Agent on your Deepgram Client

[imdadahad]

Earlier, one of my engineers notified me of a blocker they experienced while attempting to integrate Deepgram. To be clear, we are a paid customer.

Specifically, when providing a URL to the audio file in the Deepgram request, we discovered that you are not sending this request with a User-Agent.

This is unsafe and, as we've experienced, causes our AWS firewall to block such requests. An HTTP request without a user agent is similar to suspicious web scraping requests that are typically filtered out by firewalls. By not sending this, you are indicating that your service is incompatible and insecure for use with AWS WAF. I suspect Cloudflare would also block such requests, though I haven't verified this myself.

[deepgram-community[bot]]

Hey there! It looks like you haven't connected your GitHub account to your Deepgram account. You can do this at https://community.deepgram.com - being verified through this process will allow our team to help you in a much more streamlined fashion.

[deepgram-community[bot]]

It looks like we're missing some important information to help debug your issue. Would you mind providing us with the following details in a reply?

• The programming language you are working in (e.g. JavaScript, Python).
• The deepgram product you are using (e.g Speech to Text, Agent API)
• A request ID that triggered your error or issue.

[SandraRodgers]

Hi @imdadahad

Our SDKs allow you to set a custom header, so if you are able to use one of Deepgram's SDKs, you could try adding a header like this:

global: {
      fetch: { options: { headers: { "custom_header": "header_value" } } },
    },

Here's the documentation for the JS SDK, but this is also something that can be done in our Go, .NET, and Python SDKs

[SandraRodgers]

@imdadahad It has come to my attention that perhaps you are trying to validate the callback response? If so, you could compare the request_id returned to your server to the request_id in the transcription response received at the callback. By comparing the request_id in the callback request to the request_id returned in the original request, you can ensure that the callback request is legitimate.

See documentation at https://developers.deepgram.com/docs/callback#authenticating-callback-requests

[imdadahad]

Hi @SandraRodgers

We are using the client.listen.prerecorded.transcribeUrlCallback method, which does not accept custom headers. However, the client does support custom headers. We have attempted to incorporate this but have encountered issues with its implementation.

Here is a brief example:

const { createClient, CallbackUrl } = require("@deepgram/sdk")

const apiKey = '...'
const deepgram = createClient(apiKey, {
    global: {
        headers: {
            "User-Agent": "Testing..."
        }
    }
})

async function transcribe() {
    try {
        const url = 'https://api-url/audio.wav'
        const response = await deepgram.listen.prerecorded.transcribeUrlCallback(
            {
                url: url
            },
            new CallbackUrl('https://api-url/api/webhook/deepgram'),
            {
                model: "nova-2"
            }
        );
...

Once this request is initiated, our API receives it but the custom User-Agent is missing. The following are given:

{
  "host": "...",
  "accept": "*/*",
  "dg-token": "...",
  "x-forwarded-for": "...",
  "x-forwarded-host": "...",
  "x-forwarded-proto": "https",
  "accept-encoding": "gzip"
}

We are using version 3.9.0 of @deepgram/sdk

[SandraRodgers]

Are you doing this to validate the callback response? If so, please check out our documentation on Authenticating Callback Requests. To validate a specific request, you can look at the request id (or optionally use the basic-auth mechanism detailed there). But to add a rule for your WAF, I would recommend looking for the dg-token value which is set to your API Key accessor ID which you can see from a request to get-keys.

[imdadahad]

Not to validate the callback response.

AWS WAF flat out will block any request without a User-Agent header, as such requests are usually deemed as being from bots and scrapers. Is there any reason why Deepgram do not include this?

Additionally, why does the custom headers option not work in the SDK? As you can see I've added in the above code but it does not come through..

[SandraRodgers]

Thanks for that clarification. It seems there was a similar question from a customer related to validating the callback response, and I thought maybe it was from your blocked engineer.

I believe we've run into the problem with the custom headers not working due to this being a request to transcribe a remote file. When a user submits a request to Deepgram that instructs us to fetch an audio resource from a URL, Deepgram's server must issue its own HTTP request to the specified URL to fetch that audio resource. Apparently, the custom header is not sent with that request.

I'm currently working on getting answers from the engineering team about how to fix this. I will let you know as soon as I have answers, which will probably be tomorrow since the workday is coming to the end for many of our engineers. Thank you for your patience!

[imdadahad]

That makes sense, I did assume that the request was being sent by Deepgram's server. My guess is that it's being stripped somewhere in transmission due to some firewall filtering rules. In any case, your engineers will know best.

Okay sure. It's definitely one they should probably prioritise (on a security level) as it means customers who have firewalls won't be able to send these URLs for transcription.

[SandraRodgers]

Have you looked into updating your ruleset? https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-baseline.html

It's definitely inconvenient that the default configuration won't work for receiving callbacks, but updating the ruleset is probably the route you could take to quickly unblock your engineers.

[imdadahad]

Hmm, we can have a look, but I'd be wary of making wider firewall ruleset changes to cater for a third-party integration. It may compromise our security by now allowing requests that would otherwise typically be blocked.

I guess if there's the option to define scope, such that it's restricted to only requests from Deepgram? We'd probably have to take a closer look..

[SandraRodgers]

From what I understand, your remote-audio is behind an AWS WAF managed by the core ruleset which will filter out anything without a User-Agent. Unfortunately, at this time, we currently do not have an ability to send custom headers to a remote server we are fetching audio from. And I can't give you a firm answer about Deepgram adding a User-Agent header because that is something the engineers would have to investigate at a later date.

As of now, I would suggest one of two options:

1. Update your ruleset in AWS
2. Use the dg-token header method to validate requests to the remote audio source. You can add a rule to your AWS WAF to allow all traffic containing the dg-token header, with your accessor ID as its value. This approach ensures that requests from Deepgram can successfully access your audio, bypassing the filtering restrictions without requiring custom headers or additional SDK modifications.

I hope this helps!