Custom LLM integration with Voice Agent

[satyam-veris]

I'm trying to integrate my custom LLM with Deepgram Voice Agent. When using their default providers (OpenAI/Anthropic), everything works fine. With OpenAI, I see this response format in the WebSocket messages:

{
    "type": "ConversationText",
    "role": "assistant", 
    "content": "How can I help you?"
}

I've set up my custom LLM REST endpoint at /chat in my localhost and exposed it publicly using NGROK. and configured the agent:

{
  "think": {
    "provider": {
      "type": "custom",
      "url": "my-server/v1/chat/completions"
    }
  }
}

My LLM receives and processes requests successfully (confirmed via logs), but the responses never appear in the WebSocket messages. What's the correct response format for custom LLM integration?

My server.js file looks like

app.post('/chat', async (req, res) => {
  try {
    const { messages, thread_id = 'default' } = req.body;
    const lastMessage = messages[messages.length - 1];
    const response = await callAgent(mongoClient, lastMessage.content, thread_id);
    console.log('response', response);
    // Format response for Deepgram
    const finalResponse = {
      id: `chatcmpl-${Date.now()}`,
      object: 'chat.completion',
      created: Math.floor(Date.now() / 1000),
      model: "custom-langgraph-agent",
      usage: {
        prompt_tokens: 0,
        completion_tokens: 0,
        total_tokens: 0
      },
      choices: [{
        message: {
          role: 'assistant',
          content: response,
          type: 'ConversationText',
        },
        finish_reason: "stop",
        index: 0
      }]
    };

    res.json(finalResponse);

  } catch (error) {
    console.error('Error:', error);
    res.status(500).json({ error: { message: 'Internal server error' } });
  }
});

Sample config for deepgram Agent:

const baseConfig = {
  type: "SettingsConfiguration",
  audio: audioConfig,
  agent: {
    listen: { model: "nova-2" },
    speak: { model: "aura-asteria-en" },
    think: {
      provider: {
        type: "custom",
        url: '<server_url>/chat',
        headers: [
          { key: 'Content-Type', value: 'application/json' }
        ],
      },
      model: "custom-langgraph-agent",
      // provider: { type: 'open_ai' },
      // model: 'gpt-4o-mini'
    },
  },
};

I am using this repository as a interface https://github.com/deepgram-devs/deepgram-voice-agent-demo.

Environment:

• Custom LLM running on Express server
• Using ngrok for testing
• Deepgram Voice Agent in browser

[deepgram-community[bot]]

Thanks for asking your question. Please be sure to reply with as much detail as possible so the community can assist you efficiently.
_{Consider joining our Discord community for more opportunity to engage with your fellow Deepgram users. You can earn points which can be redeemed for cool stuff by being active in our communities!}

[deepgram-community[bot]]

Hey there! It looks like you haven't connected your GitHub account to your Deepgram account. You can do this at https://community.deepgram.com - being verified through this process will allow our team to help you in a much more streamlined fashion.

[deepgram-community[bot]]

It looks like we're missing some important information to help debug your issue. Would you mind providing us with the following details in a reply?

• A request ID that triggered your error or issue.