Unable to get updated toggles in Unleash Proxy

[aash-d]

I have self-hosted Unleash Core and Unleash Proxy for both Staging and Production Environments each. The configuration for the services is the same for both environments.

When I create a new toggle or modify an existing one via Unleash Core's UI, I can get the updated configuration via Unleash Proxy URLs in a few seconds on Staging Environment. However, on production, I get the stale list of toggles after making an update on production's Unleash Core UI. The configuration for both the environments has been kept the same as well. It is only when I redeploy Unleash Proxy Instances that I get the updated configuration.

I haven't found any anomaly in log messages as well i.e I didn't get an error or warning messages. Unleash Proxy Instance on production seems to be calling the Unleash Core instance on regular basis. I gather this from the following log messages that get printed routinely. This means that Proxy instance is reguarly calling the core instance to get the updated config.

[36m[2023-08-16T09:37:36.624] [DEBUG] /services/client-metrics/last-seen-service.ts - �[39mUpdating last seen for 3 toggles
-- | --

Can someone please help be debug this to find the core issue? Please let me know if there is any details required from me.

[kwasniew]

I've just tested a self-hosted proxy against our demo instance and it works fine for production envs. Can you test with your proxy connecting to our hosted demo instance (https://app.unleash-hosted.com/demo)?

Here's my video reproducing your case:

Screen.Recording.2023-08-16.at.12.41.53.mov

[aash-d]

I tried the following approaches:

• Local Proxy --> Production Server (DOESN'T WORK)
• Local Proxy --> Local Server (WORKS)
• Local Proxy --> Dev Server (WORKS)
• Dev Proxy --> Production Server (DOESN'T WORK)
• Dev Proxy --> Dev Server (WORKS)
• Prod Proxy --> Prod Server (DOESN'T WORK) (problem)

Based on the above data, it seems that the production unleash server is the problem. But I am not able to go beyond this observation to figure out the root cause.

Apart from this, I tweaked the refreshInterval setting but with no success.

I don't think there are any other network proxies in my infra. But is there a way I can validate this claim?

Unleash proxy calls the server at routine intervals - refreshInterval to get the latest config. Is there a way to see the request and the response made corresponding to this API call? It will give me further insight into what might be happening under the hood.

[kwasniew]

Now that we think that the problem is with the production server I'd connect to the server directly from the Node SDK. Because proxy is a wrapper around Node SDK. Node SDK gives us a listener for changes.

const { initialize } = require('unleash-client');

const client = initialize({
  appName: 'my-application',
  url: 'https://app.unleash-hosted.com/demo/api',
  customHeaders: {
    Authorization: 'DefaultStrategyVariant:production.e31bc7aa5d26bab4edd2089d6ea784f9ea64e7f752fee00260857987',
  },
});

client.on('changed', event => console.log('changed', event));

setInterval(() => {
  console.log(client.isEnabled('TestDefaultVariant'));
}, 1000);

This should emit changed event every time you make a change in your backend. I copy-pasted my demo setup but you'll need to swap your key and URL to your setup.

[aash-d]

Hi @kwasniew ,

Sorry for the late reply.

I tried the debugging process you suggested. And change event is not fired when connected to the production instance. Finally, figured out the issue thanks to you!

Now that we know what is the issue, can you please let me know if there is a way to fix this?

[kwasniew]

So as the next step I'd create 2 brand new self-hosted instances: one with the version matching your failing production version and another one with the very latest Unleash version (we released 5.4 yesterday). Then I'd connect the Node SDK to both of them and check if the changed even is triggered.

[aash-d]

Upgrading the unleash core server's version to the latest - 5.4.1 - fixed the issue. Thanks a lot for helping out.

[kwasniew]

Great! Glad we managed to solve it