🐛 Fix request hanging in periodic concurrency greater than 4

[nv-hwoo]

While testing out the new periodic concurrency mode, @matthewkotila and I have noticed that the 5-th request would consistently hang forever and wait for response from the server.

The issue was that max_threads_ was set default to 4, and if this value was less than the concurrency value, the request created after the max_threads_ number of times (e.g. the 5-th one in our case) will be missing input data. This is because the infer_data_manager that prepares all the input data, uses max_threads_ count to generate the input data and populate the requests with the input data. And I believe this is why the server - when it received the empty 5-th request - did not pass it down to the model.

The fix checks the max_threads_ value against the concurrency range (similar to how regular concurrency mode does). Not quite sure why we have max_threads_ option in the first place, but I feel like we want to eventually move away from using max_threads_ and just deduce the thread numbers from concurrency level.

Thanks @matthewkotila for drilling down and investigating the bug and @Tabrizian & @rmccorm4 for tips and insights.

[matthewkotila]

@Tabrizian @rmccorm4 Is it reasonable to create a feature request for verbose mode on the server logging when a request without inputs is received? That would have helped save so much time debugging this.

[rmccorm4]

This is because the infer_data_manager that prepares all the input data, uses max_threads_ count to generate the input data and populate the requests with the input data. And I believe this is why the server - when it received the empty 5-th request - did not pass it down to the model.

Did you confirm the server is actually receiving the request?

Is it reasonable to create a feature request for verbose mode on the server logging when a request without inputs is received? That would have helped save so much time debugging this.

I believe the server would return an error on reading the request that the inputs didn't match model config. So I suspect the server didn't even get this far.

[nv-hwoo]

@rmccorm4 No this is just my guess of what happened after the request has been sent. Was too tired to go further 😅 Given that the input wasn't properly generated for the request, I think there is a possibility that request didn't even reach the server.