[Feature] Add support for streaming LLM output

[AlessandroSpallina]

Is your feature request related to a problem? Please describe.
As a user who wants to chat with the Cat using self-hosted LLMs I would like to have a smoother experience than the one I have now, self-hosted LLMs are generally slower than the one hosted with a premium subscription, waiting tens of seconds before starting to read the first word is killing the user experience and the user's patience. According to my inference timing tests, on my hardware one single LLM inference takes 8-19 seconds with llama 2 13b (4 to 8-bit quantization), as a person who is chatting with a bot, I would prefer to start reading the output the LLM is providing instead of waiting 19 seconds and then have the full block of text.

Describe the solution you'd like
Enable support for output stream, LangChain already supports this option.

Describe alternatives you've considered
An alternative would be to use lighter models, but this will affect the LLM performances and I don't want it.

[pieroit]

Problems:
1 - the Cat contains an agent, so every reply may contain between 1 and 3 different generations. Which one would you stream and how?
2 - not all LLMs and their APIs support streaming

[AlessandroSpallina]

1 - let’s only stream to the frontend the last output, i.e. the one the user will read.
2 - if streaming is not supported, in that case only, the user will see directly the block of text; streaming otherwise

[KevinZhang19870314]

Take a look for this project, it based on LangChain & FastAPI, maybe can help you to get some ideas.

streaming_mode – The streaming mode to use for the route.

[pieroit]

@AlessandroSpallina @KevinZhang19870314 I'm reading some code. Let's see how much of a refactor this could be

[nicola-corbellini]

This was solved with #492