[libshortfin] Initial implementation of LLM inference server.

[stellaraccident]

This is very much a first draft that needs a fair bit of work to turn into a final form. As the first real user of libshortfin's core APIs, a number of rough spots were worked out even to get it to this first stage.

Just a few things that need attention:

• Sampling is just a greedy decode loop right now, but this can be easily extended because it is done per item and is just linear Python (async).
• Scaffolding for multi device in a single process is in place but we're still working on the models to mate up with it.
• A lot of configuration is hardcoded.
• The batcher is a toy. The aim is to implement something like sglang's cache aware batcher and radix attention in its place. Given that, no effort was taken to make the batcher more than write-once code.
• There are many opportunities to use different shortfin fibers, workers and executors to better balance the host-side activity. This currently just runs async on a single worker for the moment.

[monorimet]

I don't have any deep insights on this yet, I've left a few nit comments for typos but you already have noted the areas needing most improvement.

I see some of this could be abstracted out and reused for the next inference server implementation, but some of it, like the batcher, might not have a very substantial base class to carve out (and we might want to resolve some of your bugaboos about current implementation before trying to reuse it).

Anyway, this seems to cover all the points I have in my head, and there's lots for us to iterate on. Thanks.

Some class naming conventions were a bit general, but it seems like an intentional choice, and I see no problem besides a bit of readability -- e.g. InferencePhase, InferenceExecRequest. I doubt this would ever make much of a difference in developer experience and it saves us long ugly class names, but maybe worth mentioning even if it's extremely subjective.

[rsuderman]

Overall this looks good. I noticed a few TODOs which I assume are still WIP but the structure felt right to me. Only high level thought was having some utility / abstraction for data transfer. The direct invocations of the IREE h2d or d2h commands felt slightly out of place (but thats just a nit).

[stellaraccident]

Overall this looks good. I noticed a few TODOs which I assume are still WIP but the structure felt right to me. Only high level thought was having some utility / abstraction for data transfer. The direct invocations of the IREE h2d or d2h commands felt slightly out of place (but thats just a nit).

Yeah, I swallowed my own objections while typing those transfers. To do them right needs more system configuration, which I don't have yet (the way you stage transfers and allocation is system and use case specific), so I just did the least bad thing and wrote it out long hand for now. That's why today, I'm building out the system config layer more -- that is where you root fixes for things like this.