Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Small patch reflecting some recent changes in `sf.Program` and `sf.ProgramFunction`. Was originally included as part of this PR, which adds an integration test to shortfin llm serving: #373 But, parsing it out, since that may take a little more time to make adjustments/add workflow file. Without it, you get the following error when trying to launch the server: ```text [2024-10-30 11:59:09.939] [info] [manager.py:40] System manager command processor stopped [2024-10-30 11:59:09.991] [error] [on.py:121] Traceback (most recent call last): File "/home/amd/stephen/repos/forks/SHARK-Platform/.venv/lib/python3.12/site-packages/starlette/routing.py", line 693, in lifespan async with self.lifespan_context(app) as maybe_state: File "/home/amd/.pyenv/versions/3.12.5/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/amd/stephen/repos/forks/SHARK-Platform/.venv/lib/python3.12/site-packages/shortfin_apps/llm/server.py", line 42, in lifespan service.start() File "/home/amd/stephen/repos/forks/SHARK-Platform/.venv/lib/python3.12/site-packages/shortfin_apps/llm/components/service.py", line 69, in start self.inference_program = sf.Program( ^^^^^^^^^^^ TypeError: __new__(): incompatible function arguments. The following argument types are supported: 1. __new__(cls: object, modules: collections.abc.Sequence[_shortfin_default.lib.local.ProgramModule], *, devices: collections.abc.Sequence[_shortfin_default.lib.local.Device], trace_execution: bool = False, isolation: _shortfin_default.lib.local.ProgramIsolation = ProgramIsolation.PER_FIBER) -> _shortfin_default.lib.local.Program Invoked with types: nanobind.nb_type_0, kwargs = { modules: list, fiber: _shortfin_default.lib.local.Fiber, trace_execution: bool } [2024-10-30 11:59:09.991] [error] [on.py:59] Application startup failed. Exiting. ``` With it, you're able to start server, send requests, and receive responses.
- Loading branch information