Investigate raylib and others for ui

[adeebshihadeh]

openpilot's UI used to be written in https://github.com/memononen/nanovg, and in retrospect that it was nicer to work in than . Qt is nice for cabana, but it turns out to not add much value to the openpilot ui and adds a bunch of conceptual and dependency complexity.

Some options

• https://www.raylib.com/
• https://github.com/Immediate-Mode-UI/Nuklear
• https://github.com/raysan5/raygui

We're looking for:

• cross platform: Linux, macOS, etc.
• small, simple. single header is ideal
• Python bindings? would be great to have all or most of the UI written in Python

The onroad camera view is a good test for trying out these libraries.

[adeebshihadeh]

@deanlee want to try to make a small demo in raylib of the onroad camera view?

[deanlee]

I tried writing a simple demo based on Raylib to display the road camera (without shaders and transforms, just to quickly get a feel for Raylib's basic features).
Raylib can simplify some OpenGL code, and its text rendering performance is also somewhat better (currently, in the drawHud function of the Qt UI, the three drawText function calls consumes about 3% of the CPU).
Raylib might be a good option. The real challenge is the workload of migrating the entire UI to Raylib, including rewriting network management, WiFi manager, and multilingual support, among other features. Additionally, Raylib itself lacks a signal-slot mechanism. All logic is driven by a while loop, and input/output would also need to be restructured, making the workload quite significant. Considering that the current UI is relatively stable, if the goal is just to save space, it might be worth first optimizing other dependencies that can be relatively easily removed before eventually migrating to Raylib, RayUI, or another library.

Although it's possible to use Raylib within a Qt widget to render graphics, this actually increases complexity and maintenance difficulty. There's no good way to gradually transition to Raylib and remove the Qt dependency through multiple PRs. It looks like a complete rewrite would be necessary.

[adeebshihadeh]

Qt was a big improvement over the old APKs, and I expect this to be a similar improvement. Smaller dependency, simpler library, better docs, etc. I also expect it to help with #30294.

I think we do have to find some way to do this incrementally; like you said, it's too much to do all at once. If you can't figure out a way to do it, I can start it. Want to put up a draft PR with the demo you wrote?

[deanlee]

I'll commit a draft after adding the shader, transformations, and other features to ensure it matches the CameraView's output.

[AskAlice]

fyne is quite nice although it's written in golang which would be a bit of a hurdle, though it wouldn't be the first time the UI were separate from the py.
I think it also might be worth considering from this list https://github.com/sudhakar3697/awesome-electron-alternatives#python

[deanlee]

PR #33326 provides a draft on using Raylib to display the road camera, with automatic scaling to fit the window while keeping the aspect ratio.

[deanlee]

First Step: Migrate Onroad Widgets to Raylib
1.Organize the existing code by encapsulating related functions into specific classes to improve clarity and maintainability, The new class will expose only two public functions: updateState() and draw(), facilitating a smoother transition to Raylib.

• ui: refactor CameraView to serve as a generic vision stream display class #33457
• ui: show driver camera in popup on demand #33459
• ui: refactor driver monitor updating and rendering into DriverMonitorRenderer class #33375
• ui: refactor hud updating and rendering into HudRenderer class  #33458
• ui: refactor model updating and rendering into ModelRenderer class #33702
• WIP: refactor CameraView to single-threaded implementation #33773

After step 1, the AnnotatedCameraWidget will function as a container for various components such as HudRenderer, AlertRenderer, DriverMonitorRenderer, ModelRenderer, and others. Its responsibilities will include updating transformations, managing window size, and handling user input. It will not handle state updates or rendering directly; instead, these tasks will be delegated to the drawing functions of its contained classes.

Following Step 1, we can also remove the UIScene struct from uiState to streamline it, allowing uiState to hold only the most essential state information required by the entire application.

2. Migrate to raylib

• ui: refactor spinner using raylib #33738 (Introduced a new raylib/util.cc file containing utility functions to manage common tasks such as font handling, application initialization, and other reusable operations.)
• refactor selfdrive/ui/qt/text using raylib. (Implement Raylib-based ScrollArea in this step, or use the raygui lib)
• ui: refactor installer using raylib #33756
• WIP: remove QtDBus dependency, use pure C++ and libdbus #33808
• Replace QtNetwork with libcurl
• refactor selfdrive/ui/qt/setup/updater using raylib. (implement Raylib-based Wifi manager)
• Port the rendering functions of the newly organized classes from Qt to raylib, ensuring better performance and seamless integration with raylib's rendering capabilities.

Tools:

• cabana: implement custom camera widget and remove ui/cameraview dependency #33387

Misc.

• ci: add cameras to ui reports #33401
• ci: add all onroad hud elements to ui report #33395
• ci: add onroad alerts to ui report #33394
• ci: add driver camera to ui report #33409
• ci: display lead car indicator in on-road screenshots #33416
• ci: integrate offroad alerts and update notifications to ui report #33408
• ui: remove QtConcurrent dependency #33389

The final directory structure might look like this:

alerts.cc – Handles onroad alerts.
dmon.cc – driver monitor updates and rendering.
hud.cc – HUD updates and rendering.
model.cc – model updates and rendering.
annotated_camera.cc – container for update and rending onroad window.
onroad_home.cc – container for the Raylib window.
raylib_helpers.cc – helper functions for tasks such as managing fonts and drawing text.

Here's a rough draft, written from scratch (interface issues can be addressed later). Although there is a lot of temporary code and commented-out sections with no cleanup yet, approximately 600 lines of onroad code have already been reduced.