Ambient lighting(SSAO) bringup

[Try]

Since ambient lighting (SSAO in particular) is highly requested effect, I decide to start working on it.
This topic intended to negotiate performance/quality of the effect.

Current implementation is based on a directional occlusion method(https://kayru.org/articles/dssdo/), sampling done in world-space to improve quality.
Feature is optional, can be enabled in menu: Options -> Extended configuration -> Cloud shadows

(note: test is done on GTX960, 1920x1017, window mode, no npc in the world, no lighting pass, no fog pass)

Radius 1m, 16 samples, FPS = 60

IMAO most promising one, works with ~30, when running with everything else

Radius 1m, 32 samples, FPS = 37

Radius 1m, 64 samples, FPS = 22

Radius 0.5m, 16 samples, FPS = 65

Radius 0.5m, 32 samples, FPS = 40

Radius 0.75m, 32 samples, FPS = 38.5

All together. Radius 1m, 16 samples, FPS = 35

[Nindaleth]

Thank you for implementing this, it looks good! Just tested briefly on a 6700 XT, 2560x1440, standing in Xardas's tower door:

• SSAO on = ~100 FPS
• SSAO off = also ~100 FPS
  I measure almost 3 CPU threads fully used, am I bottlenecked by a slow single-thread performance of my CPU? Running Ryzen 1700X.

[Try]

SSAO seems to work with almost zero cost for you - that is nice :)
Basically my main worry is algorithm implementation details - it's supper cache unfriendly. All this 20 frame drop of FPS on my machine is textureFetch calls.

I measure almost 3 CPU threads fully used,

Hm, that is unexpected, since Ryzen 1700X is at least 4x more powerful then my laptop.
Can you please check what are those threads? Thanks!

[Nindaleth]

SSAO seems to work with almost zero cost for you

There's no free SSAO :-)

Can you please check what are those threads?

How can I go about that? I've taken a more detailed look - there are 26 threads including the original parent, 16 of those use about 10-15% of one CPU core each, the rest not much. In total 270-300 % of one CPU core = 3 threads used of the 8C/16T CPU.

I've ran pstack on the running game (= backtrace on all threads), but I don't see much there, almost all threads are in a waiting state at any moment. I'm sure C++ allows you to name the threads so you can see more than just 26x the same binary name in htop output - they use the feature at my work, it could help to identify which thread is tasked with what kind of work.

Here's the full list of backtraces (alsa and pipewire are audio-related): https://pastebin.com/5Yv5YLsh

Seeing that the GPU usage is 60-70% and any given CPU worker uses only a small part of CPU core... can I easily decrease the number of CPU workers? I'd like to make sure it's not the thread synchronization that's limitng the performance.

[Nindaleth]

Found something in game/utils/workers.hpp. Made a quick and very imprecise test with these results:

MAX_THREADS	batchSize constant	FPS
default (16)	default (16)	110
1	1	60
1	32	60
4	4	60
4	16	80
4	32	90
16	4	105
16	128	95
32	32	105

Just going into the pause menu increases the GPU usage to almost 90% and 163 FPS (my configured global FPS limit) or 100% and 185 FPS when unlimited, which still seems to point to some kind of a bottleneck on the CPU side.

EDIT: I also used the opportunity to measure the impact of SSAO in the pause menu. I can't believe this, but disabling SSAO decreases the FPS a little...

[Try]

How can I go about that? I've taken a more detailed look - there are 26 threads including the original parent

Sysprof might me a way to go on Linux(see #188 (comment)).
Profiling only the main thread is also fine, because all the others are parallel-for kind of threads.

185 FPS when unlimited, which still seems to point to some kind of a bottleneck on the CPU side.

Can very-well be a command recording, since there is no other activities, when in menu.

disabling SSAO decreases the FPS a little...

That is why I don't like working with performance...

[Nindaleth]

Can very-well be a command recording

What do you mean by that?

Sysprof might me a way to go on Linux

I don't know Sysprof much, didn't see a choice of which threads to profile. I have a ~25-second recording of me standing still after loading a savegame, how can I give you the information you need? This is the recording starting point:

[Try]

What do you mean by that?

Sorry, I should explain: in Vulkan/Metal/DX12 there is a concept of command buffers, so application records all drawings in one or more command buffers and submits them for execution. Generally this should be cheap, except we have 65k draw-commands (too mush)

I don't know Sysprof much, didn't see a choice of which threads to profile.

Recording only main thread is also fine: all game has is parallel_for kind of threading, we can see it in __sched_yeild. __sched_yeild - is active waiting for 'for' to finish.

This is the recording starting point

Generally is OK (Total column gives some info already), yet three-view is nicer.
Also capture seems to be polluted: calls like ~VBuffer, ~Interactive are related to game shutdown - should not be there

[Nindaleth]

Generally this should be cheap, except we have 65k draw-commands

I see, thanks for the explanation. 65k is probably too much even for Vulkan, right? Is it possible to batch them (or cull some of the drawn stuff)?

Also capture seems to be polluted

I did the recording like this: start the game fullscreen and load the save, then alt-tab, start recording, then alt-tab, stand in-game for a bit, then alt-tab and stop the recording. Every alt-tab from the game I saw swapchain is outdated - reset renderer in log, but I'm not quitting the game as such.

I tried another recording now, started the game windowed, then just clicked over to the sysprof window and did another recording, verified in log no renderer reset is happening, however the ~VBuffer and ~Interactive are still there in the callgraph in basically the same percentage (also tried running without the -v flag).

Tried the same (run windowed, no Vulkan validation) but with Lavapipe Vulkan software rasterizer instead of Radeon RADV - I get about 0.2 FPS that way (that's expected), however there is still the same amount of destructors present in the recording even with this different driver. Is there anything else I can try to prevent spending CPU time on the destructors?

[Try]

65k is probably too much even for Vulkan, right? Is it possible to batch them (or cull some of the drawn stuff)?

Yes, that is too much. According to basemark data craw count should be around 10k for desktop.
Culling is tough: generally we need more scene data or some super-generic algorithm. Batching is fined - just need to be done.

Is there anything else I can try to prevent spending CPU time on the destructors?

I see only one possible root-cause for that: record didn't stop, when you click on stop, but continue up until end of runtime. Note: there is no case when ~Interactive can be called during gameplay.

Anyway, I'm looking forward to optimize call-count and then go from there

[Nindaleth]

I see only one possible root-cause for that: record didn't stop, when you click on stop, but continue up until end of runtime.

I don't know really. I can avoid an interference like that - loaded the game windowed, started recording, stopped recording, copied the recording file somewhere else, only then quit the game - but I can still see ~Interactive in the copied recording for some reason. I added an actual destructor that logs the call and you're right, the destructor is not called before quitting the game, so I don't know, maybe the implementation prepares the destructor somehow beforehand and it's counted in the callgraph... I even tried multiple recordings of one running process, still some ~Interactive present in all of them. 🤷

I might try some other tool instead if that helps. What's the best way to get more detailed debugger metadata compiled into the binary? Setting CMAKE_BUILD_TYPE to Debug or something else?