Silhouettes consume high amounts of RAM

[adroitwhiz]

Expected Behavior

Scratch should not consume excessive RAM, in order to be usable on devices with low amounts of it.

Actual Behavior

Because of the way GPUs' pipelines work, reading data back from the GPU is slow, which could potentially bottleneck blocks like "touching sprite" and "touching color" which require reading and processing color data.

To mitigate this, scratch-render implements a software rendering pipeline, which does not require reading any data back from the GPU. This software renderer stores texture data in "silhouettes".

Unfortunately, this means that the more "silhouettes" there are, the more RAM will be used.
Uncompressed texture data takes up 4 bytes per pixel, so projects with many large costumes will consume high amounts of RAM.

One instance of this can be seen with the "Bad Apple Animation" project, as described in this issue. The reproduction steps have changed slightly-- the project no longer crashes while loading, but watching the animation quickly causes RAM usage to skyrocket. This is somewhat of a pathological case-- there are 730 costumes, each with a resolution of 1537x1153. This means 6.76 MiB of texture data per costume, for a total of nearly 5 GiB across all costumes. Other projects like Seagulls consume a few hundred MiB in my testing-- likely higher in full-screen mode or on high-DPI screens.

I can think of a few different ways to reduce the amount of RAM used by silhouettes:

• For vector skins, don't grab the silhouette's texture data from the SVG renderer until it's actually needed. This was attempted in Implement updateSilhouette to allow updates to happen when needed #394, but due to certain places in the codebase not properly calling updateSilhouette to indicate that they needed the texture data, it caused bugs and was reverted in Revert "Merge pull request #394 from paulkaplan/defer-silhouette-upda… #398. Once Handle matrix + silhouette updates better #555 is merged, that issue will be fixed and such a change can be implemented again.
• For other types of skins, the texture data could be read back from the GPU when needed (by rendering the skin to a framebuffer and reading that). This will be slower than passing the texture data to the silhouette as soon as the skin receives it, but the RAM reduction may be worth it.
• Silhouettes' texture data could be automatically disposed of if unused for a certain amount of time. Tuning the heuristics used for determining when this data is discarded could quickly become a huge time sink, however.
• Certain functions, like pick, could be updated to work on both the CPU and GPU so as to not require silhouette texture data to exist. This could, however, cause mysterious bugs due to differences between the CPU renderer and GPU renderer-- such a class of bugs already exists in the codebase and is causing quite a lot of trouble.
  • Alternatively, some functions could be rewritten to only use the GPU. While reading data back from the GPU does slow things down, functions like pick appear to only be called when a mouse click event is fired, which is likely not often enough to make the slowdown noticeable. For such functions, the performance penalty from reading GPU data may be preferred to the RAM penalty of keeping that data for the CPU rendering pipeline.

Steps to Reproduce

In Chrome:

1. Open a project, preferably in a development build of the GUI so that source maps are present and you can see non-minified function names. Typically, projects with many costumes (e.g. animations) exhibit this RAM usage pattern. I recommend Bad Apple, or Seagulls if you don't have 5 GB of RAM to spare.
2. If the project is an animation, make sure to watch it all the way through to ensure the silhouettes get created. If it's not an animation, poke around and try to view as many costumes as you can.
3. Open the dev tools.
4. Go to the Memory tab.
5. Select "Heap snapshot".
6. Select the JavaScript VM instance that corresponds to the Scratch GUI instance where the project was run. Note that it may only indicate that a very small amount of RAM is being used-- it is wrong.
7. Click "Take snapshot", and be very patient. It can take a few minutes to load, and the "Building dominator tree" step in particular takes a long time.
   
8. Observe the amount of RAM used by Silhouette objects:
   

Operating System and Browser

All

[fsih]

/cc @cwillisf @fsih

[Joeclinton1]

Perhaps you could consider storing the silhouette as a quadtree. Quad trees are an efficient method of compression which unlike many other compression techniques still allow you to perform a pixelColour lookup operation in sublinear time. (specifically O(log(n)) ).

This method would greatly reduce ram usage for costumes with large amounts of solid colour, or large amounts of transparent pixels.

[Joeclinton1]

@adroitwhiz The first method of just not creating the silhouette array until it's needed (by touching colour) would be quite effective. The projects on Scratch which use lots of large costumes use them mostly to display frame by frame videos, for art, or for backgrounds. In most cases, these large costume sprites will not need the touching colour functionality, so if by default the silhouette was 0 and only upon calling touching colour was it created, the silhouette would never need to be made.

This would greatly reduce the time to load and the RAM used for all projects and especially projects with large costumes. The only downside is it would lag slightly on the first touching colour, but a scratcher knowledgeable of how the underlying code works, could mitigate the lag by just calling touching colour once at the start, to preload the silhouette. However I expect in 99% of use cases the lag would be unnoticable.

I think this solution would only require changing 3 or 4 lines of code. So I think it could be quite simple, though I am aware that there is alot I don't know about how the renderer works.

Unlike my suggestion about cropping transparent pixels, this method would improve all projects. And since games with artwork which don't need touching colour, are quite common, I think this is something which would be worthwhile.

I hope you consider this suggestion. :)

[cwillisf]

@Joeclinton1, I like these ideas! It sounds like there's a lot of alignment between your suggestion and some of @adroitwhiz's earlier comments, especially the first bullet point in the initial post on this issue.

As a non-profit organization, the Scratch team always runs a little lean. Right now we're stretched extra thin, though, so unfortunately I'm not likely to be able to implement this kind of improvement soon. If you feel strongly about optimizing silhouettes, one way to improve the chances of improvement (heh) is to put up a pull request with an implementation and show in the PR that it has significant positive impact without negative impact. We're extremely risk-averse when it comes to the potential for compatibility problems; we've been bitten quite a few times by optimizations that missed an edge case, including some that came from me or others within the Scratch team. In this particular case, I recommend checking out #394 / #398 and #555 as @adroitwhiz mentioned.

I understand if you don't have time for all of that... in that case, I'm sure we'll get to this "someday" but I can't guarantee when. Sorry!

[Joeclinton1]

@cwillisf Yes, this idea is really just adroitwhiz's first bullet point. I read over it a second time today, and realized that it was quite a good solution!

I'll try and give it a go myself and will be sure to take into account the linked issues. Currently the silhouette size is problematic to my next project, so I'm quite incentivized to improve it ;)

I'll test all the cases that are discussed in the linked issues. Tests should be done in the playground right? Are there any automated tests I can run? Also is it possible for tests run in development to not be indicative of tests run in production?

[Joeclinton1]

Any update on this? I didn't find time last year to work on this, but I might consider doing it this year. I didn't receive any answer to the question about the best way to do the tests, so I'll just assume that anything is fine.

[GarboMuffin]

It's been mentioned elsewhere that a Scratch mod I work on has some patches to address this issue:

• TurboWarp@2d0f525
• TurboWarp@0cf0454
• TurboWarp@b37f83d
• TurboWarp@35cefa9
• TurboWarp@85c67bf

I'm not suggesting these are completely bug-free, but they may be a good starting point (and it's been a while since we've identified it to be the cause of any bugs). To ensure that license ambiguity is not the thing that prevents this from getting merged, I release the linked patches as public domain or CC0, whichever you prefer.

[adroitwhiz]

I wouldn't spend any time working on pull requests for the Scratch editor right now--it's my understanding that there isn't anyone on the Scratch team who's assigned to even work on the editor themselves, much less review work from others, and that's been the case for a few years now with no signs of change.

[griffpatch]

Sadness... :(

…

On Thu, 31 Aug 2023 at 17:51, adroitwhiz ***@***.***> wrote: I wouldn't spend any time working on pull requests for the Scratch editor right now--it's my understanding that there isn't anyone on the Scratch team who's assigned to even work on the editor themselves, much less review work from others, and that's been the case for a few years now with no signs of change. — Reply to this email directly, view it on GitHub <#584 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTM3PXXMZPMDJ72CE3CGILXYC6JXANCNFSM4MFZQ3BQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[cwillisf]

I'm still here! Unfortunately, @adroitwhiz is effectively correct, even if one or two of the facts aren't strictly accurate. I'm incredibly grateful to have all of you in our community, and I truly wish I could accept your help more quickly and more frequently. As you've noticed, it's been tough for a while now... even more so since mid-August.

I can't always share information about our internal priorities, but if you'd like to keep tabs on what I'm working on within the Scratch Foundation, you can take a look at the bottom of this page: https://github.com/cwillisf?org=scratchfoundation. Some of my work is in private repositories, but most of my work is public. Even with private work, the goal is usually for the work to become public in the future.

[aixgeek]

yes，how to solve it？