etc2comp texture import is extremely slow

[jesperkondrup]

bugsquadedit: it's slow due to etc2comp, it needs to be disabled if you are not doing mobile, There probably should be an option for this when you create the project, though if we make it happen on background it might not be too bad.

---

Operating system or device - Godot version:
Windows 10, Godot v. 3.0 alpha build 1

Issue description:

it takes a very long time to import textures into Godot. This is especially true for large textures, 2k and up.
When we imported 5 textures in 4k resolution into our project, it took around 4-5 minutes to import them. Compared to Unity, Unreal engine and other 3d programs this is a very long time.

Steps to reproduce:
Import a few textures that's at least 2048x2048 in size.

Link to minimal example project:

[Keetz]

Tested on Ubuntu 16.04, Godot_v3.0-alpha1

Importing 6 6k textures, it takes a while, but quicker than the 4-5 minutes.

Also noticed both on Windows and Ubuntu, after importing a texture, it gets reimorted when applied to a model.

[27thLiz]

@Keetz the reimport happens because by default it imports Textures in 2D mode. But when Godot notices that it's used in 3D, it reimports with suitable 3D settings.

[reduz]

it's slow due to etc2comp, it needs to be disabled if you are not doing mobile. It gets reimported when applied to a model because it will recompress it to video mem (by default it just uses regular png compression when importing)

…

On Thu, Aug 3, 2017 at 6:21 AM, Andreas Haas ***@***.***> wrote: @Keetz <https://github.com/keetz> the reimport happens because by default it imports Textures in 2D mode. But when Godot notices that it's used in 3D, it reimports with suitable 3D settings. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10059 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF-Z26PJetGnaFezCqiSBjwnBMXw6YrNks5sUZEWgaJpZM4OsCSN> .

[27thLiz]

mhh, could we defer the etc2comp stuff until it's needed? (e.g exporting to mobile / debug on device)

[reduz]

@Hinsbart No, that would be pretty complicated, and you would hate it to have to wait 2 hours to test your project on a mobile device.

Eventually I think the best solution would be to move the reimporting of textures to a background thread.

[Keetz]

@reduz how is etc2comp disabled?

Why do the etc2comp happen as default? Just a design choice or?
I see the practicality of this if you develop for mobile, but if not, it just becomes a real slowdown and annoyance, but this isn't a problem if it is easily disabled of course.

It seems more intuitive to import quickly, and then let the user decide how to compress if needed.

[ghost]

Is it possible to make importing of textures to run in the background? My biggest problem with new Godot importer is that whole engine becomes unresponsive when importing. Could it be possible to replace (re)imported textures temporally with some replacement texture until the new one is loaded? Maybe checkerboard one or some with importing written all over it?

[reduz]

@Keetz It probably should be an option for when you create the project, though if we make it happen on background it might not be too bad.

@MrMaidx
the initial import is usually fast because it uses the existing file as base compression (or should be), it's the reimport into vram that is slow. And yes, the idea is that it should run on a thread.. but one thing at a time :D. Need to make a stable release first, then will work on that

[ghost]

@reduz Thank you for a quick answer. I'm glad that it's planned. :)

[Keetz]

@reduz Making it happen in background sounds good yea.

If by setting an option at the project creation means it can't be changed in the project later on i would say it's a no go. That would make it a big hurdle to export your game to multiple platforms.

[jesperkondrup]

For me as an artist it makes sense to have textures import as fast as possible, so I can try them on my shaders right away and then later make changes to import settings.
In Unity I just tried importing 7 images in 6k resolution and it took around 15 seconds. Same textures in Godot took around 10 minutes to import (at least on my Windows 10 machine).

[reduz]

Rasmus changing it later on means welcome to reimporting everything and wait a few hours for it to finish :P

…

On Fri, Aug 4, 2017 at 10:51 AM, jesperkondrup ***@***.***> wrote: For me as an artist it makes sense to have textures import as fast as possible, so I can try them on my shaders right away and then later make changes to import settings. In Unity I just tried importing 7 images in 6k resolution and it took around 15 seconds. Same textures in Godot took around 10 minutes to import (at least on my Windows 10 machine). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10059 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF-Z26MEFUn6N8JXtcAhzCC4aUFUyX1Mks5sUyHwgaJpZM4OsCSN> .

[kubecz3k]

I added a note to the first post, on how this issue probably should be addressed.

[eligt]

This call:

Ref<Texture> tex = ResourceLoader::get_singleton()->load(image, "Texture");

For a 1200x1200, 990kb PNG takes ~80ms (160ms in debug mode) on a really fast machine with a fast SSD (Samsung EVO Pro).

It seems to me like that's too high, not sure if it's related to this issue. I tried looking into etc2comp but I'm not sure how you disable it, I can find the thirdparty library files but no references inside Godot's source.

I feel like this is an important issue that should be looked at for 3.2?

[Calinou]

@eligt To disable etc2comp, uncheck Rendering → Vram Compression → Import Etc 2 in the Project Settings. ETC2 is only needed on mobile platforms that use the GLES3 renderer, so you won't miss out on anything if you're working on a desktop project or if you're using the GLES2 renderer.

We could disable it by default, as most people working on mobile projects are using the GLES2 renderer anyway (where only ETC1 is available).

This pull request aims to improve ETC2 import speeds, but it looks like progress on it has stalled.

[eligt]

@Calinou I did that but it does not seem to have any effect on the ResourceLoader::get_singleton()->load call at run-time so maybe this is a different issue. Or maybe it's normal for a 1200x1200 900kb PNG to load in 100+ milliseconds from an SSD? I think read-rate for my SSD is over 3000MB/s so it's not the bottleneck.

[Calinou]

@eligt I don't think the SSD is the bottleneck here. Large textures will take a while to be decoded and uploaded to the GPU if they're not in a VRAM-compressed format (e.g. uncompressed, WebP or PNG textures). Moreover, PNG is relatively slow to decode compared to an uncompressed format like TGA.

[eligt]

@Calinou I feel like 100+ milliseconds just to load the texture is way too much though, even when factoring in PNG decompression and loading in VRAM, considering my hardware:

Processor (CPU)	  Intel® Core™ i7 Quad Core Processor 7700HQ (2.8GHz, 3.8GHz Turbo)
Memory (RAM)	  32GB Corsair 2400MHz SODIMM DDR4 (2 x 16GB)
Graphics Card	          NVIDIA® GeForce® GTX 1060 - 6.0GB GDDR5 Video RAM - DirectX® 12.1
1st M.2 SSD Drive	  1TB SAMSUNG PM961 M.2, PCIe NVMe (up to 3000MB/R, 1700MB/W)

[chrisphyffer]

I think that this was one of the reasons why I was on/off with Godot. I would leave godot for several minutes to just load my textures into a material. My focus is primarily on Art.

When I disabled Rendering -> Vram Compression -> Import Etc 2, all of my textures would import really fast. That was the turning point for me. This reason IS the primary reason why I am now able to STAY in Godot. Now I can drink clean water (Godot) instead of going back to Mountain Dew (ahemunreal) to quench my thirst.

I think it would be really cool to have a dialog box to specify the target platform of your project beforehand. This way the artists who have no clue as to what compression is, etc, will be able to adapt to Godot easy and be retained. :)

[eligt]

@phyffer Could you define "really fast"? Like, how many textures of what size and how long do they take to load? I've disabled all texture compression and load times are still what I'd consider slow, look at my above comment for reference.

[chrisphyffer]

@eligt Here are my results:

Machine : cpu @ i8700k

4 textures are:
1 AO+Rough+Metallic,
1 Normal,
1 Albedo
1 Grayscale Mask

Time: 2048 x 2048 * 4 PNG Textures
< 1 Second to import
< 1 second to use in a spatial material

Time: 4096 x 4096 * 4 PNG Textures
~40 Seconds

Time: 4096 x 4096 * 4 PNG Textures to replace 2048 PNGs already used in a spatial material
< 9 Seconds to import

Compared to etc2 on, it definitely made Godot workable. But it is definitely similar to results you are getting

[qq715152910]

godot 3.2 Still not resolved.
I still hope to make some useful updates, such as addressing the slow import of resources, and the performance optimization of GD scripts. At present, it takes about an hour to import 1G resources. If you import 10G resources, you have to sit in front of the computer and sleep for two days. However, the resource editor file system browser of the 700M-1G import is now broken, and nothing will be displayed. File list, this may be the shortcomings of the editor itself, so I think that optimizing and solving the existing problems is the better way for Godot. If I can only import 100-500M resources, then it is destined that Godot cannot develop large games , But my path is now blocked and I cannot move forward. Hope that the next version will solve such problems, I wish Godot better and better!

[Calinou]

@qq715152910 Have you disabled Import Etc2 in the Project Settings?

Either way, asset importing will be significantly faster in Godot 4.0.

[qq715152910]

@qq715152910 Have you disabled Import Etc2 in the Project Settings?

Either way, asset importing will be significantly faster in Godot 4.0.

DESC:

I only enable the s3 tc and etc options, but not etc2, because I am working on a 2d project, and the project resources are large and large, so I try to subcontract the resources and package them into PCK / ZIP one to one, Unfortunately, the following error occurs when using the load method call after loading:

NOTE:

Loading textures is normal and can be used normally, the program will not crash, but there will be multiple consecutive errors below, The error can be reproduced，The error is similar to #35179

CODE:

ProjectSettings.load_resource_pack("res://scene/1464.zip")
var index = 0
for h in 14:
    for l in 18:
	var texture = load("res://scene/1464/map/%04d.png" % (index+1))
	index = index + 1

ERROR:

**> E 0: 00: 00.518 FileAccessWindows :: _ get_modified_time: Failed to get modified time for: D: /godot/Samples/game/scene/1464/map/0001.png.

   <C ++ Error> Method failed. Returning: 0
   <C ++ source program> drivers \ windows \ file_access_windows.cpp: 344 @ FileAccessWindows :: _ get_modified_time ()
   app.scene.map.gd:33 @_ready ()**

[jitspoe]

I've run into some textures that can take near-infinite time to import. There's a loop of a loop of a loop with some very large values here:

            for (int iRed2 = iMinRed2; iRed2 <= iMaxRed2; iRed2++)
            {
                for (int iGreen2 = iMinGreen2; iGreen2 <= iMaxGreen2; iGreen2++)
                {
                    for (int iBlue2 = iMinBlue2; iBlue2 <= iMaxBlue2; iBlue2++)
                    {

iMaxRed2 = 252645152, iMaxGreen2 = 252645152, iMaxBlue2 = 252645152

This code looks suspect:

		int iMaxRed2 = iColor2Red + (int)a_uiRadius;
		if (iMaxRed2 > 15)
		{
			iMinRed2 = 15;
		}

Should it be setting iMinRed2 there or iMaxRed2 there?

Here's a texture that exhibits the problem:

If you import it as a normal map with just the red and green channels, it imports. If you import as a RGB texture with compression, Godot freezes for some undetermined amount of time.

I also entered an issue on the etc2comp github, but it looks like that hasn't been touched in years: google/etc2comp#46

[Calinou]

@jitspoe Note that you don't need to use ETC2 compression if you're only targeting desktop platforms. In this case, disable Rendering > Vram Compression > Import Etc2 in the Project Settings to speed up import times.

We might want to do this by default in 3.2.2 due to the low amount of people targeting mobile platforms with the GLES3 renderer. (GLES2 can only use ETC1 on mobile platforms.)

[akien-mga]

Here's a texture that exhibits the problem:

If you import it as a normal map with just the red and green channels, it imports. If you import as a RGB texture with compression, Godot freezes for some undetermined amount of time.

What exact import settings are you using? I can't reproduce a freeze here with this texture. It's imported near instantly.

[jitspoe]

Is this before or after your change? :D

I think this is what I was using:

[akien-mga]

Is this before or after your change? :D

I'm testing with 3.2.1-stable to be sure :)

I set the same settings that you have but I still get an instant import for that texture. :/
In any case, the problematic code that you found definitely seems bogus, and the huge max values you had are likely due to uninitialized variables, so it could explain why it doesn't happen reliably.

[jitspoe]

That's interesting. I was having this issue back when I was using 3.1, and sometimes I'd have a texture that wouldn't import, so I'd load up 3.2 to import it, then go back to 3.1. I just assumed it was a bug that had been fixed, so I never entered an issue. Now that I've fully migrated over to 3.2, I was surprised to see it again. Perhaps it's something specific to my project settings?