Hanging when importing PNG with etc/etc2 VRAM compression enabled [etc2comp bug]

[calebROlson]

Godot version

3.4 beta 4 (6a058cb)

System information

Windows 10

Issue description

Importing a png with import settings as follows leads to hang (using Qodot clip texture as an example):

[remap]

importer="texture"
type="StreamTexture"
path.s3tc="res://.import/clip.png-5fc9a9edc976377600ca7c1c2d5af40b.s3tc.stex"
path.etc2="res://.import/clip.png-5fc9a9edc976377600ca7c1c2d5af40b.etc2.stex"
metadata={
"imported_formats": [ "s3tc", "etc2" ],
"vram_texture": true
}

[deps]

source_file="res://addons/qodot/textures/special/clip.png"
dest_files=[ "res://.import/clip.png-5fc9a9edc976377600ca7c1c2d5af40b.s3tc.stex", "res://.import/clip.png-5fc9a9edc976377600ca7c1c2d5af40b.etc2.stex" ]

[params]

compress/mode=2
compress/lossy_quality=1.0
compress/hdr_mode=0
compress/bptc_ldr=0
compress/normal_map=0
flags/repeat=1
flags/filter=false
flags/mipmaps=true
flags/anisotropic=true
flags/srgb=1
process/fix_alpha_border=false
process/premult_alpha=false
process/HDR_as_SRGB=false
process/invert_color=false
process/normal_map_invert_y=false
stream=false
size_limit=0
detect_3d=false
svg/scale=1.0

I ran MSVC profiler which has the following info:

Function Name	Total CPU [unit, %]	Self CPU [unit, %]	Module	Category
 + godot.windows.tools.64.exe (PID: 20356)	289492 (100.00%)	0 (0.00%)	Multiple modules	
| + [External Code]	289440 (99.98%)	645 (0.22%)	Multiple modules	Other | IO | Networking | Driver | Graphics | File System | Kernel
|| + Concurrency::details::`anonymous namespace'::_Task_scheduler_callback	270934 (93.59%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||| + Concurrency::details::_DefaultPPLTaskScheduler::_PPLTaskChore::_Callback	270934 (93.59%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||| + Concurrency::details::_TaskProcHandle::_RunChoreBridge	270932 (93.59%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||| + Concurrency::details::_PPLTaskHandle<unsigned char,Concurrency::task<unsigned char>::_InitialTaskHandle<void,<lambda_33c0edf7e2bf8e11477062206c55a36b>,Concurrency::details::_TypeSelectorNoAsync>,Concurrency::details::_TaskProcHandle>::invoke	262563 (90.70%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||| + Concurrency::task<unsigned char>::_InitialTaskHandle<void,<lambda_33c0edf7e2bf8e11477062206c55a36b>,Concurrency::details::_TypeSelectorNoAsync>::_Perform	262563 (90.70%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||| + Concurrency::task<unsigned char>::_InitialTaskHandle<void,<lambda_33c0edf7e2bf8e11477062206c55a36b>,Concurrency::details::_TypeSelectorNoAsync>::_Init	262563 (90.70%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||| + Concurrency::task<unsigned char>::_InitialTaskHandle<void,<lambda_33c0edf7e2bf8e11477062206c55a36b>,Concurrency::details::_TypeSelectorNoAsync>::_LogWorkItemAndInvokeUserLambda<std::function<unsigned char __cdecl(void)> >	262560 (90.70%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||| + std::_Func_class<unsigned char>::operator()	262551 (90.69%)	2 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||| + std::_Func_impl_no_alloc<<lambda_052e919cc0e5399df76dff3972c0cac1>,unsigned char>::_Do_call	262549 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||| + std::_Invoker_ret<unsigned char,0>::_Call<<lambda_052e919cc0e5399df76dff3972c0cac1> &>	262549 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||| + std::invoke<<lambda_052e919cc0e5399df76dff3972c0cac1> &>	262549 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||| + std::_Invoker_functor::_Call<<lambda_052e919cc0e5399df76dff3972c0cac1> &>	262549 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||| + <lambda_052e919cc0e5399df76dff3972c0cac1>::operator()	262549 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||| + std::_Func_class<void>::operator()	262549 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||| + std::_Func_impl_no_alloc<<lambda_33c0edf7e2bf8e11477062206c55a36b>,void>::_Do_call	262549 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||||| + std::_Invoker_ret<void,1>::_Call<<lambda_33c0edf7e2bf8e11477062206c55a36b> &>	262549 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||| + std::invoke<<lambda_33c0edf7e2bf8e11477062206c55a36b> &>	262549 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||||||| + std::_Invoker_functor::_Call<<lambda_33c0edf7e2bf8e11477062206c55a36b> &>	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||| + <lambda_33c0edf7e2bf8e11477062206c55a36b>::operator()	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||||||||| + std::_Packaged_state<unsigned int __cdecl(void)>::_Call_immediate	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||||| + std::_Func_class<unsigned int>::operator()	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||||||||||| + std::_Func_impl_no_alloc<std::_Fake_no_copy_callable_adapter<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int &,int &,unsigned int &>,unsigned int>::_Do_call	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||||||| + std::_Invoker_ret<unsigned int,0>::_Call<std::_Fake_no_copy_callable_adapter<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int &,int &,unsigned int &> &>	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||||||||||||| + std::invoke<std::_Fake_no_copy_callable_adapter<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int &,int &,unsigned int &> &>	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||||||||| + std::_Invoker_functor::_Call<std::_Fake_no_copy_callable_adapter<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int &,int &,unsigned int &> &>	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||||||||||||||| + std::_Fake_no_copy_callable_adapter<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int &,int &,unsigned int &>::operator()	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||||||||||| + std::_Invoke_stored<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int,int,unsigned int>	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||||||||||||||||| + std::_Invoke_stored_explicit<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int,int,unsigned int,0,1,2,3,4>	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||||||||||||| + std::invoke<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int,int,unsigned int>	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||||||||||||||||||| + std::_Invoker_pmf_pointer::_Call<unsigned int (__cdecl Etc::Image::*)(unsigned int,unsigned int,unsigned int),Etc::Image *,unsigned int,int,unsigned int>	262548 (90.69%)	0 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||||||||||||||| + Etc::Image::IterateThroughWorstBlocks	262548 (90.69%)	1 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||||||||||||||||||||| + Etc::Block4x4::PerformEncodingIteration	262546 (90.69%)	9 (0.00%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||||||||||||||||| + Etc::Block4x4Encoding_RG11::PerformIteration	234767 (81.10%)	1 (0.00%)	godot.windows.tools.64.exe	Kernel
||||||||||||||||||||||||||||||||||| + Etc::Block4x4Encoding_RG11::CalculateG11	118403 (40.90%)	51463 (17.78%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||||||||||||||||||| - Etc::Block4x4Encoding::CalcPixelError	40705 (14.06%)	24172 (8.35%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||||||||||||||||||| - Etc::Block4x4Encoding_R11::DecodePixelRed	21797 (7.53%)	21791 (7.53%)	godot.windows.tools.64.exe	Kernel
|||||||||||||||||||||||||||||||||||| - Etc::ColorFloatRGBA::ColorFloatRGBA	3745 (1.29%)	3743 (1.29%)	godot.windows.tools.64.exe	Kernel

I suspect this has to do with this change from #51402.

Setting flags/anisotropic=false or flags/mipmaps=false does not fix the hang.

Steps to reproduce

Import png with above import definition on Godot 3.4 beta 4 using etc/etc2 compression. Godot will hang at re-importing assets pop-up.

Minimal reproduction project

vram-compression-hang.zip

[Calinou]

@calebROlson Does this occur in 3.3.2 or 3.3.3?

[calebROlson]

It does not occur in 3.3.3 or 3.4 beta 3.

Only seems to happen with the beta 4, both the dev snapshot and one I built myself with Visual Studio 2019.

[calebROlson]

I'm seeing this on 3.3.3 stable now. This is not a regression from 3.4beta3 (it is present in 3.3.3). Sometimes the operation finishes eventually, and other times it hangs until I force restart Godot. Once the import succeeds, I don't see the issue again until I re-import. I am running the 64-bit versions.

Seems similar to google/etc2comp#46 and #37488 ? I noticed my callstack calls Etc::Block4x4Encoding_RG11::CalculateG11 instead of the RGB8 functions. It looks like there is still an issue in that library, and that #37488 was not the whole fix.

I am running GLES3. etc 2 is "only supported when using the GLES3 renderer".

I was able to workaround by disabling etc2 vram compression in the project settings for each affected release.

[calebROlson]

I found a fork of etc2comp (https://github.com/abergmeier/etc2comp) which has numerous fixes, including threaded executor support. After a few tweaks, I was able to build it with 3.3.3 and this fixed the issue for me and my textures imported in a couple seconds. However, the maintainer uses c++17 and I needed to switch my SConstruct file to use that instead of c++14 (is there a downside to doing this?).

I will attach my patch file for reference.

etc2comp_fork.patch

[Calinou]

However, the maintainer uses c++17 and I needed to switch my SConstruct file to use that instead of c++14 (is there a downside to doing this?).

We'd like to keep 3.x buildable with a C++14 compiler, but this is not an issue for the master branch which already requires a C++14 compiler.

[fire]

@calebROlson Would you be interested in assistance to make this a Godot 3 pr?