+hmaxib double counting referenced binary

[zzydxm]

We currently use +hmaxib to avoid one too big process crashing the service. But referenced binary will be double counted binary vheap, even when the entire calculation is inside the process, for example, this will crash:

erl +hmax 1000000000 +hmaxib true
Erlang/OTP 26 [erts-14.2.1] [source-b6184975a6] [64-bit] [smp:72:72] [ds:72:72:10] [async-threads:1] [jit:ns] [systemtap]
Eshell V14.2.1 (press Ctrl+G to abort, type help(). for help)
1> B= <<0:100000000>>,ok.
ok
2> C=[B||N<-lists:seq(1,10000)],ok.
ok
*** ERROR: Shell process terminated! ***

In OTP 27, maybe more things are "correctly" counted into binary vheap, for example ets:insert/lookup. Which makes even more double counting and much easier to make process get killed:

erl +hmax 1000000000 +hmaxib true
Erlang/OTP 27 [erts-15.1] [source-caa6a847de] [64-bit] [smp:72:72] [ds:72:72:10] [async-threads:1] [jit:ns] [systemtap]
Eshell V15.1 (press Ctrl+G to abort, type help(). for help)
1> A=ets:new(a,[]).
#Ref<0.1727942139.3704487938.40245>
2> B= <<0:100000000>>,ok.
ok
3> C=[binary:part(B, N, 100)||N<-lists:seq(1,500)],ok.
ok
4> ets:insert(A,{k,C}).
true
*** ERROR: Shell process terminated! ***

This makes the feature really hard to be used practically.

Is there a way to not double counting these referenced binary in heap? Thanks!

[jhogberg]

ets:insert/2 and friends were correctly counted in 26, too: 27 just fixed under-counting on the creation of certain binaries. It may have made the symptoms worse but it's not the root cause.

Binaries will now only be double-counted when sharing is broken, such as when they're sent through a message (sharing-preserving helps here) or when inserted/looked up from ETS. I plan to fix this for good in OTP 29, along with the pesky issues of small references keeping large off-heap binaries alive. The changes in 27 were largely a preparation for this.

[zzydxm]

Binaries will now only be double-counted when sharing is broken

Is this in 27.1 or some other versions?
C=[B||N<-lists:seq(1,10000)],ok. crashed seems advising the binary is actually double counted even if calculation is only local.
Also ets:lookup will crash too

Eshell V15.1 (press Ctrl+G to abort, type help(). for help)
1> A=ets:new(a,[]).
#Ref<0.2500968853.719978497.88894>
2> B= <<0:100000000>>,ok.
ok
3> C=[binary:part(B, N, 100)||N<-lists:seq(1,300)],ok.
ok
4> ets:insert(A,{k,C}).
true
5> ets:lookup(A,k),ok.
ok
6> ets:lookup(A,k),ok.
ok
*** ERROR: Shell process terminated! ***

[jhogberg]

That's not as local as you think, those statements don't run in the shell process but are sent between the shell and an evaluator process, breaking sharing. If you try it again with sharing-preserving enabled, it will take more effort to crash it. Likewise, ets:lookup/2 "double-counts" because sharing is removed: hence "and friends" in my previous post.