Random values in struct padding

[Ollhax]

I’m making a real-time strategy game where you’ll control a lot of units, and I’ve set up the unit data as arrays of unmanaged structs. This way I can access it as a byte array and e.g. perform CRC operations on the entire gamestate very quickly, which lets me compare it with other clients and make sure they’re in sync.

My problem is when I run the game in Release, some of my structs now have random data in them that change the checksums and cause false desync warnings. I found out this is caused by bytes changing within the padding of my structs. I managed to create a somewhat simple repro case that still displays the problem - at least on my machine:

using System.Runtime.InteropServices;

struct Struct
{
    public byte A;
    public int B;
    public int C;
    public uint D;
}

struct Struct2(Struct e, bool f)
{
    public Struct E = e;
    public bool F = f;
}

public static class Test
{
    public static void Main()
    {   
        Span<byte> bytes = stackalloc byte[20];
        MemoryMarshal.Write(bytes, new Struct2(default, default));

        for (int j = 0; j < bytes.Length; j++)
        {
            Console.Write(bytes[j]);
            Console.Write(' ');
        }
        Console.WriteLine();
    }
}

Each run results in the last three bytes changing, e.g:

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 230 215 195

If I merge Struct and Struct2 or remove/change any of the fields, the output becomes all zeroes.

This is the layout of Struct2:

Size: 20 bytes. Paddings: 6 bytes (%30 of empty space)
|===============================|
|  0-15: Struct E (16 bytes)    |
| |===========================| |
| |     0: Byte A (1 byte)    | |
| |---------------------------| |
| |   1-3: padding (3 bytes)  | |
| |---------------------------| |
| |   4-7: Int32 B (4 bytes)  | |
| |---------------------------| |
| |  8-11: Int32 C (4 bytes)  | |
| |---------------------------| |
| | 12-15: UInt32 D (4 bytes) | |
| |===========================| |
|-------------------------------|
|    16: Boolean F (1 byte)     |
|-------------------------------|
| 17-19: padding (3 bytes)      |
|===============================|

The last three bytes at the end is the padding that is causing my grief. Interestingly, the first padding (between A and B) seems to get cleared, or I’m just being lucky that it's zeroed.

So my questions:

• Just to confirm, is this a bug or expected behavior? According to this, the contents of the padding bytes is indeterminate, so I guess it’s expected and I'm picking up random data from the stack or some CPU register.
• If it’s expected, is there a way to instruct the runtime to clear the padding? Either for this struct in particular, or (preferably) for all structs in that assembly. I could use [StructLayout(LayoutKind.Sequential, Pack = 1)], but I’ve read about a lot of reasons why it should be avoided, and it’s easy to forget to apply it on each struct.

[huoyaoyuan]

This looks like a gap in JIT: default(Struct2) generates initialization of 20 bytes, however new Struct2(default, default) initialize only 17 bytes:

        Struct2 Foo() => default;

        Struct2 Foo2() => new Struct2(default, default);

; Method CSPlayground.Program:Foo():CSPlayground.Struct2:this (FullOpts)
G_M000_IG01:                ;; offset=0x0000
       vzeroupper 

G_M000_IG02:                ;; offset=0x0003
       vxorps   xmm0, xmm0, xmm0
       vmovdqu  xmmword ptr [rdx], xmm0
       vmovdqu  xmmword ptr [rdx+0x04], xmm0
       mov      rax, rdx

G_M000_IG03:                ;; offset=0x0013
       ret      
; Total bytes of code: 20

; Method CSPlayground.Program:Foo2():CSPlayground.Struct2:this (FullOpts)
G_M000_IG01:                ;; offset=0x0000
       sub      rsp, 24
       vzeroupper 

G_M000_IG02:                ;; offset=0x0007
       vxorps   xmm0, xmm0, xmm0
       vmovdqu  xmmword ptr [rsp], xmm0
       vmovdqu  xmm0, xmmword ptr [rsp]
       vmovdqu  xmmword ptr [rdx], xmm0
       mov      eax, dword ptr [rsp+0x10]
       mov      dword ptr [rdx+0x10], eax
       mov      byte  ptr [rdx+0x10], 0
       mov      rax, rdx

G_M000_IG03:                ;; offset=0x0027
       add      rsp, 24
       ret      
; Total bytes of code: 44

Then MemoryMarshal.Write makes the difference visible. Not sure what's the desired behavior here.

If it’s expected, is there a way to instruct the runtime to clear the padding?

It seems that default(T) will compiled to initialize all the bytes belongs to T.

@EgorBo any idea about the expected behavior here?

[EgorBo]

Should a constructor call with values initialize the padding? It can cause performance regression.

I think it depends on whether JIT decides to promote a struct to registers or not (based on heuristics), if not - it typically end up using e.g. SIMD and copies/zeroes padding too

[Clockwork-Muse]

.... as a side question, isn't the behavior here going to be platform-dependent anyways, given by default Pack will be 0? That you're not even guaranteed to have the same packing/padding over different platforms, never mind the randomized data.

[jakobbotsch]

This is indeed expected behavior. We do not make any guarantees about the value of padding unless the struct has LayoutKind.Explicit since it would generally make stores/loads more expensive.

[Ollhax]

@jakobbotsch Understood, thanks for the confirmation. In practical terms, can I expect any garbage values in padding to only appear at the end of structs, or could it appear in the middle somewhere as well?

I'm considering an option where I use reflection to calculate the size of the struct without final padding bytes (so 17 bytes in my example above) and only copy that part to the byte arrays, cutting out the garbage. I know this would still rely on undefined behavior, but if it works on windows it's at least an option to consider.

[jakobbotsch]

The garbage can appear in the middle as well. It completely depends on how the JIT decides to break the struct into primitives (this depends on how the struct's fields are used throughout the function).
If there is padding between two fields that the JIT decided to introduce primitives for, then on stores/loads the JIT is generally not going to generate code to store/load that padding.

[Ollhax]

It seems that default(T) will compiled to initialize all the bytes belongs to T.

I don't think I could enforce using only that, and it would be too error prone to try to remember it.

I realize that I am relying on undefined behavior here. It's a shame that there aren't stronger guarantees, it's extremely useful to be able to cast data to Span<byte> and deal with it directly - it both simplifies a lot of code and makes it possible to serialize things much faster. I don't need my code to run on any and all exotic hardware out there, but I do need to run it very quickly on a PC.

I understand the risk of a performance regression if you decide to zero out the padding, but at least from my (admittedly brief) testing it's rare for structures to have this problem. Like I mentioned in my example above, the problem disappears if I try to merge the two structs, or if I change any of the contents. Perhaps it's possible to direct the changes to only these corner cases, meaning any performance regressions would not be noticeable in most code?

[jakobbotsch]

I understand the risk of a performance regression if you decide to zero out the padding, but at least from my (admittedly brief) testing it's rare for structures to have this problem.

It's not all that rare. For example, Span<T> on 64-bit platforms has 4 bytes of padding, so stores/loads of Span<T> would become more expensive if we had to provide guarantees about the value of padding.

[Ollhax]

@Clockwork-Muse .... as a side question, isn't the behavior here going to be platform-dependent anyways, given by default Pack will be 0? That you're not even guaranteed to have the same packing/padding over different platforms, never mind the randomized data.

Yeah, it's not optimal - but I'm targeting only windows, so I think I might get away with it. I guess I'd still be vulnerable to changes across .NET versions if I output the serialized state as save data. I would prefer if there was a performant way to do this that was also portable, but e.g. writing each struct field individually is not even close to being fast enough for my needs - I'll have thousands of units with several dozens of data arrays per unit, and serialization should take no more than a couple of milliseconds.

I'm still considering using Pack=1 to avoid the padding to begin with. If I ever need to support a platform that requires aligned access, I guess I'd need to push in a few manual padding bytes here and there to make it work. I'm just not very keen, it's a lot of field accesses that become slower for the sake of avoiding a rare problem.

If I may dream for a bit, it might be useful to have a [StructLayout(LayoutKind.Deterministic)] setting that guarantees a certain layout, but unlike LayoutKind.Explicit it would not require a lot of manual work to set up. Say it works like LayoutKind.Sequential but with exact rules for padding that is the same for all platforms, and guarantees that the struct and padding is fully zeroed on creation. This would probably not work universally, but there's already plenty of precedence for that in other StructLayout settings. I'm sure it would be useful for a lot of real world scenarios out there, and the cream on top would be a where T: deterministic generic constraint to be sure it's applied on the structs that need it.

[huoyaoyuan]

I realized that it's indeed also indeterministic to inspect a struct in bytes, when endianness is encountered.

[Clockwork-Muse]

Yeah, it's not optimal - but I'm targeting only windows, so I think I might get away with it.

Windows on ARM is (about to become more of) a thing. Windows also runs on a number of other architectures, although presumably you're using an x86-64 TFM.

I guess I'd still be vulnerable to changes across .NET versions if I output the serialized state as save data.

If I ever need to support a platform that requires aligned access...

Note that this sort of thing is one of the cases I would prefer putting in the extra work in now (explicit layout, manual padding bytes, reordering fields, whatever).

[Ollhax]

I appreciate everyone's input about this issue! In the end I decided to enforce Pack=1 on my unit data structs to avoid the padding garbage.

[Clockwork-Muse]

The game runs at 60 fps so I have 16.6 ms per update to work with. All of the game logic needs to fit into this budget, and since the game runs a lockstep multiplayer model the speed of the slowest computer in the network game that dictates the speed of the game.

Yes/no. You likely want rendering to be 60fps+, but especially for mass-unit RTSes the actual game simulation can run at a far slower rate. Keep in mind that most of the high player-count FPS games run at 20-30Hz (and 60-120Hz generally only happens for lower player counts). Especially since presumably you're sending player commands over the wire, not actual unit position. Note that even with a lower simulation rate it's still possible to have smooth interpolation.
And there's other options here too - you likely don't need to calculate the checksum every simulation frame, for instance, although doing so would likely be simpler....

Depending on the algorithm used, doing the hashing as a single buffer may benefit from SIMD instructions. Note that the general recommendation in some places seems to be to use something more modern than CRC, if available.

The speed of your serialization is likely highly dependent on the types being blittable, though, so if they're not you're likely to have a performance penalty. Also, you can't use float/double for any of this, since the results of floating-point calculations can be hardware dependent.

• Atomicity: What @huoyaoyuan is talking about is "field tearing" (the classic example being a 64-bit value being manipulated as two 32-bit values), but the more fundamental problem you're more likely to encounter would be "value tearing", where different parts of the struct are updated independently. Yes, this is primarily a multi-threading problem. A multithreaded engine is more difficult to write, but does have other advantages (even if all you do is rendering on a separate thread).
• ARM: ARM is currently the most popular gaming platform.... that being mobile phones (and the Nintendo Switch). With Windows pushing for ARM (again) I expect this to go up. It may be worth it to spend the effort "now" for future proofing.
• Endianness: At least Windows on ARM only does little-endian.

[Ollhax]

@Clockwork-Muse Yeah, rendering is decoupled from the update rate. It's correct that I can lower the update rate and I probably will, but what I said is still true even if it's at 30 fps / 33.3 ms or 15 fps / 66.6 ms, just to a slightly lesser degree. I want to maximize the number of units I can support - or minimize the system requirements - and having serialization take up a significant portion of that frame time when it doesn't have to seems wasteful to me.

And there's other options here too - you likely don't need to calculate the checksum every simulation frame, for instance, although doing so would likely be simpler....

I currently do checksums every 50 frames, but I don't think that actually makes sense and I'll probably change it. It degrades the reporting accuracy, and I need the checksum to be quick enough to fit inside the normal update time anyway, or there'd be spikes every N frames.

Thinking about this some more, I guess I could spread out the work so I do one system at a time across multiple frames, but the system is already pretty complicated. But that's an option if I ever need to change this.

The speed of your serialization is likely highly dependent on the types being blittable, though, so if they're not you're likely to have a performance penalty.

Do you mean if I'd go the route of doing field-by-field serialization? The cost would be mainly having to do thousands of thousands of operations instead of a single big one. I ran a small synthetic test on this now, serializing an array of 1000 structs with a meager 8 fields took 14x the time when done field-by-field, vs casting the entire array to Span<byte> and writing it in one go.

Also, you can't use float/double for any of this, since the results of floating-point calculations can be hardware dependent.

Yeah, I'm using fixed point types.

• Atomicity: Alright. I do already have separate threads for rendering, window management etc, but none of that touches the component arrays. Though I might want to split up some of the component update work to separate threads at some point, so it's good to know about this.

• ARM: I haven't done mobile games in a few years, but from what I hear it's not even close to being a viable market for most indie game developers these days. Good point about the Switch though, I actually worked on it just a few weeks ago so it's a bit embarrassing that I forgot it's running ARM 😅 But it's not a platform that makes sense to support for this particular game, even if I could somehow get a NativeAOT build to run on it.

[Clockwork-Muse]

Thinking about this some more, I guess I could spread out the work so I do one system at a time across multiple frames, but the system is already pretty complicated. But that's an option if I ever need to change this.

Yes, this is some of what I was thinking of.

Do you mean if I'd go the route of doing field-by-field serialization? The cost would be mainly having to do thousands of thousands of operations instead of a single big one. I ran a small synthetic test on this now, serializing an array of 1000 structs with a meager 8 fields took 14x the time when done field-by-field, vs casting the entire array to Span<byte> and writing it in one go.

No, if you were going to do field-by-field you may as well just hash it per-field.

What I was referring to is that serializing the struct itself as a Span<byte> has limitations based on the type of the field. Outside of that set of types you may have a performance penalty. Of particular concern is likely bool and string.

I haven't done mobile games in a few years, but from what I hear it's not even close to being a viable market for most indie game developers these days.

.... I don't know why mobile would be any different than Steam here....

[Ollhax]

What I was referring to is that serializing the struct itself as a Span has limitations based on the type of the field. Outside of that set of types you may have a performance penalty. Of particular concern is likely bool and string.

I'm not keeping strings or any types that would not pass through the unmanaged constraint in my data structs, that would stop me from typecasting to Span<byte>. I heard about the non-blittable bool issue, but haven't seen any practical problems arise from it. Do you have any examples?

[Clockwork-Muse]

Not at this time