Suggestion: Formal Schematic API with a supported Schematic Format

[gabizou]

So, at Sponge, we've been hard at work with adding new features as they've been introduced to the game, and namely, one of them is Schematics. As some of you are aware, 1.10 officially now supports creating, placing, and exporting schematics.

A couple of premiers: The vanilla schematics support serializing and deserializing TileEntity instances and likewise, Entity instances. Support is also added for performing rotations on a schematic.

The issue that we've found with the vanilla implementation of Schematics is that the file sizes are through the roof with how they're implemented.

Fortunately, some of the Sponge Team have devised a more condensed format without losing data.

The format supports everything as long as it is 1.8+ (due to how block states can be serialized as their id strings into a Palette for even smaller file footprints), and of course, the content versioning is required for the uses of content upgrading as Minecraft's version changes and data needs to be fixed.

My suggestion however, is while Sponge is soon to provide it's own API, I'd imagine that it's something really neat to see if Paper manage to add such an API for plugins to use, and further down the line, a common sharing of schematics can be done with Sponge+Paper for everyone to benefit :)

[Techcable]

Why do we need to use the pallete? Isn't compression enough?
Other than that this sounds great, especially if we can use WorldEdit to use it.

[Deamon5550]

@Techcable Especially in ecosystems such as forge where you can easily exceed 65535 total blockstates the next best alternative while still being able to support everything would probably be varints, and if you have the data value as the lowest four bits of the global id then you're almost always going to be needing two bytes for a varint with the high id mod states taking 3 bytes.

For the lazy we could take the same route as the chunk packets and say that if the palette isn't present then it is using the global id map (which for vanilla means ids are 13 bits wide). Which is probably fair if your palette size isn't less than 256 (chunk packets use this cutoff also, anything more than 8 bits wide uses the global palette).

That linked format specification is still under some development so if you have any suggestions then I'll take them.

[Techcable]

@Deamon5550
How much more efficient are the manual palettes then compression of varints?

[Deamon5550]

@Techcable Well you can ignore the compression since both would be compressed anyway (the whole NBT of the schematic gets gzipped either way). Using a palette the worst case is 13 bits, and the best is 4 bits. Using a varint your best is 8 and your worst is 16 (any material other than stone, dirt, grass, cobblestone, planks, saplings, and bedrock will use 16).

While you could make an argument that a majority of the average schematic is made up of one of those best case materials for the varint, unless you have > 256 unique materials the palette will beat it anyway.

I still need some numbers so I'll probably run something across the vast store of schematics on my server to see what the average case actually is.

[Deamon5550]

@Techcable Here we go, have some stats.

The varint is larger than a palette for all things except the very largest schematic I had.

[Techcable]

@Deamon5550 Can you redo those stats but take into account that the varints will be compressed?
Its certainally true an uncompressed palette will be much smaller than uncompressed varints, but what about when compressed with zlib?

[electronicboy]

Please refrain from bumping old tickets with stuff that doesn't add to the discussion of the ticket in question, those plugins, beyond having schematics, have nothing pertaining to the nature of this issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in a long time. If the issue still applies to the most recent supported version, please open a new issue referencing this original issue.

[gabizou]

Bah.... stale......

Anyways, some comments:

• The specification is still maintained (will be releasing a v3 RFC very soon, adding 3D biomes due to the nature of 1.16's updates and where 1.17 is going)
• WorldEdit formally adopted the specification for schematics post 1.13 due to the intrinsic nature of palette usage.
• Sponge has had first class support of the schematic with little issue of utilizing the format to copy paste standard sized schematics (standard being less than 200k blocks).

That being said, I'm fairly busy on my own, and I doubt I'll be able to produce such a PR/RFC of an API to support everything that a Schematic can represent and decouple the nature of Blocks being potentially re-registerable (with the way the game is pointing to, 1.17's update already makes some registries semi dynamic)