Preference or order of texture format extensions

[javagl]

tl;dr: Should there be any hints, implementation notes, recommendations, rules or constraints for the order and preference of multiple texture format extensions within a single texture?

---

The following archive contains five glTF assets (with some shared textures and binary buffer file):

KhronosImageTypeSource 2024-11-07.zip

They follow the naming pattern TexturedUnitSquare..., with the following suffixes:

• (none): Uses a PNG texture. Booooring.
• Webp: Uses a WEBP texture (with PNG fallback)
• Ktx: Uses a KTX2/basisu texture (with PNG fallback)
• WebpKtx uses both extensions
• KtxWebp also uses both extensions

The difference between the latter is only in the order of the declarations of the extensions.

Specifically, the WebpKtx one declares

  "textures": [
    {
      "extensions": {
        "EXT_texture_webp": {
          "source": 1
        },
        "KHR_texture_basisu": {
          "source": 2
        }
      },    
      "source": 0,
      "sampler": 0
    }
  ],
  "images": [
    {
      "uri": "example.png"
    },
    {
      "uri": "example.webp"
    },
    {
      "uri": "example.ktx"
    }
  ],

The KtxWebp one declares

  "textures": [
    {
      "extensions": {
        "KHR_texture_basisu": {
          "source": 1
        },
        "EXT_texture_webp": {
          "source": 2
        }
      },    
      "source": 0,
      "sampler": 0
    }
  ],
  "images": [
    {
      "uri": "example.png"
    },
    {
      "uri": "example.ktx"
    },
    {
      "uri": "example.webp"
    }
  ],

(One could try out further combinations, e.g. omitting the fallbacks, or swizzling around the images while keeping the extension declaration order the same, but the current one is enough to bring the point across...)

One can drag-and-drop one glTF file (together with the binary and image files) into different online viewers.

• When dragging-and-dropping the KtxWebp or WebpKtx file into https://gltf-viewer.donmccurdy.com/ , then it always displays the KTX texture
• When dragging-and-dropping the KtxWebp or WebpKtx file into https://sandbox.babylonjs.com/, then it always displays the WEBP texture

(I wanted to try it out in the sample viewer, but stumbled over KhronosGroup/glTF-Sample-Viewer#584 )

---

Right now, the order that is used by a certain engine might depend on various aspects, like the order within the JSON, or the (somewhat random) order in which the extensions are happened to be checked for support internally. In any case, it is not predictable. Should that be changed?

[lexaknyazev]

By the JSON spec, object keys are unordered so both texture extensions are semantically equal. In general, engines are free to choose the format based on their internal logic. For example, they could prefer PNG even when KTX/Basis is supported.

[javagl]

I wonder whether the preference that is implemented in the viewers (gltf-viewer:KTX and babylon.js:WEBP) was a conscious decision, or happens to be a manifestation of an implementation detail. (My gut feeling is that it is the latter - otherwise we could let the implementors argue about who made the 'right' choice here 😁 )

An aside: At the first glance, it may look obscure to associate three image files (with different formats) with one texture. But a reasonable use case is when these images are indeed given via URIs - basically to let the client pick the ~"smallest/best representation that they can process".

But in that manner, I agree that this decision can be left to the engine.

[donmccurdy]

I agree that it's OK to leave this decision to the engine. 👍🏻

For three.js - it is intentional that we prefer extension-provided textures over PNG/JPEG, on the assumption the extension was added for this reason, and that older formats are intended as a fallback.

We have not implemented any particular preference if multiple extensions (like WebP vs. KTX2) compete. My feeling is that users typically create separate models rather than embedding different paths into one, if they need to express some context-dependent resource choice – even PNG/JPEG fallback textures seem to be rare in practice.

[bghgary]

FWIW, glTF loader extensions in Babylon.js have an optional order property on extensions which defines the order in which extensions are executed. Users can change this value if they want. That said, we don't define this order for the webp/basisu extensions by default.

[lexaknyazev]

Slightly tangential: given that WebP is supported in all major browsers and generally wins over JPEG and PNG (considering WebP lossless, see https://developers.google.com/speed/webp/gallery2), it may be reasonable to recommend WebP as a default option for exporters instead of JPEG/PNG.

[javagl]

Just another data point, now that the sample viewer bug was fixed via KhronosGroup/glTF-Sample-Renderer#10 :

The sample viewer always prefers KTX for the WebpKtx and KtxWebp assets.

My feeling is that users typically create separate models rather than embedding different paths into one

This may be a niche case in some way. But I think that the idea is roughly to have one GLB with many texture images (or maybe even multiple GLBs that share the same texture images), and provide these in a way that minimizes the network transfer size, depending on which format the client supports. A client could find that 100KB GLB file and then download only 1MB/4MB/8MB of image files depending on whether it supports KTX/WEBP/PNG. If the goal is to have a standalone GLB and decrease its size with WEBP, then providing the WEBP together with the PNG fallback would defeat the purpose...

[donmccurdy]

This may be a niche case in some way. But I think that the idea is roughly to have one GLB with many texture images ... depending on which format the client supports ...

yes, I think it's a pretty common use case (different resources for different clients) but that it's typically solved by creating multiple versions of the entire model. The asset pipeline is just much less complicated that way. Clients might also need (for example) lower texture resolution on a mobile device. See nytimes/rd-bundler-3d-plugins#2.