WebXR: Implement antialiased multiview using OCULUS_multiview #15

felixtrz · 2023-10-18T01:17:26Z

This is a port of mrdoob#25981 made by @snagy with minor changes to resolve conflicts from rebase (r149 to r156)

Related issue: mrdoob#20368

This is a crack at reimplementing multiview for XR devices that support the OCULUS_multiview extension. This extension lets us use multiview along with MSAA.

This is a huge speed improvement for applications that are CPU limited and draw count bound. Even the VR ballshooter example runs at a higher framerate when shooting on the quest 2 with multiview enabled than without.

Unfortunately the extension has one big problem - the multisampled render to texture extension this uses will discard the frame buffer if other texture operations are used during rendering. This is outlined in KhronosGroup/WebGL#2912 . I've added a deferred upload of texture data (such as video frames) to mitigate this problem, by deferring uploads until after rendering of the main scene has finished. This adds a frame of latency to the texture uploads, and is only active if multiview is enabled.

Some apps (mostly ones that render stuff like real time reflections to a texture) won't work with this extension without refactoring when those renders happen, to ensure that they don't interrupt the main frame.

Most experiences will get a free perf gain from this extension, but some will get broken. Because of this, I do not enable multiview by default and instead added a 'multiviewStereo' flag to the renderer properties to enable it. If this flag isn't set, everything should work as before.

Whenever you do a texture upload or bind another framebuffer or anything else in the list in the link in the first post, it drops the framebuffer and nothing else draws to that buffer for the rest of the frame.

The workaround in this PR (deferring texture uploads until the frame is complete) is the best thing I can come up with to fix most experiences without large changes to three. For a lot of apps (like the ball shooter examples), that's fine. They get a huge perf boost with no visual cost.

The biggest problem with this workaround is that skeletal meshes upload their bone textures with TexImage, which is one of the prohibited operations. With the workaround, that means that all bone animation lags by one frame. If this PR lands, the next thing I would do is try to move the skinnedmesh processing and upload before the multiview scene begins rendering, so they don't interrupt the render but also aren't delayed.

Beyond that, other textures could potentially be uploaded the same way - before the multiview scene loop begins. Stuff like movie textures are one frame delayed with the workaround, which could be fixed with some work. We have movies in our app but the frame delay on a movie is less consequential (for us) than the skinned mesh delay.

The trickiest usage to fix will be applications that switch off to render another view (such as a mirror reflection) in the middle of the scene. We would also have to move this to the start of the frame so it doesn't interrupt rendering, and I'm not sure we can easily guarantee that without changing a lot of dev expectations on how three works. It's possible to fix this on a per-app basis as a dev, though - our app includes a camera that renders its own view of the scene, and we were able to render that before the multiview frame by just architecting our code properly.

I'm skeptical that we could get it to a universal, always-on implementation until the browser and drivers allow more control over when the multisample resolve happens. (AFAIK this is possible once we're using Vulkan as our base API in the browser, instead of GLES).

This contribution is funded by Meta

dmarcos · 2023-10-18T17:55:40Z

src/renderers/webgl/WebGLBackground.js

@@ -68,7 +68,7 @@ function WebGLBackground( renderer, cubemaps, cubeuvmaps, state, objects, alpha,
 			if ( boxMesh === undefined ) {

 				boxMesh = new Mesh(
-					new BoxGeometry( 1, 1, 1 ),
+					new BoxGeometry( 10000, 10000, 10000 ),


Is this needed related to multiview?

According to @snagy: the WebGLBackground code has some special casing and is not multiview-aware, the original dimension of 1x1x1 becomes really obvious as a box around your head with multiview, and making the box really big effectively negates this effect.

I see. concern is that this might affect non-vr (2D) applications perhaps causing the background to be clipped? 10,000 is 10km

src/renderers/webgl/WebGLTextures.js

src/renderers/webxr/WebXRManager.js

dmarcos · 2023-10-19T19:08:15Z

src/renderers/webgl/WebGLTextures.js

@@ -1138,6 +1182,7 @@ function WebGLTextures( _gl, extensions, state, properties, capabilities, utils,
 		}

 		textureProperties.__version = texture.version;
+		return true;


is this return needed?

yes, on line 493 it's used to help determine whether or not to defer texture uploads

dmarcos · 2023-10-20T18:51:20Z

Thanks for the patience and the effort!

github-actions · 2023-10-20T18:53:51Z

📦 Bundle size

Full ESM build, minified and gzipped.

Filesize `dev`	Filesize PR	Diff
661.1 kB (164 kB)	661.1 kB (164 kB)	+0 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

Filesize `dev`	Filesize PR	Diff
454.1 kB (110 kB)	454.1 kB (110 kB)	+0 B

* WebXR: Implement antialiased multiview using OCULUS_multiview * remove this. pointers in WebGLTextures * remove this. pointers in WebGLTextures * remove unnecessary condition when setting texture params

…edium#15) * WebXR: Implement antialiased multiview using OCULUS_multiview * remove this. pointers in WebGLTextures * remove this. pointers in WebGLTextures * remove unnecessary condition when setting texture params

* WebXR: Implement antialiased multiview using OCULUS_multiview * remove this. pointers in WebGLTextures * remove this. pointers in WebGLTextures * remove unnecessary condition when setting texture params

WebXR: Implement antialiased multiview using OCULUS_multiview

2988e59