From 9b4988f17b5a36d8e399f408b24c4925fb17ddd3 Mon Sep 17 00:00:00 2001 From: Hamish Willee Date: Mon, 21 Aug 2023 19:18:48 +1000 Subject: [PATCH] Add transformer - draft --- .../generatekeyframe/index.md | 24 +++++++++++++++---- .../web/api/rtcrtpscripttransformer/index.md | 16 +++++++------ .../sendkeyframerequest/index.md | 15 +++++++++--- 3 files changed, 40 insertions(+), 15 deletions(-) diff --git a/files/en-us/web/api/rtcrtpscripttransformer/generatekeyframe/index.md b/files/en-us/web/api/rtcrtpscripttransformer/generatekeyframe/index.md index 9046c8f7bb9bd44..e994686258a6d73 100644 --- a/files/en-us/web/api/rtcrtpscripttransformer/generatekeyframe/index.md +++ b/files/en-us/web/api/rtcrtpscripttransformer/generatekeyframe/index.md @@ -8,9 +8,13 @@ browser-compat: api.RTCRtpScriptTransformer.generateKeyFrame {{APIRef("WebRTC Encoded Transform")}} -The **`generateKeyFrame()`** method of the {{domxref("RTCRtpScriptTransformer")}} interface ... +The **`generateKeyFrame()`** method of the {{domxref("RTCRtpScriptTransformer")}} interface causes a video encoder generate a key frame. -https://w3c.github.io/webrtc-encoded-transform/#abstract-opdef-generate-key-frame-algorithm +This method is called by a transformer that is processing outgoing frames to force a new complete frame to be sent (rather than a delta frame). +It might be needed by a WebRTC Encoded Transform that encrypts frames, to ensure that if a new encryption key is added, a key frame encrypted with that key is sent as soon as possible. + +The sender can specify a restriction identifier for the requested frame, which will restrict what encoders can be used. +The promise returned by the method will resolve just before enqueuing the corresponding key frame in a `RTCRtpScriptTransformer` readable. ## Syntax @@ -23,15 +27,25 @@ generateKeyFrame(rid) - `rid` {{optional_inline}} - - : A string ... + - : A string containing a "restriction identifier" ("rid"), which selects a video encoder to generate the frame based on capabilities such as: max-width, max-height, max-fps, max-fs (frame size), max-br (bitrate), max-pps (pixels per second), max-bpp (bits per pixel) depends (streams this steam depends on for interpretation). + The format is defined in [rfc8851](https://www.rfc-editor.org/rfc/rfc8851). + + The RID helps select the video encoder to generate the key frame based on its capabilities. + The first encoder that matches the specified `rid` is used. + If no encoder matches the `rid` then the first encoder is used, and `rid` is set to the encoder's restrictions. ### Return value -`Promise` +A {{jsxref("Promise")}} that fulfills with the timestamp of the frame. ### Exceptions -InvalidStateError, +- `InvalidStateError` + - The encoder is not processing video frames or is `undefined`. +- `TypeError` + - The provided `rid` but does not conform to the grammar requirements specified in Section 10 of [RFC8851] +- `NotFoundError` + - There are no video encoders. This might be raised if the corresponding `RTCRtpSender` is not active or its track is ended. ## Examples diff --git a/files/en-us/web/api/rtcrtpscripttransformer/index.md b/files/en-us/web/api/rtcrtpscripttransformer/index.md index 73ab732c1ef9d39..295564620cd601e 100644 --- a/files/en-us/web/api/rtcrtpscripttransformer/index.md +++ b/files/en-us/web/api/rtcrtpscripttransformer/index.md @@ -7,11 +7,15 @@ browser-compat: api.RTCRtpScriptTransformer {{APIRef("WebRTC Encoded Transform")}} -The **`RTCRtpScriptTransformer`** interface of the [WebRTC API](/en-US/docs/Web/API/WebRTC_API) is an abstraction for a stream object that can be used to transform encoded video and audio frames as are sent or received. +The **`RTCRtpScriptTransformer`** interface of the [WebRTC API](/en-US/docs/Web/API/WebRTC_API) is the worker-side counterpart of a {{DOMxRef("RTCRtpScriptTransform")}}. -Like the {{DOMxRef("TransformStream")}} interface, this represents an abstraction a concrete implementation of the [pipe chain](/en-US/docs/Web/API/Streams_API/Concepts#pipe_chains) _transform stream_ concept. +The transformer exposes a `readable` and `writable` stream into the worker, along with an "options" object provided to the {{DOMxRef("RTCRtpScriptTransform")}} on construction. +When the associated `RTCRtpScriptTransform` is assigned to a {{DOMxRef("RTCRtpSender")}} or {{DOMxRef("RTCRtpReceiver")}}, encoded media frames from the WebRTC sender or receiver pipelines are enqueued on the `readable` stream. -This is only available in dedicated workers (not the main window). +A WebRTC Encoded Transform pipes the `readable` through one or more {{DOMxRef("TransformStream")}} instances to the `writable`, and pipes the readable. +The "options" allow the transform code to determine whether the encoded media frames are incoming or outgoing. + +Objects of the type are only available in dedicated workers (not the main window), and are created when you construct a {{DOMxRef("RTCRtpScriptTransform")}}. {{AvailableInWorkers}} @@ -27,14 +31,12 @@ This is only available in dedicated workers (not the main window). ## Instance methods - {{domxref("RTCRtpScriptTransformer.generateKeyFrame()")}} - - : ? + - : Requests a video encoder generate a key frame. Called by a transformer when processing outgoing frames. - {{domxref("RTCRtpScriptTransformer.sendKeyFrameRequest()")}} - - : ? + - : Requests the sender send a key frame. Called by a transformer when processing incoming encoded video frames. ## Examples -### ? - In the following example ... ## Specifications diff --git a/files/en-us/web/api/rtcrtpscripttransformer/sendkeyframerequest/index.md b/files/en-us/web/api/rtcrtpscripttransformer/sendkeyframerequest/index.md index a85d4075755e45a..1e491c78ab289bd 100644 --- a/files/en-us/web/api/rtcrtpscripttransformer/sendkeyframerequest/index.md +++ b/files/en-us/web/api/rtcrtpscripttransformer/sendkeyframerequest/index.md @@ -8,7 +8,13 @@ browser-compat: api.RTCRtpScriptTransformer.sendKeyFrameRequest {{APIRef("WebRTC Encoded Transform")}} -The **`sendKeyFrameRequest()`** method of the {{domxref("RTCRtpScriptTransformer")}} interface ... +The **`sendKeyFrameRequest()`** method of the {{domxref("RTCRtpScriptTransformer")}} interface may be called by WebRTC Encoded Transform that is processing incoming encoded video frames, in order to request a key frame from the sender. + +This method might be called when a new user joins a WebRTC conference, in order to reduce the time before the user can start decoding the video (delta frames unless the recipient has a key frame and all subsequent key frames). +It might also be needed by a WebRTC Encoded Transform that encrypts frames, to ensure that if a new encryption key is added, a key frame encrypted with that key is sent as soon as possible. + +The message causes a Full Intra Request (FIR) to be sent back by the receiver of a video frame from a packetizer. +It should only be called when when receiving video and, for whatever reason, a user will not be able to decode a frame. ## Syntax @@ -22,13 +28,16 @@ None ### Return value -`Promise` +A {{jsxref("Promise")}} that fulfills with `undefined`. ### Exceptions +- `InvalidStateError` + - : The de-packetizer is not processing video packages, or is undefined. + ## Examples -### ??? +TBD. ## Specifications