Bits: Dynamic replacement of server-provided content (tokens, "shortcodes 2.0")

[dmsnell]

Status

See the proposal on Core Make

Summary

Blocks established a unified and elegant interface for arranging and composing high-level web content, specifically "block" content. One area the block model has been deficient, however, is in establishing more granular semantic types of "inline" content that carry meaning not associated with a block. These inline semantic parts represent content or metadata that comes from outside the document being edited and may be relatively static or highly dynamic.

These small semantic things will be referred to as Bits.

Examples:

• When building a query loop template block one might want to assign the source for an image block as the featured image of the post or page in the current iteration of the loop.
• When writing an email newsletter one might want to include the name of the email recipient who will receive the rendered document.
• When creating block patterns one might want to include content that populates dynamically when the pattern is inserted, such as the date of the start of the current week.
• One might want to include in a post information about the post's author, including name, avatar, etc....
• When building a reusable block pattern for displaying a book review, one might want to connect an image's source, a paragraph's content, and more from a custom field pulled from the database.

"tokens" are now "bits"

This proposal previously discussed the terms "dynamic tokens" and "inline tokens" and more briefly, "tokens." In all cases the new terminology will follow the term "Bits" unless explicitly distinguished therefrom.

If we could introduce blocks everywhere we might not need Bits, but we cannot, for instance, include a block inside an HTML attribute or even inside a RichText field. A Bit is a kind of indicator or marking inside a document for a region that will be replaced based on some external data. They can appear in almost any place within a document, within a block, within HTML.

Overview of this proposal:

• Bits are atomic HTML content and cannot be nested. They are a placeholder that may carry their own intrinsic type and set of attributes.
• Bits can only exist within valid HTML structure. They cannot define HTML structure. For example, a Bit may represent a place where content should appear inside a <p> tag, but they cannot determine which HTML tag name to print. (There are examples with shortcodes doing this, e.g. <[my_heading_tag]>…</[my_heading_tag]>).
• Bits are required to supply a plaintext fallback value that will render in cases where they cannot be sufficiently replaced.
• Bits can only supply a limited replacement value. Bits provide suggested replacement values and WordPress will subjectively decide whether to allow the output. When rejected, a Bit's fallback value will be rendered instead. This subjective limitation exists to ensure that Bits don't accidentally break or change a page in a dramatic way that wasn't intended.
• Bits should appear in the editor in such a way to indicate that it's a placeholder token. For example, inside a RichText area a Bit may be outlined by a border and contain the name of the type of Bit it is and a preview of what its replacement content might look like.
• Bits can appear in plaintext contexts, such as in the URL field for an image block, whereby the Bit might represent the corresponding value of plugins_url() in PHP in order to set an image block's source as an asset in a plugin's contents.
• Bits are parsed and presented to block implementations inside the editor, but a block author need do nothing in order to support Bits inside a block. All existing blocks become candidate containers in which Bits will appear.
• Bits are processed in PHP through register_bit() function or one similarly named. The input arguments will be: the Bit name and attributes, whether the Bit is found inside an HTML attribute or in markup, and the context attributes provided by surrounding blocks. The output shall be an HTML template that WordPress will render into the place the Bit is found.
• A <Bit> editor element provides block authors a way to insert Bits around content in their blocks.
• A Bit format or entity shall be created for use in RichText for inline token content.

Mockups

• Rethink "Featured Image" in the context of blocks  #27617 (comment) Setting an image's source as the featured image for a post c/o @jasmussen

Usage

Indicate that an image should use the featured image as its source.

<!-- wp:image -->
<img src="</{featured-image-url}>">
<!-- /wp:image -->

// blocks/image/edit
import { bit } from '@wordpress/bits';

const useFeaturedImage = () => setAttribute( { url: bit( 'core/featured-image-url' ) } )

// featured-image-bit.php

function render_featured_image_url_bit( $name, $attributes, $content_type, $context ) {
	return get_featured_image_url();
}

register_bit( 'core/featured-image-url', 'render_featured_image_url_bit' );

Insert day of week of post publishing time

Consider we have written "When I published this post on [Day of week of publish time], I was hungry." The snippet inside [] represents where we have inserted a Bit through RichText.

function render_author_bit( $name, $attr, $content_type, $context ) {
	$value = isset( $attr->value ) ? $attr->value : 'display_name';

	if ( ! in_array( $value, [
		'display_name',
		'first_name',
		'last_name',
		'nickname',
		'user_login'
	], true ) ) {
		// Only these author meta pieces are supported.
		return ;
	}

	switch ( $content_type ) {
		case 'attribute':
 			return get_the_author_meta( $value );

		case 'markup':
			return array(
				'<span class="author post-author"></%author></span>',
				array( 'author' => get_the_author_meta( $value ) )
			);
	}
}

register_token( 'core/post-author', 'render_author_bit' );

Related discussions

• Block Based Themes: Dynamic values in static HTML theme files #20966: How to add something like a base URL in front of an image src
• Inline Dynamic Content Solutions #21932: Inline dynamic content in blocks (this issue)
• Create a system for inserting site/post metadata into generic blocks #37753: Insert site/post metadata inside blocks without creating too many blocks
• [Block Library]: Add new Read More block (post link) #37649: Create dynamic attributes for things like custom "Read more" links

[dmsnell]

cc: @mtias @gziolo @draganescu

[mtias]

Thanks for summarizing your thoughts here. The first thing I'd like to elucidate is whether the abstract concept makes sense for all these use cases that are outlined. For example, I'm not sure the featured image case has any resemblance to a "display current year" token beyond its dynamic nature. There could be more straightforward paths if we could just mark an attribute as being dynamic under certain conditions:

• The media attribute of a Cover block when it's used to display a featured image.
• A heading block could have its content attribute mapped to "site title".

I think these are different in that they don't deal with explicit markup beyond what the block has already defined, and should be thus trivial to resolve from a block perspective because once "edit" consumes the attribute it doesn't matter where the url for an image came from. It doesn't need to provide any further fallbacks outside of setting the media attribute to a specific value if necessary.

---

For proper inline shortcodes / tokens, I am not sure we should support arbitrary fallbacks outside of plain text within the RichText instance. Curious how you see that. It seems the syntax should optimize for brevity <!-- token -->current year<!-- token /-->, especially if we can enforce a reduction to just a text identifier against the registry of tokens, without any extra attributes or explicit configuration passed through. But if we prefer to avoid any fallback, including plain text, then we can fold into <!-- token:current-year /--> and be succinct. There's a risk if the semantics follow blocks too closely that it'll erode clarity from nested blocks, if the lines is too blurry between properly nested blocks and cascade of tokens with arbitrary markup. I think tokens should not have to specify nor deal with markup.

[dmsnell]

Thank you for sharing your thoughts and considering this design.

There could be more straightforward paths if we could just mark an attribute as being dynamic under certain conditions:

One thing I've struggled with is determining how to mark that attribute as dynamic. Ideas I've explored with special URLs like token://query/featured-image seemed promising but then I found it was too difficult to preserve these values since they don't start with http or https. In fact I also considered creating "fake" URLs that look real (e.g. https://wordpress.org.token/query/featured-image, but was concerned about the impact of intentionally introducing invalid data.

That led me to look for out-of-band markers, and paired with the goal of not having to parse the full HTML document, led me to this current syntax and idea.

I'm not sure the featured image case has any resemblance to a "display current year" token beyond its dynamic nature.

Could you expand this in a different way? One thing I'm starting to believe is that the very thing which all of these use-cases share is that they are dynamic and their meaning crosses block boundaries - they are a semantic of a piece of information and not of a content type (of which blocks are).

I am not sure we should support arbitrary fallbacks outside of plain text within the RichText instance. Curious how you see that.

It's something I thought I wrote in the overview above but appear to have forgotten or omitted (intentionally at the time, maybe I forgot why). It seems like a very reasonable constraint on what dynamic tokens can do but I wasn't certain we need it.

The DOM interface proposed above is one of the things I'm least certain about on this design, particularly in the way that it returns DOM and not a string (something we could easily allow people to do). What I wanted to do was prevent people from jumping in to start doing poor DOM parsing themselves while thinking they are sound (specifically thinking about handling proper escaping and not accidentally breaking into an attribute string).

The fallbacks seem free to me and I'd find it a shame to remove them, particularly since their absence implies missing content, likely in the middle of sentences. Constraining these fallbacks to plain text seems like a very good idea, and we can enforce that through the <Token> component and token() function. Still, plenty of blocks, even core blocks, forego critical fallbacks so I don't see it being all that different for tokens.

because once "edit" consumes the attribute it doesn't matter where the url for an image came from

What do you mean by this? I'm thinking specifically about situations like arranging a template for the query block where the indication that the attribute is dynamic has to persist, because there is no "url for an image" to store. I'm probably simply not understand what you are referring to here, or in what circumstances.

properly nested blocks and cascade of tokens with arbitrary markup. I think tokens should not have to specify nor deal with markup.

My only reservation on this is that I thought it would be particularly good on the server to expose a system that's easy to expand. I see the danger when thinking about some of the problems that shortcodes encouraged, specifically large posts comprising only of meaningless codes that hide all content and structure. Blocks though (a) already provide a much more solid model onto which to build that structure and content and (b) do carry similar risks with people creating their own null/void blocks. That being said I suppose I start a bit skeptical of the true danger here at least with the things I'm thinking through.

I'm far more concerned about limiting the values a token can have in the editor itself; in fact I think in most or all cases I want the token to stand out just like any typical chip/token system. It should be obvious, it should be plain, it should be terse (not sure how that translates into HTML attributes but I suspect that can all be handled through the block inspector/controls).

---

That tokens are DOM-specific and not block-specific is probably the biggest idea here. That they can carry multiple meanings in different contexts is secondary, but somewhat profound in the way it can influence plugin authors' code.

function featured_images_as_slideshows( $dom, $name, $attr, $block, $context ) {
    if ( $block->blockName !== 'core/cover' ) {
       return $dom;
    }

    // ignore whatever has been stored and make every featured-image cover block a carousel
    return slideShowGeneratingCode();
}

// run last to make sure we override the core behavior
register_token( 'core/featured-image', 'featured_image_as_slideshows', priority: 100 );

[dmsnell]

I'm definitely hitting some initial obstacles to this method. Specifically I'm having trouble figuring out how to associate the token during parse with the attributes in a block. Deferring the parse of the tokens until later on in block processing (and in PHP render functions) means we have lost some association with the attribute selector.

E.g. we say that a token applies to selector: img, attribute: src but once we're processing that token we're already looking only at that img itself, meaning the selector no longer starts from the beginning of the block, meaning it won't necessarily find the part it needs to. the token selectors would need to reference the DOM node they surround, which of course loses some connection with the attribute.

[dmsnell]

thanks for elaborating. are you able to put tokens in block implementation code or only as things people insert through RichText?

[justlevine]

Doesn't require rich text, just access to whatever block attribute might contain a token in it. Again, think about how shortcodes work, you can put them in any meta field, so long as something somewhere tells WP to filter that meta field and run do_shortcode before outputting.

[dmsnell]

sure. thank you for clarifying.

think about how shortcodes work

I guess one difference here is that we have to load those shortcodes back into the editor. if all we care about is a block author having the ability to hard-code token usage then I think that system is pretty good (some details are still fuzzy to me when it comes to modifying HTML attributes).

if on the other hand we also want to make sure people can somewhat arbitrarily use tokens (consider a plugin that lets someone replace an image's src with "the current email subscriber's avatar" while generating newletters) then I find a disconnect still with communicating that token to the block implementation: it should invalidate upon loading because it didn't know to expect a token on that attribute.

a lot of this is really fuzzy to me in the details. that may be because I'm just not familiar enough or I'm over-complicating it. it's helpful to have you share your thoughts and work on the matter.

[justlevine]

if on the other hand we also want to make sure people can somewhat arbitrarily use tokens (consider a plugin that lets someone replace an image's src with "the current email subscriber's avatar" while generating newletters) then I find a disconnect still with communicating that token to the block implementation: it should invalidate upon loading because it didn't know to expect a token on that attribute.

Any block attribute that accepts a string can theoretically accept a token. From a solely-code perspective, it doesn't matter if the src is path/to/my-image.png, {{post-author:avatar id="myUser"}}, or{{my-s3-instance:url}}/${{featuredImage:filename id=${{currentPostId}}`. It just comes down to knowing what syntax delimiters to look for, and where in the lifecycle it should run.

Just like the_content, there are already filters that let us modify the block attributes, and adding a new one to the api shouldnt be that hard. Existing patterns already let us differentiate between static/dynamic, edit/save, editor/frontend output, and even if a block is/isnt selected, so the question becomes where in the lifecycle this should happen.

[dmsnell]

lol. escaping text is hard 🙃

adding a new one to the api shouldnt be that hard

I agree! unfortunately I think it's harder than it should be. I appear to have missed your comment and others because I just noticed today that Github is hiding them behind a "See more" link on the discussion. I apologize for that oversight.

Being "a string" does make sense and may be the only way to provide the universality this proposal demands. I've adopted this approach in creating an ugly syntax below. It's slightly less familiar (doesn't use [[ or {{) but should remain more unique and searchable. #token{…}#

[gziolo]

@draganescu landed #39658 that makes the Cover block dynamic by adding the option for featured image binding. It’s a good example of the block that would benefit from the standardized API discussed here. @justintadlock shared some interesting ideas on the similar topic in the following article Featured Cover Blocks and the Future of Binding Data to Generic WordPress Blocks on WP Tavern. Here are some highlights:

Many blocks are, essentially, variations on base HTML elements. For example, Post Title is merely a variation on the <h*> element, and WordPress already has a Heading block.

Cory Birdsong (@cbirdsong) opened a ticket in January that seeks to address this issue. His proposed solution:

Instead of making tons of individual blocks for this sort of thing, it seems like it would be better to create systems for using site/post metadata within existing generic blocks.

What if, instead of plugin authors creating individual blocks, they could merely offer a switch to display content via a custom data source? There are certainly some use cases beyond core WordPress where this could be handy.

[dmsnell]

this is good and hints at some of my more recent musings on the matter: is RichText enough? do we have cases where we need tokenization of HTML attribute values that a general system would help with? or is our primary and dominant need just a basic programmable token for RichText that people can enter and you can only get those inside a RichText?

if so I think we have a narrow scope where we can provide a lot of value and rely on a simple tokenization system, a part of what's proposed here with the HTML comments

[dmsnell]

another idea is something far more mundane but try specific: directly indicate in a wrapping token the HTML attribute it's meant to replace, string-values only, nestable

<!-- token:core/featured-image replace="src" -->
<!-- token:core/featured-image replace="alt" -->
<img src="some-image.jpg" alt="" />
<!-- /token:core/featured-image -->
<!-- /token:core/featured-image -->

with this it's at least easier to scan outward when parsing our attributes to see if we're currently inside a token stack and make a direct replacement. without this connection we have to know which attributes we're currently parsing and get information about the available/supported tokens. in this sense the token remains dutifully local in its definition and effect. it not only carries a kind of global semantic unto its own (independent of the blocks its in) but it also carries nuanced meaning depending on its location and instantiation in situ.

[dmsnell]

considering what @ZebulanStanphill wrote below, another variation of linking tokens to the attributes they alter could be the attribute name. while less resilient to block version updates and changes, this could simplify the parsing inside the editor which associates a token with the attribute.

<!-- token:core/featured-image attribute="url" -->
<!-- token:core/featured-image attribute="alt" -->
<img src="some-image.jpg" alt="" />
<!-- /token:core/featured-image -->
<!-- /token:core/featured-image -->

this would veer tokens back towards block-specificity making the backend more complicated during render, but would simplify the JS loading process.

[ZebulanStanphill]

Just to be clear, wouldn't that last example actually look more like this, with the token delimiters wrapping the block delimiters?

<!-- token:core/featured-image attribute="url" -->
<!-- token:core/featured-image attribute="alt" -->
<!-- wp:core/image -->
<img src="some-image.jpg" alt="" />
<!-- /wp:core/image -->
<!-- /token:core/featured-image -->
<!-- /token:core/featured-image -->

Actually, if we're tying it back to block attributes instead of HTML attributes, why not simplify this a bit by having self-closing token comments between the block delimiters?:

<!-- wp:core/image -->
<!-- token:core/featured-image attribute="url" /-->
<!-- token:core/featured-image attribute="alt" /-->
<img src="some-image.jpg" alt="" />
<!-- /wp:core/image -->

[dmsnell]

@ZebulanStanphill I like your second idea better but I don't think either one is ideal here. For the first one I think it's risky to move the tokens outside the block; they are part of the block itself and if we move it we'll have to re-specify how our parse works. We'll also risk losing token marks by doing that.

The second one is better but it lacks something that's part-and-parcel of this proposal, which is fast and reliable parsing on the backend. If the tokens are inherently DOM-related then when we find the token comments on the server we can safely replace the contents inside without more complicated HTML parsing, something that's weak on the server anyway. We'll still probably need to do some HTML parsing with this proposal, but it will be limited to a single tag or text node.

Note that if we make the token a void comment it looks like we're inserting it as a separate node (maybe we want to insert an <img> in place that represents the featured image), and if we know it's not then we'll have to start parsing and looking for the next HTML <img> and augment that.

Wrapping, while slightly more verbose, means that we can go from the outside-in on the server (find the token, then replace its content) and inside-out on the client (parse/source the attribute, and check the parent node/stack to see if it's modified by a token(s))

[ZebulanStanphill]

Something notable about blocks is that they can hypothetically be serialized into a variety of formats. It's easy to represent the block tree as JSON. The data is clearly structured, and is not necessarily tied to HTML as an output format.

Can (and should) this apply to tokens? What's the abstract representation of a token outside of HTML? How could a token be associated with a particular HTML attribute without becoming dependent on the HTML markup? Is that a bad thing?

Is this system conceptually related to blocks, or should it be considered a separate thing entirely? Or reworded: is this an extension of the block API, or is it just a more fallback-friendly version of shortcodes?

[dmsnell]

I was mainly thinking of how much more stuff you'd be looking at in the "code editor" view of the block editor

the paragraph block has always been the best argument against my bent, which is to store everything in the JSON comments. it wouldn't seem like a game-changer to do, but definitely less convenient for making basic edits from the text view.

one-to-one binding of tokens to block attributes is way better than having to look for token interpolation in every string attribute

still allow other string attributes to 'opt in' to token interpolation

I'm extremely leery about in-band interpolated token syntaxes because of the risk of breaking other legitimate needs. one of the first situations I consider is if you want to write a blog post talking about the new tokens. if you can embed the token itself in that string then you have to also reliably escape and unescape the display form of that token syntax.

HTML comments give us the ability to avoid all of the problems associated with this and remains reliable and fast to parse. in a structural form we could have an object showing the token information and also the fallback text.

it's also interesting to me to think about authors entering in-band markup in something like an image URL or caption. that would allow far more flexibility in typing these things, but I think it would include the tradeoff of significant server-side complexity on render. since most blocks only render in JavaScript we'd have to start parsing each block on the server to inspect if tokens are allowed (on render), and if so, make those replacements.

granted, someone could use the HTML comment syntax inline, but I think (a) that would break things in weird ways and (b) nobody is going to want to do that. hopefully we'll be able to use the pill visuals that most of these token systems I've seen use, that is, it won't be possible to make a syntax error when adding one.

the big problem with this is that it comes back to HTML attributes where HTML comments can't exist. surrounding the node with the comments solves one problem (discovery and parsing on render) for another (harder to source the attributes in the editor and it's a everything-or-nothing swap).

there was an approach I considered which is using Unicode replacement characters such as the tag characters, private-use characters, and the object-replacement character, but I think this is risky with how various systems will handle those. if we used those I think we could mesh the HTML comments with inline token insertion into existing strings and get the best approach of all, if, and only if, we can reliably store and load and interact with those strings with unusual characters.

I'll have to stew on this and see what we can get away with n HTML attributes. a quick test shows that the variation selectors are persisted through the WordPress save/load cycle and into the rendered page 🤔

[justlevine]

if you can embed the token itself in that string then you have to also reliably escape and unescape the display form of that token syntax.

This particular issue has already been solved for shortcodes. Duplicate delimiters ([[my_shortcode]]) and any shortcodes in <code> tags iirc.

the big problem with this is that it comes back to HTML attributes where HTML comments can't exist.

Maybe it would be helpful to define the use cases for tokens? There's a difference set of concerns and solutions for 'outputting' dynamic WP content and 'referencing' dynamic WP attributes.

E.g. If we're approaching this as if it's the missing gap from shortcodes to blocks, then the onus can be on the content creator to only put them in places that can accept valid html. By default, it's only RichText, but a dev can enable it for other block attributes and they'll can just add a HelperText letting them know tokens are supported (I see this all the time in shortcode supporting widgets and meta blocks).

[carlomanf]

if you can embed the token itself in that string then you have to also reliably escape and unescape the display form of that token syntax.

This particular issue has already been solved for shortcodes. Duplicate delimiters ([[my_shortcode]]) and any shortcodes in <code> tags iirc.

Probably off topic, but the issue is actually not solved within the FSE context. See #37545 and https://core.trac.wordpress.org/ticket/54759

[dmsnell]

@justlevine in case you missed it I feel like I exhausted all attempts at out-of-band tokenization and also realized it's just not possible to meet the needs of several valid use-cases to do that anyway.

I might try amending my proposal below to simplify unquoting the tokens. Maybe something like ##token{…}## would suffice where #token{…}# would normally appear. The syntax probably looks funny with the # on each side, but I used that in order to have a fast way to know where the token starts and ends without starting into a more computationally-heavy parser.

[dmsnell]

@carlomanf we probably won't be free of our own problems with a new inline token proposal, because we've had similar issues with block parsing anyway, but hopefully it will be easier to fix problems of un-rendered tokens due to the fact that we don't have to carry the baggage that shortcodes carry (ambiguous parsing, using shortcodes as layout, etc…)

[gziolo]

Just wanted to share my observation based on the discussion here and when talking about similar use cases in the past. I see a few different scenarios that we try to address with a single proposal. I was able to identify 4 different types:

1. Block attributes are connected with any dynamic data. Example: Cover block always showing the current featured image for the post when set in such context.
2. Block attributes can be post-processed after providing data by the user. Example: The user types the button’s text in the File block and it gets translated on the frontend.
3. Users can insert a token inside the text input connected with any dynamic data that isn’t a block attribute. Example: The user inserts a token in the RichText of the Heading block and binds it with the site title value. Example 2: The user inserts a token in the RichText of the Paragraph block that displays the current year.
4. Users can mark content inside the text input as a token to apply special processing. Example: User marks a text as translatable.

Can you identify other scenarios? Does the original proposal try to cover all of them or only a subset?

[dmsnell]

See my best-effort solution below in which I attempt to unify the use-cases by using a plaintext format which we can practically-safely replace in any context.

I think we can make this work well enough if we relax our need for full plain-HTML fallback and admit the possibility of people needing to write token syntax into places where we don't have support (or support yet).

[dmsnell]

I've been exploring another approach which is best described as "a weird Unicode trick" but has some level of promise, even though it comes with a heavy tradeoff: encode dynamic token data as inline in-band meta information via Unicode language tags [¶23.9, BCP 47] (or some other viable invisible Unicode characters).

Summary of this approach:

• Language tags are invisible everywhere meaning we can place them directly inline to mark a dynamic token. This can be by itself or surround some plain fallback value.
• Language tokens can be quickly parsed out of a document and trivially replaced in PHP or JS.
• No fundamental editor changes are necessary to support entry. We can add support in RichText and potentially the <input type="text"> boxes to auto-complete token entry, converting to the hidden meta on entry.
• Dynamic tokens inherently support copy and paste because they are embedded in the text stream.
• These are safe in HTML attributes, no different than in HTML text, with one big caveat: if they find themselves into a URL the browser will attempt to follow it to the wrong address, that being the address with the actual bytes of the token meta included.
• When looking at the plain text of a post or copying and pasting into a document it's not obvious that you are copying a token.

Granted that the editor can parse and replace these tokens the drawbacks shouldn't be insurmountable, but it means that links/URLs inside posts might be broken if rendered outside of WordPress. On the other hand I think any in-band solution will be broken at some level outside of WordPress/the editor/a conforming implementation.

The language tokens (U+E0001, U+E0020 - U+E007F) mirror 7-bit ASCII (less the control characters) and give us a unique opportunity to JSON-encode attributes inside the token, giving us a full ability to specify token type as well as token variations, such as formatting parameters for a date token. The JSON encoding would need to represent non-ASCII characters in their \uXXXX form. Further, I do not believe it's an invalid use of the language tags to do this: as far as I can tell it's within the spirit of the Unicode spec to add meta information about a span of text using these characters.

import { token } from '@wordpress/tokens';

const tokenData = {
    name: 'dmsnell/page-views',
    attrs: {
        style: 'approx' // vs. 'exact'
    }
}

const s = `This page has been viewed ${ token( tokenData, 'many' ) } times.`;

// This page has been viewed {WP-X-TOKEN-DMSNELL-PAGE-VIEWS {\"attrs\":{\"style\":\"approx\"}}}many󠁿{/WP-X-TOKEN-DMSNELL-PAGE-VIEWS} times.
s === "This page has been viewed 󠁘󠀭󠁗󠁐󠀭󠁔󠁏󠁋󠁅󠁎󠀭󠁄󠁍󠁓󠁎󠁅󠁌󠁌󠀭󠁐󠁁󠁇󠁅󠀭󠁖󠁉󠁅󠁗󠁓󠀠󠁻󠀢󠁡󠁴󠁴󠁲󠁳󠀢󠀺󠁻󠀢󠁳󠁴󠁹󠁬󠁥󠀢󠀺󠀢󠁡󠁰󠁰󠁲󠁯󠁸󠀢󠁽󠁽many󠁿󠁘󠀭󠁗󠁐󠀭󠁔󠁏󠁋󠁅󠁎󠀭󠁄󠁍󠁓󠁎󠁅󠁌󠁌󠀭󠁐󠁁󠁇󠁅󠀭󠁖󠁉󠁅󠁗󠁓 times."

s.split(/[\u{E0001}\u{E0020}-\u{E007F}]+/ug) === [
    'This page has been viewed ',
    'many',
    ' times.'
]

This comment describes the technical approach for storing, finding, and loading dynamic tokens and assumes separate UI work to make it convenient to enter and edit tokens. That approach would probably amount to dynamically replacing the tokens inside the editor with some single-character placemark styled via CSS to add the token's title and background (the "pill" form).

[dmsnell]

The language tag characters are preserved through DB operations, wp_kses, and copy/paste through different apps. Language tag support is universal as far as I can tell, probably in part due to the fact that "support" means preserves the tag characters and hides the tag characters.

[dmsnell]

While I'm adding alternative approaches it struck me that we could try and directly resuscitate shortcodes with a new syntax, albeit this would involve more care on the parsing side to ensure we don't replace text showing a token that itself is not a token.

By re-introducing a plaintext token syntax we solve one part of the problem of making it convenient to enter tokens, though we leave unsolved the issue that requiring people to enter complicated syntax is inconvenient. We still need a UI solution to enter and edit tokens.

A plaintext syntax is similar to the idea of using hidden Unicode characters but makes the following (and probably more) tradeoffs:

• The token syntax is visible and obvious so you won't accidentally gloss over it.
• Entry at its lowest form requires no special input method.
• Broken URLs that have the unprocessed token syntax in them will be obviously wrong, vs. mysteriously wrong.
• Whereas invisible characters disappear (outside of URL contexts) the visible shortcode syntax remains visible, possibly breaking posts outside of a supported environment.

[dmsnell]

yes it is @gziolo - was going to link it here after I get a couple more steps in, notably the hooks and filters necessary to make it all work (plus a more-complete parser)

[gziolo]

I think this PR has already enough to help visualize the ideas discussed in this thread. I'm still not sure what to think about the https://token.wordpress.org/ prefix alone, but overall it is looking very promising. I only hope that the further discussion will focus on the general idea, as this prefix should speed up building the initial prototype because it makes pattern matching in the parser simpler by removing the risk of replacing something else in the attribute.

[dmsnell]

yeah I still want to do extra testing but I can recall a few cases in my head where we check if ( ! preg_match( "~^https?://~", $url ) ) { … }

#token{…}# is a valid relative URL, but I would like to make sure we don't make it easy to strip it out. the other option is we could use this on an as-needed basis with prefix="https://token.wordpress.org/" in the token args. that's more reliable in one sense but extra verbose when it's there.

[dmsnell]

We definitely have issues if we run into esc_url() but if we replace the tokens before then it should be fine.

Noting that we still have to worry about context of an attribute vs. HTML but also we don't because esc_attr() and esc_html() are identical functions (probably why it's so confusing knowing which one to use).

So I think the best idea is to include an optional context specifier that we can reserve for future use in case we need it - a context sigil.

outputs the featured-image URL for HTML display

#h{featured-image}#

outputs the featured-image URL for inclusion inside an attribute

<img src="#a{featured-image}#">

can remain un-implemented but presumably if we let people write user-oriented JavaScript this would escape the token output for that context.

#j{featured-image}#

[dmsnell]

The use of the HTML API removes the need for the sigils and breaks us through a number of challenges that were present when working on this concept in the past. My current proposal for a Bit syntax relies on what I'm calling "funky comments," which are invalid HTML tag closers that, according to the specification, turn into comments in the DOM when parsed. There are a few unique aspects of this invalid tag closer that are beneficial to Bits:

• the invalid markup ends at the first > following the start of the token. inside can be any character and it's all swallowed into the comment content.
• the first letter following </, as long as it isn't a-z, can be anything and so can be used for a clear visual indicator to create many different kinds of HTML placeholders. it's a resuable syntax for a number of related token needs.
• because these invalid tag closers are a kind of twisted valid syntax with clear and uniform rules for handling them, they cannot break HTML structure in the way other syntaxes can, and browsers hide them by default if they slip through to the rendered page.

the syntax doesn't have the same magical properties in an HTML attribute that they have inside markup, but given that I've found no magical syntax that works inside an attribute, it seems like a fair compromise here. WordPress can declare the invalid tag closer syntax inside an HTML attribute to connote a Bit or other token/placeholder, and if someone wants to use that literal value inside an attribute it will need to be escaped in some way, perhaps with a doubling of the opening angle bracket, e.g. <</{not a bit}> rendered as <</not a bit}>whereas</{a bit}>` would be rendered as a hidden comment or replaced by Bit substitution).

[draganescu]

Intro

I've been thinking about this and I struggle to land on which generic problem are we trying to solve here. Considering that Block Filters client-side and render* filters server-side allow us on the fly editing of block output, we should assume any non-generic problem "I need a dynamic foobar in my block content" can be solved individually by block authors. This confusion I have also ties with what @mtias summarised above, that dynamic attributes and inline tokens are two distinct problems.

Dynamic attributes only need "standardisation"

One thing I've struggled with is determining how to mark that attribute as dynamic.

What if simply introduce a type "function" to any attribute. We currently support:

• null
• boolean
• object
• array
• string
• integer
• number (same as integer)

We could add a "function" type, or something, and call that to return some thing or do whatever. This can be solved through any of the current filters, but this would be a generally available solution, a standard way of doing it. The "function" type would support a "default" prop for fallback if the "callback" prop function does not execute, does not exist or whatever. In the featured image example the "default" could be a placeholder image.

I think, today, for "dynamic" attributes, whatever we do, it's only going to be a standardisation of something we can achieve anyway through what the system already provides.

Dynamic inline tokens need some API

On the other hand, dynamic inline tokens are something we don't have a mechanism for. As a note, out of all the solutions here the invisible chars one I am least a fan of 😁 .

UX wise, IMO dynamic inline tokens should be similar in interaction form to a format. The same way you select some text and make it bold, you select some text and make it "dynamic":

• Select (c) ACME 2022
• click on a block toolbar icon
• see list of available dynamic tokens
• select the one you want: e.g. "ACME copy notice".

But what if these "dynamic token formats" proliferate like shortcodes? The UI will be unable to contain them. 3 plugins adding their own things and we're done.

I think that maybe precisely considering them as limited as formats would block the proliferation, particularly since I don't think enabling shortcode like randomness is a good thing for blocks.

In this view of inline dynamic token as a format, a solution could be added by simply offering a system wide "dynamic" format, which can be extended with any callback. The format would use the data HTML tag and the value of the data tag would be the callback that offers the dynamic data. On rendering the block this tag is replaced with whatever the callback returns, but still kept with the data tag, just like any format.

• developer registers a format
• the tag of the format is data
• the value of the format is acme_copyright_in_perpetuity
  • registerFormatType adds support for value like it does for className
• the system runs at block render acme_copyright_in_perpetuity and replaces <data value='acme_copyright_in_perpetuity'>(c) copyright 2019</data> with <data value='acme_copyright_in_perpetuity'>(c) copyright 2022</data>

This we we reuse a thing that exists, extend it to support something very similar on a conceptual level, and get for free the UX of the feature.

---

TLDR; at the end:

• for dynamic attributes we could just support a new type named function that has a default property and a callback property. The callback returns the value of the attribute at render time. The blocks would remain static (not dynamic) via the default value if the dynamic rendering is unavailable.
• for dynamic inline tokens we could extend the rich text format API to include support for automatically replacing <data> tags with the value returned by the callback named in the data tag's value property.

[draganescu]

is not included :D or is it?
moving target

I was joking and complaining with these expressions that - because maybe I misunderstood the intent of the discussion - I find the requirements this token system has to fulfill to increase in number :)

One thing I missed is the need to allow the users to paste tokens in editor input fields. My proposals do not cover this goal. However, I think this is a 3rd thing to consider separately.

The gist of my approach is we should think of:

1 dynamic attributes that get their data from external source at block render/edit time
2 dynamic inline text tokens that get updated when WordPress renders the post
3 dynamic tokens as values of block attributes that get replaced at block render time

My main point is to implement these as three distinct things and not try for a one time fix all solution.

Feature 1 is for block developers. It helps the use case of featured image. A dynamic attribute of featured image will fetch the URL at render time both in edit and save contexts by calling some function. This only offers uniformity for a behavior we already have, it just adds an API for making these things the same way across blocks. This can be achieved today in all kinds of ways, but not a clearly specific one. Adding a dynamic attribute with a function type will not solve the injection problem of on the fly editing saved markup.

Feature 2 is for plugin authors and users. Inline text tokens are text data we want to change on the fly to be actualized replacing the authored text data when WordPress renders the post. It IS a Rich-text-centric feature.

Feature 3 is for plugin authors and users and it is a replacement for short codes. It is useful to put dynamic data in static attributes, your example with alt as "Our site [site-title /]". Feature 1, introducing a function type, replaces the whole thing and it is the developer's decision. However if we allow users to add the alt manually, Feature 3 is what allows them to add dynamic parts to their input.

Feature 2 may be mistaken for Feature 3 but it is not. The textual data is content we should keep in most humanly readable form possible. Hence sprinkling post content weird [weird /] [sh_232 /] stuff is worse than having some <data value="randomize_site_title_joke">YahooYolo</data> in the content of post. Short code like syntax can though live fine in attributes encoded and serialized in block markup.

--

[dmsnell]

I was joking and complaining with these expressions that - because maybe I misunderstood the intent of the discussion - I find the requirements this token system has to fulfill to increase in number :)

It's really hard to detect humor in comments like this. Also I should note again that there's been fairly evident need for a system to provide tokening and shortcode-like functionality. Maybe if you share more about your fears of growing scope we could talk about specifics. The fundamental idea hasn't changed since it began: tokens should provide a way to insert placeholders that get swapped by external data and they operate on a level independent of blocks.

My main point is to implement these as three distinct things and not try for a one time fix all solution.

Curious why you see this serving us better to have two or three systems to fulfill the same need? I think one area we're still not talking on the same level about is the role of blocks. The concept of the token doesn't need blocks. It's more or less a replacement for shortcodes that avoids the parsing and semantic ambiguities that they had, as well as address the most-frequent source of "abuse" which is nesting shortcodes for layout.

When you say "Feature 1 is for block developers" and "A dynamic attribute" I interpret these things as approaching this from the perspective "how do I make dynamic block attributes" instead of "how do I insert dynamic content in a post?" It's the latter I think has more power and the ability to unify all these needs in one harmonious system.

The token system isn't that complicated from a technical standpoint apart from the UI bits. It's a lexical level abstraction instead of a sub-block-level abstraction, and so the big hope for this proposal is that nobody has to think about tokens in order for them to work. At worst you see the token syntax when editing a post while the replacement work is straightforward. At best you see pill representations of the tokens in the editor UI but previews show some semantically-appropriate value.

For example, when working in the query loop template maybe the featured image token should provide a fallback image indicating "this is a placeholder for the featured image of the post inside the query loop" but if editing an actual page would resolve to the actual featured image, or another placeholder if no image is set for a post.

Doing that with two or three different systems sounds like repeating all that work for things that are ultimately the same from the author's perspective.

[adamziel]

@dmsnell about parsing HTML specifically – I don't think a PHP parser needs to handle all the same edge cases as, say, WebKit. The backend receives markup that was already cleaned up by WebKit and then it runs its own cleanup procedures via KSES. It sounds pretty clean to me, so couldn't we have a lightweight parser that bales out of processing if anything seems fishy?

[draganescu]

Curious why you see this serving us better to have two or three systems to fulfill the same need?

Because it is not the same need. Adding a dynamic data source to a block is not a need with a token solution. Inline dynamic replacements could be a need with a token solution, but it's not the most elegant solution. Adding dynamic data in strings IS a need with a token solution.

I will admit my, possibly unjustified, fear that adding a system of {#token#}s which are resolved by global token_callback() functions, for all the things that happen to require dynamic replacement of data, will lead to an unholy proliferation of them, even to the detriment of blocks.

The more we make tokens do, the more flexible the system has to be. The more flexible a system is, the more it gets used as a replacement for any better but more strict system. Using tokens for input fields alone, adding formats (maybe) for inline text replacement and using dynamic attributes, is a way of guiding development to remain in the ebb and flow of how WordPress is architected, but also offer the possibility to achieve it all.

Out of the three problems I see as distinct in this discussion, the one I am most afraid of is the token in block attributes one. Maybe inline replaced tokens is not terrible and my format idea is just the same thing repackaged. But "dynamic attributes", which is not yet an established concept, should not be equalized to "place a token that you replace on render" without being sure that is what we want.

[gziolo]

The block attribute doesn't have to be represented by a dynamic token alone. There might be different scenarios where you would like to mix the token with the input provided by the user.

It can be represented inside the attribute sourced from HTML:

<p>Current year is #{date={"format": "YYYY"}}#.</p>

It could be also in the attribute stored in the block metadata that combines tokens and input from the user:

<!-- wp:heading {"className":"#{my/plugin-class}# my-other-class"} -->

It all would work the best with the UI supporting tokens in input fields and RichText.

[ZebulanStanphill]

One idea that sounds a little crazy but might actually work is to just use regular blocks for most "inline token" style stuff, and rework all the textual blocks to accept inner blocks. There are definitely a number of technical and UX questions to consider with this approach, but I think it's worth fully considering. I can't think of any insurmountable issues with the approach, and conceptually, it's the most simple, since it just reuses the existing block concepts (and thus inherits all of the existing APIs for free).

#10235 (comment)

https://github.com/stokesman/lineup/

It's worth noting that this would not cover use-cases like dynamic HTML attribute values, so those situations would still require one of the other solutions mentioned in this discussion.

[dmsnell]

theoretically we could create an inline block type and embed those, bypassing innerBlocks. the attribute problem persists, which is the big one

[azaozz]

This is a super interesting! Sorry for the long comment, but the topic touches so many things. Don't want to risk missing something.

Implementation ideas:

1. Include the "block source" in the block properties. Imho this is the best solution and will facilitate many other nice additions and features in the future. I am aware that some developers think that increasing the size of the post body is a big drawback, but note that it won't make the DB slower, just the size on disk will increase. Also note that uploading a single photo from a smartphone would take as much space as storing the sources for 600-700 posts (5MB photo plus image sub-sizes).

• This is pretty much the data used to generate the block's HTML in the editor, in JSON format. It would look like a nested associative array in PHP terms.
• Having the block source means that the block's HTML can be recreated in PHP very fast/easy. (Not sure if anybody remembers but there was that huge image block PR where I was trying to do that. Was a bit too early back then, before Gutenberg was even added to core. However things are much more developed now and think it will be quite easy to achieve this.)
• This removes the need for having tokens or a separate API. The only thing that would be needed is for when a dynamic content has to be added to simple text. Then perhaps a placeholder tag can be inserted. It will "split" the text at the appropriate place in the block source, and can be processed from PHP.

2. Tokens overlay the blocks but are still part of them. This resembles the old shortcodes more, but is implemented with predetermined, "built-in" storage for additional data outside of the HTML. Still, storing HTML strings as props of tokens should not be allowed.

• Tokens are auto generated and not intended to be user editable.
• Each time a token is needed a new string (token name) is generated. That name is made unique only for inside the block.
• Then a placeholder is inserted in the block's HTML, and the token is added to the block properties. Any additional data needed for the token is added to the block properties too. Something like <!-- wp-token-01 --> would probably work. I also really like the idea about using UTF-8 chars there, just need to make sure these chars won't be mangled by saving to the DB (there were some DB charset problems before, hope it's all in the past now).

The references in the block props that hold data for the tokens can probably be plain objects (maps):

{
    'wp-token-01': {someData: value, moreData: value2},
    'wp-token-02': 'somethign here...',
}

When the block is parsed the tokens will be replaced with the output of a callback function. Simple str_replace() would work as the names are well formed and unique.

This seems simple enough and also powerful enough to meet any needs for adding dynamic content to static blocks. The main idea is that all of the above is part of a predetermined API, i.e. no random token names, no direct user input, etc.

[draganescu]

@azaozz can you write some pseudocode of how "Include the "block source" in the block properties. " looks when saved? I think @adamziel explored something similar.

[gziolo]

@dmsnell has a different proposal in #42015. Example for the most complex token syntax:

#{"token":"sportier/stat", "value":{"sport": "basketball", "game": "latest"}, "fallback": "TBD"}}#

This format can be embeded in both the HTML comment of the block as a value (or part of the value) of one the attributes, but it also can be stored anywhere in the HTML saved for the block, in effect also as part of sourced attributes stored in the content.

The biggest difference to using HTML comments like <!-- wp-token-01 --> is that the token would never be visible in the content. In the proposal from @dmsnell you need to ensure those tokens are removed/replaced before serving the content.

[adamziel]

@draganescu this could be related: #42485

[dmsnell]

I've updated my proposal to use "funky comments" ¹ but retain most of the syntax elements presented in #42015.

this is a beautiful quirk in that invalid HTML tag closers are interpreted as HTML/DOM comments until the nearest >. it allows us to embed the JSON-like Bit information inside a placeholder that disappears when in an unsupported environment.

WordPress Core currently removes these as will probably the HTML parsing inside the editor, but we should be able to open the mechanisms to allow them through.

[asafm7]

@dmsnell @azaozz

This is interesting and maybe more relevant than ever since the feature of inline shortcodes is unclear: https://core.trac.wordpress.org/ticket/58333

Where does it stand? do you think it will ever be picked up?

I believe #42485, in its current state at least, isn't a solution.

[dmsnell]

@asafm7 much has changed since I first created this issue: namely the HTML API in WordPress was born which gives us major opportunities to be context-aware when parsing.

Where it stands is that I think a long-term solution is going to come through the HTML API, but it won't be ready in the short term.

A few of us have started talking about the term "Bit" to describe these tokens, these things that carry semantic meaning independent of the blocks they are in.

Work on this has been ongoing since creating this issue, but it's slow.

[asafm7]

Thanks @dmsnell .

[lgladdy]

Hey folks 👋

Liam from the Advanced Custom Fields (ACF) dev team here. I wanted to kickstart our input into this conversation, and see where our aspirations are overlapping and where we can contribute directly to Gutenberg rather than doing all the work inside ACF exclusively.

Getting (simple) meta fields as inline tokens (or Bits, if that's the term we're going with!) with live previews is our end goal.

The live preview is specifically the part I think we're going to need to add stuff to Gutenberg in order to enable. I've looked at the autocompleter system which currently only supports usernames, and wanted to achieve something similar to that, maybe with $ as the prefix to trigger, then likely a data store, combined with AJAX lookups for fields not present on the UI to select. @tkapler's mockup over on the main Custom Fields/Core Blocks ticket is a similar UI we'd like to achieve: #51373 (comment) - although, in our minds we would present this as the live data just with an outline and tip to indicate it's a live replacement.

I really like @dmsnell's example implementation of a token from #42015 - that seems to be logical - and seems like it's more viable now the HTML API is more fully featured? Specifically, that token system could allow us (and other third party devs) to add additional attributes via hooks from additional components in the sidebar when the token is selected in line, for example supporting things like "Format" inputs for those who want to transform a date/time into a specific format - replicating functionality that's currently provided by legacy jquery-datetime pickers.

The core of this issue is currently spread around a few different issues and discussions - and arguably my input is better suited to this one, but I think we need a bigger picture solution as I'm not sure any specific part of this system could work without the others being in place too.

[dmsnell]

Since I've long promised updates here I'm finally delivering some news that I think is exciting.

The HTML API has changed everything about Bits. It now provides us with a way to know the context in which a Bit is found, whether inside an HTML attribute, whether in markup, whether inside a SCRIPT element, etc…

It also provides us a novel way to allow some but not all markup as a substitute for a Bit. For example, we can allow <time datetime="2023-11-08T16:12:00-0700">two hours ago</time> but reject a function attempting to replace a Bit with an image gallery. This is possible through templating and heuristics we can apply to template content. We can also reject substitutions which introduce SCRIPT or STYLE elements, such as if a plugin attempts to render a Bit with <script>alert('boo!')</script>.

We can also process the output from a Bit's rendering function and prevent that output from bleeding into the page or changing the container in which the Bit is found. For example, if a plugin attempts to replace a Bit with </p></div></main>My sneaky content then WordPress can reject it.

The new proposed syntax relies on what I call "funky comments" which are invalid tag closers. HTML defines that if you find what looks like a tag opener but the tag name is invalid, then it should treat that as plaintext. For example, <3 > war looks like it is a tag opener for a tag named 3, but 3 is an invalid tag name, so this is rendered just as written. For tag closers, however, the situation is different. The entire span from the start of the invalid tag closer until the first > shall be interpreted as an HTML comment and inserted into the DOM tree a such. It will be hidden from the page. Any sequence of characters is possible inside of that span, as long as it isn't a >.

This funky comment provides the opportunity to build a flexible placeholder or token that is handled in a deterministic and universal way in HTML-compliant browsers, cannot be nested inside of itself, and cannot otherwise break its surrounding HTML any more than a normal typo would. It can't break HTML structure. It's fundamentally benign because it disappears from the rendered page if it isn't replaced.

</{ … }> ⃪ this is a Bit. the stuff inside of it determines its type, attributes, and fallback.

Inside HTML markup, a Bit is an existing syntax element. Inside an HTML attribute it's plaintext. Inside an attribute the </ sequence should be escaped to avoid interpreting the following content as a Bit. The escape is a double <<. Since these are attribute values they would be encoded as named character references, i.e. title="I was </{bity bit bit}> when I wrote this."

The WordPress HTML API can handle all appropriate escaping of HTML values on substitution automatically. Render functions for Bits thus return an HTML template instead of a string that will be inserted into the page.

A template takes one of several forms:

• a string for static content
• an array of two elements: a template string with placeholders and then an array of named values for the placeholder

Here's what that looks like:

function render_my_bit( $bit_type, $attributes, $content_type, $context_attributes ) {
	if ( 'attribute' === $content_type ) {
		// The HTML API knows this is found inside an attribute, so the result
		// will be `plaintext no matter <strong>what</strong>`
		return 'plaintext no matter <strong>what</strong>';
	}

	if ( 'my/static-bit' === $bit_type ) {
		// Since this occurs within markup between tags, the result will be
		// the literal value `rich text inside <strong>markup</strong>` and
		// the browser will render it boldly, or as per the stylesheet
		return 'rich text inside <strong>markup</strong>';
	}

	return array(
		'<time datetime="</%iso_time>"></%human_time></time>',
		array(
			'iso_time'   => strftime( ISO_FORMAT_STRING, $then ),
			'human_time' => __( '%s ago', human_time_diff( $then, current_time( 'U' ) ) ),
		)
	);
}

Note that the template syntax also relies on "funky comments" to define named placeholders corresponding to the values in the supplied array. The HTML API knows how to properly escape these based on where they are found, so it's possible to reuse a value in different contexts.

function render_link_bit( $bit_type, $attributes, $content_type, $context_attributes ) {
	return array(
		'<a href="</%url>"></%url></a>',
		array( 'url' => '/search?q=<🔎>' ),
	);
}

This will produce the following HTML.

<a href="/search?q=%3C%F0%9F%94%8E%3E">/search?q=&lt;🔎&gt;</a>

[ZebulanStanphill]

It would probably be good to add a paragraph or two explaining why this "funky comment" syntax is better suited to this purpose than standard HTML comments. I assume it is because the funky comment syntax is more compact and easier to detect and prevent from being stripped prematurely?

[dmsnell]

Thanks @ZebulanStanphill - I don't know when I'll get around to updating all the places, but the short answer is that you're not far off. Regular HTML comments would be fine, but would also require that WordPress find a good way to differentiate them from other normal HTML comments, plus they can be a bit expansive for short inclusions.

<a title="Go to: <!-- wp-bit:post-title -->">

The funky comments are invalid HTML. This means that nominally we shouldn't expect to encounter them from outside of WordPress. If we do find them, and if their internal syntax matches Bits syntax, then it's extremely unlikely we have found a false match.

The additional characteristic that the first character is special, which is what differentiates a funky comment from a closing HTML tag, provides a natural method for differentiating different kinds of inline tokens. This could easily be added with HTML comments, of course, but IMO from the subjective human perception angle, it's more readibly recognizable with funky comments.

Type	Funky Comment	Comment
Bit	</{post-author}>	<!-- wp-bit:{post-author} -->
Placeholder	</%avatar_url>	<!-- {{{avatar_url}}} -->
Translation	</__( "Like Post" ); ?>	<!-- translate:"Like Post" -->

I'm only tossing out some ideas here for possible comment syntaxes, but using normative comments raises questions in my head that the funky comment syntax doesn't, particularly around signaling and false-positives.

From the standpoint of the HTML API they are now equally-easy to find. The HTML API changed everything in this proposal related to syntax as it's fast enough and reliable enough to detect any kind of HTML token or sub-syntax.

[lgladdy]

Can we apply a similar namespace to Bits that we do block bindings? ie: <!-- wp-bit:{namespace/source:key} --> (or </{namespace/source:key}>)

This would mean the UI for selecting a bit could know all registered sources and let the user pick that, then the source could provide a list of all the bits available.

For example, a core bit source could be "core/post-meta" like a binding, you'd hit the trigger for whatever opens this UI, it would list all the sources in a select, defaulting to the last used source, then let you pick from a list of available "bits" to display at that point.

In ACF's case, we can register "acf/field" as a bit provider, some callback function in the registration lets us return all the valid bits available, and then once the user has selected, it would save <!-- wp-bit:{acf/field:field_name} --> (or </{acf/field:field_name>) which would then use the display callback to get the actual value for that bit.

Each bit source would need to provide two callbacks, one for available bits, one for the value of the selected bit.

[dmsnell]

Yes absolutely @lgladdy, and that has always been part of the plan. However, I think there's also room from the human-perspective to allow some implicit namespacing for Core. Unlike block comment delimeters, I do expect to find that people often would rather type out Bits directly rather than use a UI, so I haven't only represented the fully-verbose forms in all my examples. namespacing can be viewed as part of the minimal specification.

[lgladdy]

With block bindings shipping in 6.5, I'd love to try and get some momentum on carrying on with the bits proposal here.

It feels like block bindings are going to highlight how hard it is to output those same sources in the middle of a core block like a paragraph, so this would be a great opportunity to push this forward I think!

[dmsnell]

Good news @lgladdy! Updates to the HTML API in WordPress 6.5 bring us a step closer, as it introduces the concept of "funky comments" which makes it more viable to identify and replace Bits with the proposed syntax.

[jarekmorawski]

I ran into Bits while working on the new product page blocks in WooCommerce. We’ll need them to surface dynamic content, like the name of the currently viewed product in the add-to-cart notice or the out-of-stock message. I thought I could help Bits take shape and after learning more about the feature, I created a prototype using the existing link control UI.

Note: the video has audio. I talk through some of the design choices and explain the rationale.

bitsux_i1.mov

Why link control? Firstly, it makes logical sense because Bits are like enhanced links. Additionally, a text string cannot contain both a link and a Bit, so consolidating the UIs reduces confusion.

Here’s a quick rundown of the UI and UX:

• Bits are displayed as clickable pills in-line,
• the pills’ appearance matches the text style, giving users an idea of how the data will look on the front-end, which they can edit as needed
• users can add a Bit by clicking the Link button in the block toolbar,
• by default, the standard link UI is displayed; users can click the new Resources link (name WIP) to access custom fields and variables (also WIP),
• resources have types, such as custom fields, date, time, people, calculations, metadata, etc. (the exact definition of Bits is still being clarified),
• resources can be searched just like links; clicking an item immediately inserts a Bit into the canvas,
• clicking the Bit opens a contextual menu with additional settings. For example, if the Bit is of the date type, options like date format and locale will appear,
• each Bit comes with a fallback text field, allowing users to input a string that is displayed if the Bit cannot be rendered for any reason,
• users can swap one Bit for another from the same menu.

Future explorations

As a future improvement, we'd explore adding a keyboard shortcut to let users quickly insert Bits similarly to block. We'd use // or [ as a way to quickly open the Bit list. It'd work just like the in-line block inserter, meaning that users could continue typing to search variables and custom fields.

We'd also explore displaying the actual Bit data in the content instead of just the popover.

---

Bits are a fairly new concept and it isn’t entirely clear how they connect to other features, like Block Bindings. They’re one and the same, though I don’t think we can develop a coherent UI that will work well for both. I look forward to hearing your thoughts! What in the prototype above works and what doesn’t? What did I miss?

[jarekmorawski]

I fully agree, they should share the DNA. Here, I aligned with the link control UI, but if we decide to decouple Bits from links, we should use the same UI as in the popover in the Connections panel in the Inspector.

[gziolo]

I believe @cbravobernal started working on a prototype for connecting block attributes with block binding sources using the sidebar’s panel that ships in WP 6.6. I even saw some initial demo that was very promising. It would be great to compare ideas 😀

[cbravobernal]

I uploaded a draft:
#62880

Can we have access to the Figma prototypes? Maybe we can use them to do some experiments faster than coding.

[jarekmorawski]

Sure! Here's the Figma. Warning: it's a bit messy.

[cbravobernal]

Like my PR 😆

[gziolo]

There is now a formal proposal from @dmsnell posted on Make Core: Proposal: Bits as dynamic tokens.

[eric-michel]

@jarekmorawski your proposal is both extremely thorough and amazing! This would fulfill a dire need that we've had for years now.

My main concern is the ease of creating custom Resources (or whatever they're eventually called) in a theme or plugin. It would be critical that (at least most) custom Resources be creatable with only PHP and something like a resource.json configuration file.

Custom block development has required WP developers to delve into React and huge NPM libraries + a build process to do things the right way (outside of plugins like ACF that provide a PHP-only mechanism for scaffolding custom blocks), even to make relatively simple blocks.

This has been frustrating for us in some cases because of how difficult it might make maintaining one-off blocks (returning to them potentially years later and trying to use an old build process with outdated NPM packages etc).

Custom Resources would very often be one-off additions to a child theme for an individual site, left untouched for potentially many years before needing any tweaks. Having some kind of build process with a bevy of dependencies will make that a nightmare.

Long story medium: developers will need a way to rapidly build durable custom Resources that don't require a bunch of tooling. It needs to be as simple as creating a Shortcode.

[gziolo]

Thank you for sharing your concerns @eric-michel. That looks like a feasible scenario in regards to Bits but also Block Bindings. Well, it really how far we want to go with the user interface. If we are happy with the visual indicator like the pill that is only a placeholder for dynamic data then registration for the types of Bits can happen on the server. Things get a little bit more nuanced if we want to offer the preview for values, but I’m sure we could figure out how to compute it on the server and show on the client. For Bits if would offer inline editing we need to provide a way to register custom JavaScript code that handles server requests to update data.

[jarekmorawski]

Thanks for all the comments so far. Much appreciated! I updated the flows to consider the existing Block Bindings UI and other applications for bits, like metafields. I also looked at the Connections panel to see if we can connect the dots between the proposed UX for Bits and the existing flows. Curious to hear your thoughts!

New inline Bits UX

bits_inline.mov

Bits in metafields like Alternative Text

bits_fields.mp4

Updates to the Connections UI and how it connects to Bits

bindings_panel.mp4

cc @mtias @jasmussen @gziolo

[cbravobernal]

I'm doing a small experiment here for block bindings:
#62880

[gziolo]

I’m looking forward what you figure out together for block bindings. That would be the best next step as it might still take some time until we will be able to build full prototype for Bits. The good part is that the flow for connecting with post meta should be very similar in both cases.

[BenBroide]

@jarekmorawski looks amazing
In which pool request can I see the code?

[jarekmorawski]

It's a Figma prototype. There's no live code yet.