New parsing fixes are a double-edged sword

[afbagwell]

Great in that you were able to track down all the issues you found with the prompt parser. However, I have a ton of finely tuned prompts that now push out different results, sometimes significantly altering general subject matter.

Automatic1111 did a similar fix to his code in the past, and he put a checkbox in the settings that allowed people to use the old, error-prone parser so they could re-capture the way their old prompts worked. Would you consider doing something like this?

Rolling back to the last commit on May 14 is a workaround, but then there's no way to take advantage of new fixes and features unrelated to prompt parsing.

[brknsoul]

I saw this in the "patch notes" discussion, but I still don't understand what's changed. Could someone give me a primer?

[brknsoul]

I know the patch notes, but what has changed? Could you give me an example?

[afbagwell]

"prompt parser was adding commas and spaces as separate words and tokens and/or prefixes"

This would imply that even a simple prompt like this: "wide angle shot, mountains, river" would be interpreted differently by the new parser based on the presence of commas and spaces being treated as words unto themselves.

If you have a long prompt, that has a knock-on effect that will also spill over into the 75 max token chunks that it will get broken up into

[brknsoul]

Ahh, so with the update, "braided red hair" should be interpreted as one phrase, rather than "braided, red, hair" 3 distinct phrases?

[afbagwell]

I think it's more subtle than phase versus single words. Vlad would have a better idea of what exactly was happening with the comma and space issues. Combined with the other note about fixing bracket matching, that could have all kinds of hard-to-track-down effects combined with the punctuation fix.

[brknsoul]

So, as I understand, nothing's really changed with how we go about writing prompts, it's just that the same prompt we used pre-patch will result in different images post-patch, even with the same settings.

Is this about it?

[vladmandic]

the same argument can be made anytime there is a fix that changes behavior - there should be a switch to preserve old behavior "for compatibility reasons". i'm strongly against that unless old behavior is proven to be better. yes, i have different philosophy about that than a1111 as i'm against having too many switches - i've already removed 50+ of them from settings.

what has changed? here's an example if someone wants to dig in:

• INPUT a (white (lion:1.4)), cat [mouse] [tiger:0.8], in a jungle
• OLD: [['a', 1.0], ['white', 1.1], ['lion', 1.4], [',', 1.0], ['cat', 1.0], [mouse:0.9], ['tiger', 1.0], [', in a jungle', 1.0]]
• NEW: [['a', 1.0], ['white', 1.1], ['lion', 1.54], ['cat', 1.0], [mouse:0.9], ['tiger', 0.8], ['in a jungle', 1.0]]

[vladmandic]

In general commas get ignored when tokenizer is done, they only waste space. Commas are for human readability only.

[afbagwell]

I will consider it if someone can demonstrate in actual image how this change made it worse for them. Not purely based on a theory.

This is a showcase of some of the work I've done:
https://www.deviantart.com/bigbluewulf

Each one of the these finished images required a lot of work within the tool, using long, complicated prompts with lots of qualifiers and weight emphasis.

For each one of these finished works I have many more starting images saved that I intend to work on or are in-progress. None of these prompts produce the same output now with the parser changes. Oh, the images are similar. But faces change. Colors change. Body proportions change. (That's a huge one, in my case.) Background scenery shifts to different locations or perspectives. The only way to work with a lot of this prior material now is to roll back to version c99c141.

Here's three examples, although this kind of thing happens with my entire history of saved images. Each pair uses the exact same prompt and interface settings. The only difference is before and after the parser changes:

shopkeeper_before | shopkeeper_after
tracker_before | tracker_after
rogue_before | rogue_after

[vladmandic]

can you send me actual prompts for one of those examples?

[afbagwell]

Sent to the email registered with your GitHub profile. I do not want my prompts posted on public forums.

[afbagwell]

The prompt for the rogue I emailed to you might have other issues. It was the only one of the three where I didn't test regenerating the original until after I made my post, and I can't get it to match the original output from when it was first generated on April 26, even if I roll the repository back to a commit made at the end of that day. The other two (shopkeeper and tracker) prompts do generate the respective "before" images when using commit c99c141 and the "after" images on the latest commit. I forwarded the shopkeeper prompt to your email if you want to use that one instead.

[vladmandic]

i went over the prompts provided by @afbagwell (i cannot post them here as requested), and as a result I've introduced a compatibility options in UI -> Settings -> Stable Diffusion -> Prompt attention parser

i did find one bug in my new parser (fixed), so I ask to give it a try one more time.
but old parser really makes a mess - i didn't even notice originally, but it totally misses parsing newline characters, so \n gets added to actual word. horrible.

btw, comparing output of new vs old parser, new parser used 123 tokens while old used 133 tokens - that's ~10% saving.

[afbagwell]

A hearty thank you for taking a look at this and coming up with a solution!

I imagine, however, a1111 is going to have to wrestle with what to do about the bugs you uncovered in their parser. They already have one backward compatibility option that undoes parser fixes. Strangely enough having that turned on is the only way I can re-create a lot of my old prompts on Lexica when it was using the SD 1.4 model.

[afbagwell]

i did find one bug in my new parser (fixed), so I ask to give it a try one more time. but old parser really makes a mess

Just ... wow. Trying your full parser, at least with my dynamic prompt templates, is definitely moving things in a fantastic direction.

I use a wildcard setup to randomly pick different ethnicities and nationalities in my subjects. Under the old parser this worked fine, but there was often a kind of similar look to some of the choices. I assumed this was a limitation of whatever model I was using.

With the fixes, the variety of faces has jumped considerably. Really happy about this! Definitely will be using the "full parser" option for any new work.

[FullBleed]

Maybe we need some kind of "parser versioning" that is saved with prompt meta so improvements can continue to be made without creating issues with previous prompts...

[Threnos]

@vladmandic So, correct me if I'm wrong, the most advanced version is "Fixed attention", then there goes "Full parser" with some mentioned bug, and a compatibility "original parser".

[vladmandic]

full parser is my parser and its set as default.
original parser is what is default in a1111.
fixed attention is what in a1111 is enable_emphasis=False - basically, forget about () or [] and just use word:strength. that is oldest.

[Threnos]

Many thanks for the clarification!

[vladmandic]

aaaand, i've just added a Compel parser, as its used in ComfyUI and InvokeAI (although they use older version and they call it "Temporal Weighting").

its not selected by default as its a bit different, but i like it:

input:

a cat+ playing with a (red ball)++ in the forest beach- (mountain, house)---

output:

[['a', 1.0], ['cat', 1.1], ['playing with a', 1.0], ['red ball', 1.21], ['in the forest', 1.0], ['beach', 0.9], ['mountain , house', 0.73]]

[Shadowghost]

Nice, did you ever think about implementing non-normalized weight calculation and CPU bound noise calculation like ComfyUI currently implements it?

[vladmandic]

i'm not familiar with ComfyUI, send me some links to read up on that?

[Shadowghost]

Main repo (I guess you know that one): https://github.com/comfyanonymous/ComfyUI
FAQ (explains the difference to normalized approach): https://comfyanonymous.github.io/ComfyUI_examples/faq/
CPU noise calculation would require lookup in the code I guess

[vladmandic]

that faq entry about normalized approach is pretty much wrong - that is up to tokenizer to normalize/not-normalize, its not up to prompt parser. can i play around with tokenizer? probably. but i'd need to see some real value of that before i dive in.

on cpu noise calculation, a1111 added it as option recently and i've skipped that as i don't see sufficient value. it slows down generate a lot and only benefit is that its slightly better reproducability of results cross-platform.

[Shadowghost]

Interesting, thanks for looking into it anyway!

I guess the value would be mainly on large prompts because the emphasis would be way stronger without normalization, imho more reflecting the real intent

[YKefasu]

i think it would be great if we can compare the parser using plot xyz script

[YKefasu]

@vladmandic is that possible?

[vladmandic]

i could add it to xyz script, but what would it do? for example, prompt (pretty girl:1.2) means nothing to compel while prompt (pretty girl)++ means nothing to auto.

[r7vz9h3]

Is this the expected behavior regarding nested weighted parentheses?

My prompt:
((apples:0.7) and (pears:1.1):1.1) in a ((wicker basket) on a [bright] and [sunny day:0.7]:1.1)

Full parser
[['apples', 0.77], ['and', 1.0], ['pears', 1.1], ['1.1 in a', 1.0], ['wicker basket', 1.21], ['on a', 1.1], ['bright', 0.99], ['and sunny day', 1.1]]

A1111 parser
[['apples', 0.77], [' and ', 1.1], ['pears', 1.21], [' in a ', 1.0], ['wicker basket', 1.21], [' on a ', 1.1], ['bright', 0.99], [' and sunny day', 1.1]]

This isn't how I normally prompt but at a glance it seems the full parser broke at :1.1):1.1). Even weirder how it applied the :1.1 correctly to apples but not to and (pears:1.1).

Not sure if this is a bug or a design choice to not support the syntax I used. And neither noticed the [sunny day:0.7] but I'm not sure if I understand correctly how this emphasis syntax should work.

Edit: the [sunny day:0.7] gets interpreted as a special legacy syntax:

Prompt schedule:
[[28, '((apples:0.7) and (pears:1.1):1.1) in a ((wicker basket) on a [bright] and :1.1)'],
 [40, '((apples:0.7) and (pears:1.1):1.1) in a ((wicker basket) on a [bright] and sunny day:1.1)']]

Edit: this was on commit 0891b30ffe235bc17fe9a2b392f2926ac3f089c4

[vladmandic]

full parser broke at :1.1):1.1)

yup, i didn't account for that in the new parser. i'll fix that.

sunny day:0.7

that's not attention, thats a schedule. its not legacy, its how it should work.

combining schedule and attention never worked as attention is unconditional and schedule may omit words.
not sure if its worth trying to fix that?

what happens with [(sunny day:1.1):0.7]?

[r7vz9h3]

Oh, I was confused by the a (white (lion:1.4)), cat [mouse] [tiger:0.8], in a jungle example above.

This is ((apples:0.7) and (pears:1.1):1.1) in a ((wicker basket) on a [bright] and:1.1) [(sunny day:1.1):0.7]:

Prompt schedule: [[14, '((apples:0.7) and (pears:1.1):1.1) in a ((wicker basket) on a [bright] and:1.1) '], [20, '((apples:0.7) and (pears:1.1):1.1) in a ((wicker basket) on a [bright] and:1.1) (sunny day:1.1)']]
Prompt parse-attention: Full parser [['apples', 0.77], ['and', 1.0], ['pears', 1.1], ['1.1 in a', 1.0], ['wicker basket', 1.21], ['on a', 1.1], ['bright', 0.99], ['and', 1.1]]
Prompt parse-attention: Full parser [['apples', 0.77], ['and', 1.0], ['pears', 1.1], ['1.1 in a', 1.0], ['wicker basket', 1.21], ['on a', 1.1], ['bright', 0.99], ['and sunny day', 1.1]]

Prompt parse-attention: A1111 parser [['apples', 0.77], [' and ', 1.1], ['pears', 1.21], [' in a ', 1.0], ['wicker basket', 1.21], [' on a ', 1.1], ['bright', 0.99], [' and', 1.1], [' ', 1.0]]
Prompt parse-attention: A1111 parser [['apples', 0.77], [' and ', 1.1], ['pears', 1.21], [' in a ', 1.0], ['wicker basket', 1.21], [' on a ', 1.1], ['bright', 0.99], [' and', 1.1], [' ', 1.0], ['sunny day', 1.1]]

[vladmandic]

ok, i just wrote a proper unit test for that method and that forced me to fix some annoying border cases.
if there is a border case missing, let me know...

Prompt: '[black] [[grey]] (white) ((gray)) ((orange:1.1) yellow) ((purple) and [dark] red:1.1) [mouse:0.2] [(cat:1.1):0.5]'
Schedule 20 
  Fixed:  [['[black] [[grey]] (white) ((gray)) ((orange:1.1) yellow) ((purple) and [dark] red:1.1)  ', 1.0]]
  Compel: [['[black] [[grey]] white gray orange:1 . 1 yellow purple and [dark] red:1 . 1', 1.0]]
  A1111:  [['black', 0.9090909090909091], [' ', 1.0], ['grey', 0.8264462809917354], [' ', 1.0], ['white', 1.1], [' ', 1.0], ['gray', 1.2100000000000002], [' ', 1.0], ['orange', 1.2100000000000002], [' yellow', 1.1], [' ', 1.0], ['purple', 1.2100000000000002], [' and ', 1.1], ['dark', 1.0], [' red', 1.1], ['  ', 1.0]]
  Full :  [['black', 0.9090909090909091], ['grey', 0.8264462809917354], ['white', 1.1], ['gray orange', 1.2100000000000002], ['yellow', 1.1], ['purple', 1.2100000000000002], ['and', 1.1], ['dark', 1.0], ['red', 1.1]]
Schedule 50
  Fixed:  [['[black] [[grey]] (white) ((gray)) ((orange:1.1) yellow) ((purple) and [dark] red:1.1) mouse ', 1.0]]
  Compel: [['[black] [[grey]] white gray orange:1 . 1 yellow purple and [dark] red:1 . 1 mouse', 1.0]]
  A1111:  [['black', 0.9090909090909091], [' ', 1.0], ['grey', 0.8264462809917354], [' ', 1.0], ['white', 1.1], [' ', 1.0], ['gray', 1.2100000000000002], [' ', 1.0], ['orange', 1.2100000000000002], [' yellow', 1.1], [' ', 1.0], ['purple', 1.2100000000000002], [' and ', 1.1], ['dark', 1.0], [' red', 1.1], [' mouse ', 1.0]]
  Full :  [['black', 0.9090909090909091], ['grey', 0.8264462809917354], ['white', 1.1], ['gray orange', 1.2100000000000002], ['yellow', 1.1], ['purple', 1.2100000000000002], ['and', 1.1], ['dark', 1.0], ['red', 1.1], ['mouse', 1.0]]
Schedule 100
  Fixed:  [['[black] [[grey]] (white) ((gray)) ((orange:1.1) yellow) ((purple) and [dark] red:1.1) mouse (cat:1.1)', 1.0]]
  Compel: [['[black] [[grey]] white gray orange:1 . 1 yellow purple and [dark] red:1 . 1 mouse cat:1 . 1', 1.0]]
  A1111:  [['black', 0.9090909090909091], [' ', 1.0], ['grey', 0.8264462809917354], [' ', 1.0], ['white', 1.1], [' ', 1.0], ['gray', 1.2100000000000002], [' ', 1.0], ['orange', 1.2100000000000002], [' yellow', 1.1], [' ', 1.0], ['purple', 1.2100000000000002], [' and ', 1.1], ['dark', 1.0], [' red', 1.1], [' mouse ', 1.0], ['cat', 1.1]]
  Full :  [['black', 0.9090909090909091], ['grey', 0.8264462809917354], ['white', 1.1], ['gray orange', 1.2100000000000002], ['yellow', 1.1], ['purple', 1.2100000000000002], ['and', 1.1], ['dark', 1.0], ['red', 1.1], ['mouse', 1.0], ['cat', 1.1]]

[TheOnlyHolyMoly]

Update from me:

The parser patch is a masterpiece, I'm almost back to my original images. thanks very much Vlad!!

Additionally I have tested reverting back to a known-working commit (several states) before the parser update and that still would not give me the original images, so I the only thing I can image it is down to a change in one of the associated dependent libraries...(git reset --hard commitID does not roll back the associated libararies to what they were on that date I guess). In fact I'm now closer to my original images with Vlad's latest update than with the roll-back to the previous commit.

I guess we have to get used to our images probably not being fully reproducible over time (too many associated components involved that will change independetly of the repo). Additionally, keeping a legacy burden too long just makes any program just very complex as all historic states of modules need to always be carried along (including error testing, fixing, dependencies, etc).

The best option - if reproducibility is key for your business - would be IMHO versionizing the repo folder of working version as backup before "git pull" and then keeping that frozen version away from updates for the sake of reproducibility. If such versionizing is your thing, you should probably consider keeping your model safetensor files outside the repo folder structure so all repo versions would access a common model library.

look at symlink or -modeldir flag.

[vladmandic]

yes, its true that some underlying libs have direct impact on outputs - for example, i still believe that transformers==4.19 have "nicest" output, but i cannot keep ancient version for that long...