Failed to update secure chat state

[weiluntong]

Expected behavior

Chatting should just work, as my friends are all on vanilla clients without mods. I'm using Paper on the server for optimizations (vanilla kept giving me "can't keep up" messages despite having 1 or 2 players on at the time of testing), but I haven't installed any plugins or datapacks, and the vanilla clients have seemed to connect without any issues until yesterday.

Observed/Actual behavior

The first time I observed this behavior was yesterday, and I saw this in the logs:

[18:16:24 INFO]: UUID of player nachocheesefries is 52682139-39f6-4909-9143-75b1640baaa8
[18:16:25 INFO]: nachocheesefries joined the game
[18:16:25 INFO]: nachocheesefries[/172.19.0.3:49072] logged in with entity id 140749 at ([world]572.3000000119209, 70.0, 1305.531163827157)
[18:16:29 WARN]: Failed to update secure chat state for nachocheesefries: 'Chat disabled due to missing profile public key. Please try reconnecting.'

However, at the time, the player had immediately greeted his friends (small server), and chat was working, so I assumed it might be an isolated incident that might resolve itself. Today, I see this:

[00:18:56 INFO]: UUID of player nachocheesefries is 52682139-39f6-4909-9143-75b1640baaa8
[00:18:57 INFO]: nachocheesefries joined the game
[00:18:57 INFO]: nachocheesefries[/172.19.0.3:60390] logged in with entity id 204887 at ([world]575.7368089439759, 70.5625, 1308.9914279460818)
[00:19:01 INFO]: <TeaBaggins> ew
[00:19:03 INFO]: <MakunaHatata> ew
[00:19:10 WARN]: Failed to update secure chat state for nachocheesefries: 'Chat disabled due to missing profile public key. Please try reconnecting.'
[00:19:10 INFO]: <nachocheesefries> shut
[00:19:14 INFO]: <nachocheesefries> shit
[00:19:18 WARN]: Failed to update secure chat state for nachocheesefries: 'Chat had an invalid signature. Please try reconnecting.'
[00:19:19 WARN]: Failed to update secure chat state for TeaBaggins: 'Chat had an invalid signature. Please try reconnecting.'
[00:19:25 WARN]: Failed to update secure chat state for TeaBaggins: 'Chat disabled due to broken chain. Please try reconnecting.'
[00:19:26 WARN]: Failed to update secure chat state for nachocheesefries: 'Chat disabled due to broken chain. Please try reconnecting.'
[00:19:29 INFO]: nachocheesefries lost connection: Disconnected
[00:19:30 INFO]: nachocheesefries left the game
[00:19:37 INFO]: UUID of player nachocheesefries is 52682139-39f6-4909-9143-75b1640baaa8
[00:19:37 INFO]: nachocheesefries joined the game
[00:19:37 INFO]: nachocheesefries[/172.19.0.3:58682] logged in with entity id 204986 at ([world]578.4084010905993, 70.0, 1306.3428790735063)
[00:19:38 WARN]: Failed to update secure chat state for MakunaHatata: 'Chat had an invalid signature. Please try reconnecting.'
[00:19:40 WARN]: Failed to update secure chat state for nachocheesefries: 'Chat disabled due to missing profile public key. Please try reconnecting.'
[00:19:40 INFO]: <nachocheesefries> Um
[00:19:49 INFO]: <nachocheesefries> I got banned from chat
[00:19:53 INFO]: <nachocheesefries> For trying to say shut
[00:19:54 WARN]: Failed to update secure chat state for MakunaHatata: 'Chat disabled due to broken chain. Please try reconnecting.'
[00:19:58 WARN]: Failed to update secure chat state for TeaBaggins: 'Chat disabled due to broken chain. Please try reconnecting.'
[00:20:05 INFO]: TeaBaggins lost connection: Disconnected
[00:20:05 INFO]: TeaBaggins left the game
[00:20:09 INFO]: UUID of player TeaBaggins is af8b9fb2-43c9-480f-833b-54b52277827d
[00:20:10 INFO]: TeaBaggins joined the game
[00:20:10 INFO]: TeaBaggins[/172.19.0.3:33522] logged in with entity id 205059 at ([world]575.0303482884933, 70.0, 1307.0327221378259)
[00:20:14 INFO]: <nachocheesefries> ew
[00:20:16 INFO]: MakunaHatata lost connection: Disconnected
[00:20:16 INFO]: MakunaHatata left the game
[00:20:17 INFO]: <TeaBaggins> fox u
[00:20:21 INFO]: UUID of player MakunaHatata is 088767b2-3118-4997-b0a4-618251efdcd3
[00:20:22 INFO]: MakunaHatata joined the game
[00:20:22 INFO]: MakunaHatata[/172.19.0.3:60992] logged in with entity id 205080 at ([world]578.0926235867769, 67.0, 1274.2269720065838)

Seemed that after that, they were speaking just fine. Initially they'd guessed swearing was banning their chat privileges, but the proceeded to test an extensive vocabulary of profanities, and the chat persisted.

Steps/models to reproduce

So because of the nature of the issue, I'm not sure how to reproduce it, as it's not even consistently banning chat as far as I can tell right now.

Plugin and Datapack List

/plugins output:

Server Plugins (0):

/datapack list output:

There are 3 data pack(s) enabled: [vanilla (built-in)], [file/bukkit (world)], [paper (built-in)]There are no more data packs available

Paper version

This server is running Paper version 1.21.1-77-master@4ff58c4 (2024-09-12T18:05:09Z) (Implementing API version 1.21.1-R0.1-SNAPSHOT)
You are running the latest version
Previous version: 1.21.1-76-e945cfe (MC: 1.21.1)

Other

No response

[kennytv]

Are you sure there are absolutely no mods or plugins involved? I don't think it's possible for Chat disabled due to missing profile public key to appear unless they use a modified client, or are using an older client version

[weiluntong]

I just verified with them today that they just use the official launcher from Mojang

[electronicboy]

[00:19:40 INFO]: <nachocheesefries> Um
[00:19:49 INFO]: <nachocheesefries> I got banned from chat
[00:19:53 INFO]: <nachocheesefries> For trying to say shut

Are you using a proxy? Do you have some chat spam/filtering system?
them saying that screams that something in your setup improperly cancelled a chat related packet and thus broke the chain

[weiluntong]

I do not have any chat spam/filtering system

I have multiple services that I have dockerized, including the vanilla Minecraft server that I've built off a very basic OpenJDK image from dockerhub, I'd post a link to the base image, but I don't know your rules on external links. I use a dockerized nginx as a reverse proxy to access them all, and handle SSL, though TCP streams don't use SSL certs. my reverse proxy also handles connections to the minecraft container. It's very basic:

# /etc/nginx/stream.d/default.conf
server {
    listen 25565;
    proxy_pass minecraft:25565;
}

Further up the chain, if you want to see where streams are declared on the same level as http, I have this:

# #/etc/nginx/nginx.conf
stream {
    include /etc/nginx/stream.d/*.conf;
}

You can actually see in the logs that everyone has the same IP, I was guessing that might be due to the reverse proxy, and it hadn't caused problems before, and it's not like I need to IP ban anyone, because if that were the case, I could just do that at the reverse proxy level and just remove them from all of my services. Again though, it's a small server for people I know personally, so I doubt I'll need to do any banning, or any chat spam/filtering.

I've entertained the idea that it might be causing problems though, and I'm looking into IP Transparency and Direct Server Return, but that's a huge undertaking that I'm not at all familiar with, especially how to translate it from physical server clusters to Docker's virtual switch.

So I was hoping you guys might have other ideas as to what's causing this, as in maybe Paper was indeed doing something that might have resulted in malformed POST requests when retrieving Key pairs or something, and it just hadn't been caught yet as it's not consistent.

[Folas1337]

I can confirm this happening on my server too but I'm definitely using a chat filtering system based on Skript.

I think it used to work fine but I can reproduce this and will try to find more information and share here as soon as I can.

The problem for me seems to be that the Skript is listening to two events: on chat AND on command. For the on chat event I can replace parts of the message while for the on command that does not work and thus the event is cancelled instead. The problem is that some commands such as /msg are really a chat message and it being cancelled results in the chain breaking as @electronicboy pointed out.

I think it used to be the case that for commands such as /msg no chat event was fired and thus it worked fine but potentially something has changed and now the chat event does fire and thus this issue arises.

As I said, I'll dig a little deeper and troubleshoot this some more to see if I can find a solution to my problem at least.

[Folas1337]

Well this even happens on 1.21 build 1 of paper so it's potentially always been this way.

I was also able to verify that this happens in any command that's a message in reality so you definitely can't cancel a command such as /minecraft:msg Folas1337 Hi. Does paper provide any way to tell apart "non-message commands" from commands which are actually messages such as /minecraft:msg and then also a way to replace parts of that message/command? Or alternatively any way to cancel the command while not breaking the message chain?

This is all pretty confusing and the fact that I also partially don't know what I'm talking about does not help the fact either 😕

Alternative solutions to this are definitely also welcome 😅

[electronicboy]

The PlayerCommandPreprocessEvent is fairly long known to be broken and will break the chain state if it's being used to cancel stuff, that is probably what is causing your issue

[Folas1337]

The PlayerCommandPreprocessEvent is fairly long known to be broken and will break the chain state if it's being used to cancel stuff, that is probably what is causing your issue

That's good to know but then I wonder what a good solution to the problem would be? 🤔 Is there any other way to cancel a command that I should use instead or should the chat plugin I'm using instead see if the command was canceled and then stop even trying to send a chat message so it wouldn't break the chain?

[Folas1337]

@jpenilla Suggested that paper could add the ability to cancel signed commands (which until now I didn't know they existed or that that's what they're called).

It sounds like a really good idea to me however as it would avoid these situations and allow more freedom as you wouldn't have to use all kinds of plugin APIs just to cancel their specific event cause they're using signed commands and when you switch plugins you have to support yet another one.

As a simple owner who doesn't do any coding this is basically an impossible task and even to a plugin developer this would mean having to support dozens if not hundreds of plugins' APIs if you want to somehow censor commands like I just explained.

I think this would really be beneficial.

[electronicboy]

Well, yea, cancelling a signed command preprocess event should update the chat state spigot however did not do this, and the last time I looked, I was still pretty dead from post-viral

[kennytv]

The issue regarding the command pre-process event is addressed here #11424

[Folas1337]

The issue regarding the command pre-process event is addressed here #11424

Will that PR fix the problem I explained? I'm not exactly sure and just thought I'd ask :)

[lynxplay]

I added a build-pr label to the linked PR, there should be a jar published in a couple of minutes if you wanna test it for your specific error case.

[Folas1337]

Oh snap that was really quick, I'll try it out then and let you know!

[Folas1337]

I added a build-pr label to the linked PR, there should be a jar published in a couple of minutes if you wanna test it for your specific error case.

It works for what I was able to try out. I did not expect this to be implemented so quickly, you guys are absolute beasts, it's greatly appreciated!! <3

[weiluntong]

Wait, sorry a lot happened in a very quick amount of time, and I didn't have any time to process and test the commands @Folas1337 was referring to in order to verify if we had the same issue.

I apologize, but I'm also not familiar with the minecraft ecosystem as a whole. Are we saying that sometimes messages register as a command, and as such it'll sometimes fail?

I went ahead and tried \minecraft:msg Aerothix0 Hi (Aerothix0 being myself), and it seems there wasn't an issue on my end? Though, it was to myself, and I'm not sure if that's handled differently from whispering to someone else or not.

I'm sorry, I'm making it more difficult because my server is so small that there's not a lot of opportunity with other players to keep testing things hahaha.... the one person I think who might be the one actually causing problems nachocheesefries is the one who gets on the least, and I suspect that he might be the problem simply because I've been closely monitoring the server and the issue hasn't cropped up since the others have gotten on, plus, he's the only one with op-permission-level set to 4. He hasn't used it for anything other than the one time he needed to teleport because his friend, who's super new to Minecraft was struggling to gather their bearings and they kept drifting further apart. At that time, the one use of command didn't break anything, but later on, it felt like everyone's stuff broke when he got on. Which... admittedly is only the two times more that we successfully captured in the logs I posted in the original message.

[Folas1337]

Hey sorry, I admittedly kind of piggy bagged your issue here because the exact same error you reported was the one I also got from what I did before.

Now let me go through your message and give some answers:

I didn't have any time to process and test the commands @Folas1337 was referring to in order to verify if we had the same issue.

You'd need to install Skript and use the exact same thing I did to be able to reproduce it, I would not really suggest doing so and also you'd come to the very same outcome.

Are we saying that sometimes messages register as a command, and as such it'll sometimes fail?

The exact opposite. Commands sometimes register as messages or rather they are signed with your key due to Minecraft's somewhat new chat system. Thus when you use something like /msg <user> Hello, it will be signed without you even realizing and that has probably/potentially lead to the problem your friend was experiencing.

I went ahead and tried \minecraft:msg Aerothix0 Hi (Aerothix0 being myself), and it seems there wasn't an issue on my end?

This is an example of a signed command BUT the error should only happen if your command was cancelled through the API which this one most likely wasn't.

The rest of your message is rather irrelevant to this problem.

I sincerely suggest you update paper to the latest build now and see if it ever happens again and until then this issue should be seen as solved.

You did not really have much to recreate it and the exact error log that you gave was the one I was able to reproduce by canceling a signed command which is exactly what broke Minecraft's key chain. We might never really find out why this happened to you or if it was some kind of hiccup. After all, the solution they just implemented should make it never happen again, at least with signed commands.

I am "un-piggy bagging" at this point and I'm satisfied with what has been done but you are the original poster so I can't (and won't ask to) close this ticket but you can if my explanation is enough for you to believe in :P

[weiluntong]

No I appreciate the breakdown actually. Being completely out of my element, it helped me to see what you were getting at and where our problems intersected as well as where they may differ (if they are indeed separate problems). I actually had the same idea as you, to just update and see if it happens again hahaha (which I did just last night). Unfortunately, without a reliable way to recreate it as of now, I have to agree that I just kind of have to hope that even if our problems aren't the same, then they intersected enough that a fix for you was a fix for me (or that it really was just a hiccup). Especially since you had a reliably reproducible bug that had the same output.

Though, I'd like a little more reassurance on my reverse proxy, either through exploration to see if it's causing problems, or maybe some enlightenment on how Minecraft handles secure signing and if it has anything to do with IPs connecting to the server (and essentially if local IPs can break the chain of trust), and I guess eventually to an explanation of where Paper comes into play with the actual handling Minecraft secure keys (if paper handles them at all). If anyone is willing to help me in that area.

[TWME-TW]

I do not have any chat spam/filtering system

I have multiple services that I have dockerized, including the vanilla Minecraft server that I've built off a very basic OpenJDK image from dockerhub, I'd post a link to the base image, but I don't know your rules on external links. I use a dockerized nginx as a reverse proxy to access them all, and handle SSL, though TCP streams don't use SSL certs. my reverse proxy also handles connections to the minecraft container. It's very basic:

# /etc/nginx/stream.d/default.conf
server {
    listen 25565;
    proxy_pass minecraft:25565;
}

Further up the chain, if you want to see where streams are declared on the same level as http, I have this:

# #/etc/nginx/nginx.conf
stream {
    include /etc/nginx/stream.d/*.conf;
}

You can actually see in the logs that everyone has the same IP, I was guessing that might be due to the reverse proxy, and it hadn't caused problems before, and it's not like I need to IP ban anyone, because if that were the case, I could just do that at the reverse proxy level and just remove them from all of my services. Again though, it's a small server for people I know personally, so I doubt I'll need to do any banning, or any chat spam/filtering.

I've entertained the idea that it might be causing problems though, and I'm looking into IP Transparency and Direct Server Return, but that's a huge undertaking that I'm not at all familiar with, especially how to translate it from physical server clusters to Docker's virtual switch.

So I was hoping you guys might have other ideas as to what's causing this, as in maybe Paper was indeed doing something that might have resulted in malformed POST requests when retrieving Key pairs or something, and it just hadn't been caught yet as it's not consistent.

if you need real ip,You just need to add proxy_protocol on;