Multiple websocket connections

[karoria]

Hi @terjeio,
In one of our chas, you told me that websocket will have only one connection at a time. But my general understanding is a websocket server should be able to communicate with multiple clients. Kindly clarify as I am planning to go with websocket protocol where more than one clients need to connect to UI.

[terjeio]

There is only one machine controlled by a grblHAL, you want several clients to be able to connect and send gcode at the same time?
Or are you pondering some higher level take control/release control mechanism? Currently this is done automatically on connect/disconnect.

[karoria]

If you know cncjs, there can be more clients connected to machine via LAN. UI is shown on multiple clients and coordinates are also updated on all clients simultaneously. I want to make an app like that. They use socketio node module, so sender machine becomes server in that case. But I think native websockets should also be able to do this. Not sure about embedded ws though.
For G code sending, only one client will send command, but for jog etc, all connected clients can give commands simultaneously. Controller treats them first come first serve basis.

[terjeio]

The websocket code can (IIRC) be fairly easily be changed to support multiple clients, but only one should be in charge of the machine at any give time for safety reasons. Some kind of protocol (a $ command?) could be added for that.

Note that the grblHAL core is restricted to reading commands from one input stream at a time, the only exception is when the input stream is from a SD card - then real time commands from the input stream that initiated the SD card input stream is accepted.

Multiple simultaneously output streams are allowed and supported by the core, how this is handled is up to driver/plugin code.

[karoria]

Hi Terje. I have made some progress on making customized webui. Now I am in need of multiple ws connections as discussed here. Sorry, I am poor in C language, so I cant just get it done myself as per your instructions.
I also noticed that a screentimeout/sleep in host pc drops the ws connections. How to make it persistant? Or is there a way to autoconnect when screen wakes up?
Btw, my webui shall be optimized for 7 inch touch displays, so the user can easily use it with rpi with its 7 inch display.

[terjeio]

Now I am in need of multiple ws connections as discussed here.

Can you come up with a specification for how that is supposed to work?

I also noticed that a screentimeout/sleep in host pc drops the ws connections.

It is likely that the controller drops the connection as it uses the ping/pong protocol:

Plugin_networking/WsStream.c

Lines 1073 to 1088 in a9cae63

	// Send ping every 3 seconds if no outgoing traffic.
	// Disconnect session after 3 failed pings (9 seconds).
	if(session->pingCount > 3)
	streamSession.state = WsStateClosing;
	else if(streamSession.state != WsStateClosing && (xTaskGetTickCount() - session->lastSendTime) > (3 * configTICK_RATE_HZ)) {
	if(tcp_sndbuf(session->pcbConnect) > 4) {
	tempBuffer[0] = wshdr_ping.token;
	tempBuffer[1] = 2;
	strcpy((char *)&tempBuffer[2], "Hi");
	tcp_write(session->pcbConnect, tempBuffer, 4, 1);
	tcp_output(session->pcbConnect);
	session->lastSendTime = xTaskGetTickCount();
	session->pingCount++;
	}
	}
	}

Commenting out this block of code may help - but could lead to a dead connection if the client does not shut it down properly.

[karoria]

Can you come up with a specification for how that is supposed to work?

1. My proposed system is running a rpi with 7 inch display as the main WS client (HMI) which will primarily send commands to grblHAL.
2. The same GUI should also be opened by other client devices on same LAN, to allow and facilitate for jogging, etc. You can think this of a pendant. In this case, a mobile phone will also work as a pendant for jogging. My machine is large, so it is not possible to touch the tool at the backside of job and give commands with the fixed rpi display on the front right side of machine. In such cases a pendant is advisable so that an operator can handle the operation from the back side of the machine also.
3. CNCjs also allows to connect any number of clients simultaneously (they use websockets module named socketio) and that works really perfectly. That makes it stand out of other gcode senders too.
4. grbl itself blocks most of the commands when program is in "Run" mode. So, i think that is not worry. I want to check the multiple ws which in my opinion will be great feature.
5. That said, you can send program "start" from main client and "feedhold" from another client and "overrides" also from a different client.
   I hope I am able to convey you the matter right way. Please feel free to ask. I am also planning to make my software opensource (without support).

[karoria]

Attaching some screenshots of my "WIP" client software



.

[karoria]

Your expert comments please! Basically this is not a sender. It is only a WS client running in a browser in full screen. It has all the required control buttons working nicely. It lists all files on SD in a table, you can select it and press start to run the program from it. Tabbed interface to make space for need based buttons, etc. I don't use any library/modules for this. Just plain html, css and vanilla javascript, so that it is easy for someone else to edit to their respective needs. Till now no real tests on machine, but a table testing works nice (Spindle, coolant, axis are working as expected as checked by multimeter). Larger digits on DRO is for Work coordinates and below them are machine coordinates.

[terjeio]

CNCjs also allows to connect any number of clients simultaneously (they use websockets module named socketio) and that works really perfectly. That makes it stand out of other gcode senders too.

To the same controller? Is that GRBL_ESP32 based? Others?

grbl itself blocks most of the commands when program is in "Run" mode.

grbl (and grblHAL) can only read from one input stream at a time (BTW grbl supports only one). An exception is that grblHAL (the core) does allow real-time commands from other input streams, this can be utilized by plugin code. I am not going to change the core to read from more than one input stream at a time (except for real-time commands) - if needed that has to be done at a lower level by switching input streams dynamically.

So what is needed is a detailed spec for how to handle multiple connections at the lowest level (in WsSTream.c)...

Shared input buffer? All clients can write to that and it is the clients resposnibility to shut up when another is running the machine? IMO this is bad as there is a risk of corrupting data.

Separate input buffers? How to handle that? E.g. if one connection is sending gcode to run the machine and another tries the same then what? Should the input be ignored? Or add another layer of input buffers and switch in the first one that has a complete block of gcode to process?

I believe the output buffer cannot be shared - when an output buffer is full code blocks until there is space. If one client is slow or does not read then what? Block? At least if all output is to be sent to all...

There may be other scenarioes to handle as well.

It can be mentioned that some grblHAL drivers supports what I call MPG-mode, perhaps something like that could be easier to implement? It uses signalling via a pin to negotiate who is in complete control (the sender or the MPG). When the sender is in control real-time commands are allowed from the MPG. Jogging is by sending single character commands via I2C (or UART) - these will be ignored if the controller is not in Idle mode.

It lists all files on SD in a table, you can select it and press start to run the program from it.

This may simplify things - this means there is no gcode coming from clients?

[karoria]

Oh..I see.. cncjs is a sender not a client. I have used it with grbl_esp32 via usb serial. So, controller accepts only one stream. You are right. But the server part is in the sender where cncjs is installed and that is allowing multiple clients. Ok. Now I can understand your concerns. Will try to findout other workarounds.
Have u seen my software screenshots?

[terjeio]

Have u seen my software screenshots?

Yes, they look good!

I have taken a brief look at the GRBL_ESP32 code - they have separate block (line) buffers for each client in Protocol.cpp and, AFAICT, allows commands from all clients to intermingle at any time - there seems to be no concept of which is in full control. From a safety perspective this is IMO crazy if true, you can have two clients sending gcode at the same time...

Also, it seems to me that when multiple clients connects via a websocket the last "wins", the other connections are ignored. Again, if this is true, then at least GRBL_ESP32 avoid the problem of serving more than one... Be aware that I do not know if multiple clients are allowed - you have to either dig into the code or test that to find out.

[karoria]

I will see if my ws client rpi, can work as http server for other lan connected devices.

[terjeio]

I have been thinking a bit about this and taken a look at the websocket code. It will be possible to modify (or perhaps better write an alternative?) implementation that supports multiple connections. I believe there is no need to change anything elsewhere in the codebase.

Handling the connection event is the easy part, a lot more tricky is how to handle payload data in a sane way.

Some of the complexity in handling input can easily be taken care of by letting all connections share an input buffer - but then you must be certain that only one client sends anything but real-time commands at the same time as others or you risk corrupted input. As I wrote above this can be kind of solved by having separate buffers and somehow swap in the correct buffer when a complete block/line is received. This can be achieved by changing (function) pointer(s).

Output needs separate buffers for each client - blocking (waiting) for the slowest client might become an issue that needs to be handled.

Memory allocation may also become an issue, easiest will be to have static buffers for N clients, more tricky is dynamic allocation - and this could be a bit risky on an embedded device if the heap becomes fragmented.

I guess you will encounter most of the same issues as mention above if you move the code to a RPI...

It can be mentioned that I have slightly modified the websocket code to make it easier to add code for handling multiple connections. However, adding support for multiple connections and handling the payload for more than one is not something I plan to do anytime soon.

[karoria]

Thanks for the info. I was under impression that websockets has basic nature of "asynchronous" communication.

[terjeio]

I was under impression that websockets has basic nature of "asynchronous" communication.

It depends where in the protocol layers?

I do not know what your intention is regarding real-time report requests, allow only one client to send them and send the response to all or let all send requests? If the latter is the response only to be sent to the one that made the request or to all?

If sent to all then you may end up in a scenario where a client reads data slower the the others or has simply vanished without disconnecting - what then? When the report is output it is written to a buffer that is emptied by the websocket code in another "thread", this certainly cannot be done synchronosly to multiple clients. If the buffer is full while writing output the buffer handling code blocks, or waits, for free space to become available.

I have a similar issue with sending data to multiple clients when one is connected via USB. If the USB buffer goes full (it does when there is no connection) everything stops. Too boot there is is often no good way to detect a (USB) connection coming online or going offline...
Be aware that some of the boards that supports USB has a USB <> UART chip (EPS32 has) that may mask this, this is often wired to the MCU without any handshaking. If so it means the output simply vanishes if it is not read in time.

[karoria]

Hi Terje, thanks for good insights. My initial plan was to connect multiple clients, all bidirectional. But it seems it is not easy. So, I would like to stick on one client strategy for now. I will also see if I can make the only client (in my case rpi), a server for another LAN. That way, our controllers ws stream will be intact. Further, I would like you to invest your time to other productive tasks which may be beneficial to mass.

[karoria]

Now I am in need of multiple ws connections as discussed here.

Can you come up with a specification for how that is supposed to work?

I also noticed that a screentimeout/sleep in host pc drops the ws connections.

It is likely that the controller drops the connection as it uses the ping/pong protocol:

Plugin_networking/WsStream.c

Lines 1073 to 1088 in a9cae63

	// Send ping every 3 seconds if no outgoing traffic.
	// Disconnect session after 3 failed pings (9 seconds).
	if(session->pingCount > 3)
	streamSession.state = WsStateClosing;
	else if(streamSession.state != WsStateClosing && (xTaskGetTickCount() - session->lastSendTime) > (3 * configTICK_RATE_HZ)) {
	if(tcp_sndbuf(session->pcbConnect) > 4) {
	tempBuffer[0] = wshdr_ping.token;
	tempBuffer[1] = 2;
	strcpy((char *)&tempBuffer[2], "Hi");
	tcp_write(session->pcbConnect, tempBuffer, 4, 1);
	tcp_output(session->pcbConnect);
	session->lastSendTime = xTaskGetTickCount();
	session->pingCount++;
	}
	}
	}

Commenting out this block of code may help - but could lead to a dead connection if the client does not shut it down properly.

Hi Terje, pinging again to inform you that the issue of websocket connection drop when screen is off on host device was due to macbook. Macbook doesnt give native ethernet port, so I was using USB to ethernet adapter and that's why it was dropping connection when mac screen goes off. Now, as I have connected my board to raspberry pi 4 with ethernet cable, the websocket connection stays connected even if its screen goes off. This observation is just to inform you. I have almost built my machine. I will do some test jobs and after some experience, I will share my findings here.

[terjeio]

FYI multiple concurrent websocket connections are now possible with the caveat that only one can claim the websocket "serial" stream.

I am using this to implement the ESP3D WebUI feature to dynamically switch between clients.

[karoria]

FYI multiple concurrent websocket connections are now possible with the caveat that only one can claim the websocket "serial" stream.

I am using this to implement the ESP3D WebUI feature to dynamically switch between clients.

Thanks a lot @terjeio. This is going to be super helpful to me.