Observing model instances that are going to be updated inside a Celery task

[samul-1]

Just recently started using this library and I'm loving it!

I am working on an app that allows users to submit JS code to solve puzzles. The code is executed inside node's vm2 module which allows for safe and sandboxed execution.

I have a model with a JSONField which represents the outcome of the user-submitted code execution. That data is returned by node. The actual execution of vm2 is done inside of a Celery task, in which subprocess.check_output is called on the node script passing in the user code. After node is done, the returned details object is saved to the aforementioned JSONField.

I made a consumer that inherits from ObserverModelInstanceMixin in order to observe changes to instances of that model, therefore receiving the results on my app's frontend. The flow is as follows: frontend subscribes to the results instance by connecting to the consumer -> frontend triggers code execution -> Celery task saves results to field on the observed instance.

Expected behavior: a message should be fired by my consumer notifying the frontend of the change on the results field in the observed instance.

Actual behavior: no such message is fired.

I'm assuming my setup is sound, as I tried changing fields on the instance from other spots of the application that don't use Celery and the update messages do get sent.

I'm assuming it has something to do with the fact that the change to the model is async and Django signals don't react appropriately.

Is there a way to work around this? Thank you in advance.

[hishnash]

@samul-1 when your updating the model from Celary im assuming you doing this through a django model rather than poking directly at the DB?

You'r correct in that that the signal might not be being detected, this will happen if your celery code path does not import your consumer. Since python is interpreted that module that containers your consumer needs to end up being imprecated so that the signal handler is registered.

To solve this you will need to add an import statement to your celery entry point that imports your consumer model to ensure they are registered?

[samul-1]

Yeah, I'm upading the data through a model, not the db itself.

I've investigated this a little more deeply (I had already tried importing the consumer and that didn't work), and it appears the problem is signals are process-specific, and celery runs in a different process than django. So there unfortunately doesn't seem to be a way to share those signals between celery and django.

[hishnash]

Which channel layer are you using? to support sending messages between processes you need to be using something other than the InMemoryChannelLayer. https://channels.readthedocs.io/en/stable/topics/channel_layers.html

[samul-1]

I am using the redis layer

[hishnash]

if your using the redid layer (and this is configured for the celery runtime) then the signalas should be captured by django on that process were you make the change and emitted over the redis layer to the subscribed consumers. When using celery is django configured the same way?