Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kubernetes job runner (#347) #349

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from

Conversation

jaredjennings
Copy link

These changes implement a job runner that makes Kubernetes API calls to run Dockerized neurons as Kubernetes Jobs.

There's also a bit of cruft picked up along the way, like enabling Dockerized Cortex to connect to Elasticsearch using HTTPS. Please look out for rookie mistakes: this is my first Scala code.

@DrissiReda
Copy link

Since kubernetes is dropping docker, can this allow us to use the kubernetes cluster's container runtime? (Mainly interested in containerd).

@jaredjennings
Copy link
Author

@DrissiReda, yes. The Kubernetes Job abstraction is at a higher level than Kubernetes' Container Runtime Interface (CRI), which is where Docker plugs into Kubernetes if you use Docker. And the k3s cluster I used for development and testing of this pull request already doesn't use Docker.

@jaredjennings
Copy link
Author

I've separated better the changes that add support to the entrypoint for trusting CA certs. There are separate trusts for Elasticsearch and for outgoing web services. I have not made a separate pull request for those, but I can. They are possibly of more general interest than the Kubernetes job runner.

@jaredjennings
Copy link
Author

OK. I went ahead and split the Elasticsearch authentication and CA cert stuff out into #362. I gave the commits here a hard look, and decided that the story of progression toward functionality they told wasn't something that needed to be preserved. So I collapsed the commits into one, for easier review. The "cruft picked up along the way," which I mentioned at the top of the thread, is gone.

@tl-Bruno-Braga
Copy link

@jaredjennings any updates on this? I assume that at this point this feature is still not implemented?

@jaredjennings
Copy link
Author

@tl-Bruno-Braga, here it sits. Works for me, though I haven't taken it to production yet. I've asked a couple of times on the Discord about it.

dikkadev added a commit to dikkadev/PeekabooAV-Installer that referenced this pull request Mar 16, 2022
This commits my current progress in re-creating the docker-compose
pipeline for PeekabooAV in Kubernetes.

That includes deployments, services, and hard-coded config files for
each step in the pipeline, modeled after what was done in the
[pipeline](/sett17/peekabooav-installer/tree/pipeline).

The yamls for cortex, and the set-up job, are included, although cortex
does currently not work inside of Kubernetes, due to the missing docker
runner. There is an open [PR](/TheHive-Project/Cortex/pull/349) and
corresponding issue.

Except for above mentioned cortex, the pipeline is fully functional.
Meaning one can send an email to the postfix_tx deployment, which is
then sent to the postfix_rx deployment and then processed by rspamd and
Peekaboo.

This was tested and developed with microk8s and a single node.
dikkadev added a commit to dikkadev/PeekabooAV-Installer that referenced this pull request Mar 16, 2022
This commits my current progress in re-creating the docker-compose
pipeline for PeekabooAV in Kubernetes.

That includes deployments, services, and hard-coded config files for
each step in the pipeline, modeled after what was done in the
[pipeline](/Sett17/PeekabooAV-Installer/tree/pipeline).

The yamls for cortex, and the set-up job, are included, although cortex
does currently not work inside of Kubernetes, due to the missing docker
runner. There is an open [PR](/TheHive-Project/Cortex/pull/349) and
corresponding issue.

Except for above mentioned cortex, the pipeline is fully functional.
Meaning one can send an email to the postfix_tx deployment, which is
then sent to the postfix_rx deployment and then processed by rspamd and
Peekaboo.

This was tested and developed with microk8s and a single node.
@michielboekhoff
Copy link

I know it's been a while - but I just wanted to say thank you @jaredjennings for creating this PR. I'm disheartened that this hasn't been merged yet, given Kubernetes' popularity. For now, I'm having to maintain a fork just to run Cortex on a managed Kubernetes cluster.

@blopezpi
Copy link

blopezpi commented Aug 23, 2024

Hello maintainers, I think this feature will be great and useful for the majority of people using Kubernetes since they deprecated dockershim. Can this PR be moved to develop branch? Who is the owner of this repository to validate this PR? @To-om @nadouani

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants