-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
upgrade issue #42
Comments
Hi @hooray4me! Sorry by late. Today I published versions 4.0.0 and 4.0.1 of the chart which contains some important changes. I recommend that you read and test. The HA mode of the Zabbix Server can be disabled with the following values:
HA mode only works with two or more Zabbix Server replicas. |
Today I tried to upgrade zabbix from 6.0.9 to 6.4.7 and even with
it still starts in HAmode:
Zabbix upgrade documentation says:
to
Changing docker images gave no result so I assume helm somehow defines P.S. removing ZBX_HANODENAME or setting it to null didnt take any effect |
So in the end I was able to upgrade from from 6.0.9 to 6.4.8 Problem was with undocumented parameter ZBX_AUTOHANODENAME which is hardcoded into chart, is always present on pod and is responsible for starting server in HA mode. Interestingly enough I could set
only without any other parameters in
So I did two deployment cycles - one without any parameters except ZBX_AUTOHANODENAME, and after DB was upgraded second cycle with all usual parameters without ZBX_AUTOHANODENAME. |
It is by design that the Zabbix server does ALWAYS start in HA-Mode even with Replicas set to 1. This is in order to make sure that a scale-up does just work, and has, at least didn't have when I developed that part, no negative effect except being a "HA cluster with just one node". |
Sorry, I did not read carefully. So, the problem is that apparently recently Zabbix server does not accept to upgrade the database if run in HA mode. This is actually new to me.
|
Yep, exactly - Zabbix server does not accept to upgrade the database if run in HA mode. |
I am in incubating phase for finding a solution :) |
for me, setting UPD: and setting |
An upgrade from 6 to 7 unfortunately fails (well it actually doesn't fail but it doesn't complete entirely) when using TimescaleDB due to the fact that the
I am wondering whether the best way to solve this once for all is to create a post-install and post-upgrade hook job that handles all the database schema relevant tasks. Up to now we have one Job, simply being deployed with the Chart and only taking care of initializing the database. The good thing was that it was not needed to create a custom image for that, just a bit of sed-magic. I think this has to be redesigned entirely, also for future use cases, having ONE custom image taking care of:
It should be built as a custom image, or at least using an entrypoint script mounted as a configmap or such, but the image should be based on the Zabbix Server image (needed for the actual Upgrade of DB schema). From my point of view, this should also fix the above found problem when Zabbix Server is running in HA mode. Any more comments on this? Will investigate further during the next days. |
@fibbs I was able to upgrade from Zabbix 6.5 to 7 with following steps.
Now when scaling the server back to original replicaCount value of 3 I get the following error Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(Job.spec): unknown field "metadata" in io.k8s.api.batch.v1.JobSpec Same error occurs when deploying from zabbix-community/zabbix or from local clone. Looking into it I see an if statement that affects how things are deployed depending on the replicaCount, so will look to understand this more. Once found I am guessing a PR with the same if statement could be applied to disable HA automatically for replicaCount: 1 so the ZBX_AUTOHANODENAME is not applied. |
When the server is started in single mode, it automatically upgrades the DB it self, and therefore I am questioning the need for the job at all, if Zabbix has changed it behaviour as mentioned in a comment above. By adding a false condition to the top of the job template as well as only applying ZBX_AUTOHANODENAME if replicaCount was greater than 1, I was able to use the chart to upgrade from 6.5-7. I have made a PR #102 in case it helps someone else, but I cannot comment on the validity of removing the Job entirely beyond "it worked for me" |
thanks @crowleym, indeed that's exactly the way I did upgrades, but it is a bit "hacky" and shouldn't be this way, which is why I am working on a good solution. I have an almost-working solution here in my lab, with one or two challenges to solve. One of them is to start a zabbix_server process to only upgrade the database schema and then stop, which I will try to achieve with a hacky "start process in background and loop reading its STDOUT" kind of construct. The solution will work as follows, briefly:
This is almost exactly the same as it is designed to work right now, with the following changes:
That should work fine then and without manual intervention. Of course, it would be awesome if Zabbix themselves would implement a Stay tuned, an upgrade will come. |
Hi @fibbs , I wonder if you managed to raise the issue with Zabbix SIA, if there is a support ticket we could upvote? I am facing the same problem here, although I don't use this helm project, I have my own methodology with different specs. And I got stuck also with the problem. I went around looking if someone had found a solution and I see this ticket here and something related on the zabbix forums, to no avail anyhow. I came up with some ideas, but absolutely the best solution would be a flag with As I compile my own binaries and build my own images, I was thinking that I could snoop in the source code and catch the latest But this is so ugly that I am not really happy pursuing it, so maybe, I would patch zabbix source code myself to build the image if I see that Zabbix SIA will take a long time to release a solution for it. In the end I also find it beneficial to add a new status on the HA node to inform other nodes that the database is under upgrade, they would just back off, until the node executing upgrades would just mark it finished and/or assume an active role -- that would fix the problem to avoid having a "only-upgrade-db" switch but would incur in a larger patch. If its possible , I would be glad to know the status of the conversation with Zabbix SIA and about the ticket... and I will also decide if I go forward writing / using a patched zabbix_server binary. |
Describe the bug
unable to upgrade from 6.0.13 to 6.4. I can't seem to find a way to comment out HANodeName so that the server can start up in standalone mode.
Version of Helm and Kubernetes: 1.27
Any suggestions?
The text was updated successfully, but these errors were encountered: