ood-portal broken in httpd 2.4.62

[johrstrom]

Amazon linux has updated to httpd 2.4.62 and it appears the ood portal is broken and so our CI is broken, at least for Amazon linux 2023.

This is the only indication of any failure. The server simply returns 500s.

[Mon Aug 12 20:55:06.455849 2024] [proxy:warn] [pid 9512:tid 9599] [client 172.17.0.1:51742] AH01144: No protocol handler was valid for the URL /pun/sys/file-editor (scheme 'unix'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

[johrstrom]

I believe this is an actual apache httpd bug. I've file https://bz.apache.org/bugzilla/show_bug.cgi?id=69261 with a repository that can demonstrate and reproduce the bug.

[johrstrom]

Which is to say, we'll have to start skipping the amazon linux builds in the github CI. Don't know what that means for nightlies or an actual release when it comes to it.

[treydock]

Which is to say, we'll have to start skipping the amazon linux builds in the github CI. Don't know what that means for nightlies or an actual release when it comes to it.

Can skip for Github Actions but packages and nightlies are all part of Gitlab CI and they don't run any of the E2E tests so I don't think a buggy Apache should prevent the package process.

[johrstrom]

I don't think a buggy Apache should prevent the package process.

That's good news, but even so, we can't actually distribute Amazon 2023 packages unless we require a specific httpd version (or < 2.6.62)?

[treydock]

we can't actually distribute Amazon 2023 packages unless we require a specific httpd version (or < 2.6.62)?

We just depend on any version of Apache. We only depend on >= 2.4 of < 2.5:

https://github.com/OSC/ondemand-packaging/blob/b160fcef081c7397c1be15b95e4edcc083307012/packages/ondemand-runtime/rpm/ondemand-runtime.spec#L114

It would be up to the Amazon Linux admins to avoid the broken version, which is likely just a documentation mention. I'm not familiar with how Amazon does their repos if they keep older versions of packages so it's possible to downgrade.

[johrstrom]

We only depend on >= 2.4 of < 2.5:

I wonder if we need to tighten that to >= 2.4 of < 2.4.62 because 2.4.62 on any OS isn't likely to work. I filed the bug upstream, but who knows how long it'll take to fix and get distributed.

[CSC-swesters]

Hi. Interesting bug, good that it was resolved!

Would this be possible to backport to 3.1.x as well? I suppose stable deployments of OOD might hit this as well if other distros than Amazon Linux decide to upgrade httpd with the new changes.

[johrstrom]

Yea we'll need to backport this.

[giuliolibrando]

Please update the package version now. I lost 4 days debugging this issue on Apache.

[giuliolibrando]

You alse need to update node_proxy.lua file

conn.uri = uri and (r.args and (uri .. "?" .. r.args) or uri) or r.unparsed_uri

must be changed to
conn.uri = uri and (r.args and (uri .. "?" .. r.args) or uri) or r.uri