Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update [email protected] #1614

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

MaximilianKohler
Copy link
Contributor

@kmohrf
Copy link
Contributor

kmohrf commented Dec 6, 2023

Your commit breaks the service template unit. That %i is there for a reason. Please try to understand what you modify.

@MaximilianKohler
Copy link
Contributor Author

Your commit breaks the service template unit. That %i is there for a reason. Please try to understand what you modify.

Are you talking about ConditionPathExists=/etc/listmonk/%i.toml ? Per the referenced issue #843 it causes an error, so this guide, discussed in that issue, recommends changing it.

I didn't try without changing those lines since that person had an issue + the guide says to change them, but it worked according to the guide, so that's why I suggested changing it.

@kmohrf
Copy link
Contributor

kmohrf commented Dec 6, 2023

Yes the guide you’ve linked to recommends changing some lines and in doing so breaks a very useful feature of the systemd unit: the ability to run more than one instance of listmonk with different configurations but the same service unit.

The lines you’ve removed (and replaced) in your PR are implementing a behavior that is intentional. If there is a bug in the implementation I’m all for fixing that. But you’ve changed the entire nature of the unit.

@MaximilianKohler
Copy link
Contributor Author

Good to know. I tagged you in the issue because I assumed you know more about this than me. So please feel free to make specific suggestions/changes.

@MaximilianKohler
Copy link
Contributor Author

MaximilianKohler commented Dec 11, 2023

Apparently my listmonk service stopped on its own:

systemctl status listmonk
● listmonk.service
   Loaded: loaded (/etc/systemd/system/listmonk.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sat 2023-12-09 21:18:45 PST; 23h ago

Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:24] Not an absolute path, ignoring: %S/listmonk-
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:47] Unknown lvalue 'ProtectControlGroups' in s...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:48] Unknown lvalue 'ProtectKernelTunables' in ...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:56] Invalid device node path 'False'. Ignoring.
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:58] Unknown lvalue 'RestrictNamespaces' in sec...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:60] Unknown lvalue 'RestrictRealtime' in secti...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:65] Unknown lvalue 'MemoryDenyWriteExecute' in...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:67] Unknown lvalue 'LockPersonality' in sectio...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:71] Unknown lvalue 'ProtectKernelModules' in s...ervice'
Dec 09 21:31:37 domain systemd[1]: [/etc/systemd/system/listmonk.service:74] Unknown lvalue 'PrivateUsers' in section 'Service'
Hint: Some lines were ellipsized, use -l to show in full.

Is it possible to send an email notification when that happens? I know my CSF firewall is able to email me about events. I modified these lines to try to fix the issue:
I commented out the 2nd line here:

EnvironmentFile=-/etc/default/listmonk
# EnvironmentFile=-/etc/default/listmonk-%i

I'm not sure whether to comment out these lines:

StateDirectory=listmonk-%i
Environment=HOME=%S/listmonk-%i
WorkingDirectory=%S/listmonk-%i

Or change them to:

StateDirectory=/etc/listmonk
Environment=HOME=/etc/listmonk
WorkingDirectory=/usr/bin

I was thinking to simplify it the same way bamboowonder did, but that guide doesn't cover creating a new user for User=<yourUser>. I saw another page showing that you can run it without specifying a user, so for now I've done that.

However, as noted here #843 (comment) another way to "fix this" is by running systemctl restart listmonk after systemctl daemon-reload. Is there a way to make it do that automatically?

Oh, I think it didn't restart after the server rebooted. So this needs to be changed too:
Restart=on-failure to Restart=always or something else: https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html

Summary:

This file needs major changes to be more compatible. I suggested in the issue-thread that we could have two versions or two sets of commands in the same file with one set commented out by default.

Perhaps we could leave the current file mostly the same (and/or rename it to [email protected]) and create a new [email protected] file containing:

[Unit]
Description=listmonk email service
ConditionPathExists=/etc/listmonk/config.toml
Wants=network.target
# The PostgreSQL database may not be on the same host but if it
# is listmonk should wait for it to start up.
After=postgresql.service

[Service]
Type=simple
PermissionsStartOnly=true
WorkingDirectory=/usr/bin
EnvironmentFile=-/etc/default/listmonk
ExecStartPre=/usr/bin/mkdir -p "/listmonk/uploads"
ExecStartPre=/usr/bin/listmonk --config /etc/listmonk/config.toml --upgrade --yes
ExecStart=/usr/bin/listmonk --config /etc/listmonk/config.toml $SYSTEMD_LISTMONK_ARGS
Restart=always

# Use systemd’s ability to disable security-sensitive features
# that listmonk does not explicitly need.
# NoNewPrivileges should be enabled by DynamicUser=yes but systemd-analyze
# still recommended to explicitly enable it.
NoNewPrivileges=True
# listmonk doesn’t need any capabilities as defined by the linux kernel
# see: https://man7.org/linux/man-pages/man7/capabilities.7.html
CapabilityBoundingSet=
# listmonk only executes native code with no need for any other ABIs.
SystemCallArchitectures=native

# Make /home/, /root/, and /run/user/ inaccessible.
# If you set ExecStartPre=/usr/bin/mkdir -p "listmonk/uploads" to a directory in /home/ or /root/ it will cause uploads to fail 
# See https://github.com/knadh/listmonk/issues/843#issuecomment-1836023524
ProtectHome=True

# Make sure files created by listmonk are only readable by itself and
# others in the listmonk system group.
UMask=0027
# listmonk only needs to support the IPv4 and IPv6 address families.
RestrictAddressFamilies=AF_INET AF_INET6

[Install]
WantedBy=multi-user.target

Maybe ExecStartPre=/usr/bin/mkdir -p "/listmonk/uploads" should be changed to /etc/listmonk/uploads.

And still missing is a static directory. https://listmonk.app/docs/templating/#system-templates

EDIT:

Changing Restart=on-failure to Restart=always didn't work. It still didn't start up after a reboot:

systemctl status listmonk -l
● listmonk.service
   Loaded: loaded (/etc/systemd/system/listmonk.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sun 2023-12-10 21:48:59 PST; 1min 35s ago
  Process: 861 ExecStartPre=/usr/bin/listmonk --config /etc/listmonk/config.toml --upgrade --yes (code=exited, status=1/FAILURE)
  Process: 844 ExecStartPre=/usr/bin/mkdir -p /listmonk/uploads (code=exited, status=0/SUCCESS)

Dec 10 21:48:59 domain systemd[1]: listmonk.service: control process exited, code=exited status=1
Dec 10 21:48:59 domain systemd[1]: Failed to start listmonk.service.
Dec 10 21:48:59 domain systemd[1]: Unit listmonk.service entered failed state.
Dec 10 21:48:59 domain systemd[1]: listmonk.service failed.
Dec 10 21:48:59 domain systemd[1]: listmonk.service holdoff time over, scheduling restart.
Dec 10 21:48:59 domain systemd[1]: Stopped listmonk.service.
Dec 10 21:48:59 domain systemd[1]: start request repeated too quickly for listmonk.service
Dec 10 21:48:59 domain systemd[1]: Failed to start listmonk.service.
Dec 10 21:48:59 domain systemd[1]: Unit listmonk.service entered failed state.
Dec 10 21:48:59 domain systemd[1]: listmonk.service failed.

But if I restart the listmonk service it works:

systemctl restart listmonk
systemctl status listmonk -l

● listmonk.service
   Loaded: loaded (/etc/systemd/system/listmonk.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2023-12-10 21:54:25 PST; 2s ago
  Process: 2435 ExecStartPre=/usr/bin/listmonk --config /etc/listmonk/config.toml --upgrade --yes (code=exited, status=0/SUCCESS)
  Process: 2433 ExecStartPre=/usr/bin/mkdir -p /listmonk/uploads (code=exited, status=0/SUCCESS)
 Main PID: 2444 (listmonk)
   CGroup: /system.slice/listmonk.service
           └─2444 /usr/bin/listmonk --config /etc/listmonk/config.toml

Dec 10 21:54:25 domain listmonk[2435]: 2023/12/10 21:54:25 init.go:145: reading config: /etc/listmonk/config.toml
Dec 10 21:54:25 domain listmonk[2435]: 2023/12/10 21:54:25 init.go:273: connecting to db: localhost:5432/listmonk
Dec 10 21:54:25 domain listmonk[2435]: 2023/12/10 21:54:25 upgrade.go:64: no upgrades to run. Database is up to date.
Dec 10 21:54:25 domain systemd[1]: Started listmonk.service.
Dec 10 21:54:25 domain listmonk[2444]: 2023/12/10 21:54:25 main.go:102: v2.5.1 (a6a2b69 2023-08-15T15:49:28Z, linux/amd64)
Dec 10 21:54:25 domain listmonk[2444]: 2023/12/10 21:54:25 init.go:145: reading config: /etc/listmonk/config.toml
Dec 10 21:54:25 domain listmonk[2444]: 2023/12/10 21:54:25 init.go:273: connecting to db: localhost:5432/listmonk
Dec 10 21:54:25 domain listmonk[2444]: 2023/12/10 21:54:25 init.go:593: media upload provider: filesystem
Dec 10 21:54:25 domain listmonk[2444]: 2023/12/10 21:54:25 init.go:517: loaded email (SMTP) messenger: [email protected]
Dec 10 21:54:25 domain listmonk[2444]: ⇨ http server started on 127.0.0.1:9000

I removed that line and it still failed to start after rebooting the server:

systemctl status listmonk -l
● listmonk.service - listmonk email service
   Loaded: loaded (/etc/systemd/system/listmonk.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sun 2023-12-10 22:15:32 PST; 8s ago
  Process: 905 ExecStart=/usr/bin/listmonk --config /etc/listmonk/config.toml (code=exited, status=1/FAILURE)
  Process: 902 ExecStartPre=/usr/bin/mkdir -p /listmonk/uploads (code=exited, status=0/SUCCESS)
 Main PID: 905 (code=exited, status=1/FAILURE)

Dec 10 22:15:32 domain systemd[1]: Unit listmonk.service entered failed state.
Dec 10 22:15:32 domain systemd[1]: listmonk.service failed.
Dec 10 22:15:32 domain systemd[1]: listmonk.service holdoff time over, scheduling restart.
Dec 10 22:15:32 domain systemd[1]: Stopped listmonk email service.
Dec 10 22:15:32 domain systemd[1]: start request repeated too quickly for listmonk.service
Dec 10 22:15:32 domain systemd[1]: Failed to start listmonk email service.
Dec 10 22:15:32 domain systemd[1]: Unit listmonk.service entered failed state.
Dec 10 22:15:32 domain systemd[1]: listmonk.service failed.

I added these lines from one of the guides:

TimeoutStopSec=10
RestartSec=5

And put it back to Restart=on-failure. And that worked.

Fix for not restarting after rebooting server.
* Added static dir (please correct if it's wrong)
* Reversed changes that break the ability to run more than one instance of listmonk with different configurations but the same service unit
@MaximilianKohler MaximilianKohler changed the title Update [email protected] Update [email protected] - Fix for not restarting after rebooting server Dec 11, 2023
@MaximilianKohler MaximilianKohler changed the title Update [email protected] - Fix for not restarting after rebooting server Update [email protected] Dec 11, 2023
@MaximilianKohler
Copy link
Contributor Author

MaximilianKohler commented Dec 12, 2023

@kmohrf does everything look good now with this service and the new simple.service? The only thing I'm not sure about is the HOME in Environment=HOME=%S/listmonk-%i, and if I broke something by changing ExecStartPre=/usr/bin/mkdir -p "${HOME}/uploads".

Also, I understand that %S is a placeholder, but I'm not sure what value/location it's referencing in this script/file. I would like to add some explanation of what folders it's putting things in and reading from, but I don't fully understand how it

runs more than one instance of listmonk with different configurations but the same service unit.

@MaximilianKohler
Copy link
Contributor Author

MaximilianKohler commented Dec 24, 2023

Logs:

I'm not sure which is best, but when reviewing https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html I saw these options and added them commented out in the simple.service https://github.com/knadh/listmonk/blob/master/listmonk-simple.service file:

#LogsDirectory=/etc/listmonk
#CacheDirectory=/etc/listmonk/cache

I assume that listmonk doesn't use a cache, so I can probably remove that? I don't know how to get listmonk to save logs with that though, and this (alternative?) may be the easiest/best way (which is working for me):

ExecStart=/bin/bash -ce "exec /usr/bin/listmonk --config /etc/listmonk/config.toml --static-dir /etc/listmonk/static >>/etc/listmonk/listmonk.log 2>&1"

So I added it as an option. I'm not sure if it conflicts with anything else.

There are other ways to do it, but they vary based on systemd version #1462 (comment).

@MaximilianKohler
Copy link
Contributor Author

I switched to AlmaLinux 9 and added back many of the commands I previously removed due to them causing error messages because of an old systemd version.

systemctl --version
systemd 252

Running these consecutively:

systemctl daemon-reload
systemctl status listmonk

Still results in:

/etc/systemd/system/listmonk.service:54: Invalid device node path 'False', ignoring.

Not sure if that's the expected result though. Line 54 is this:

# listmonk doesn’t handle any specific device nodes.
DeviceAllow=False

@kmohrf
Copy link
Contributor

kmohrf commented Jul 6, 2024

DeviceAllow=False doesn’t work because it expects a device path or device node specifier. Instead one should use

DevicePolicy=closed

I fixed this in the deb-package I maintain, but forgot to add a PR here.

Fix DeviceAllow=False error.
EnvironmentFile=-/etc/default/listmonk
EnvironmentFile=-/etc/default/listmonk-%i
ExecStartPre=/usr/bin/mkdir -p "${HOME}/uploads"
ExecStartPre=/usr/bin/mkdir -p "/etc/listmonk/uploads"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't default to /etc/listmonk as a) the app should be run as a non-root user and hence, permission to write to /etc/ is unlikely b) arbitrary user files/content should not go in /etc/*.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where should they go, given that ${HOME} is incompatible with ProtectHome=True?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a bit of Googling and looks like BindPaths, ReadWritePaths are the way to go. I am not familiar with it, so will have to experiment.

Copy link
Contributor

github-actions bot commented Nov 2, 2024

This PR has been marked 'stale' after 90 days of inactivity. If there is no further activity, it will be closed in 7 days.

@github-actions github-actions bot added the stale label Nov 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants