• Meanwhile years ago i’ve wrote an intro to systemd. Recently i had close contact with it, i want to write down some practical information with focus on Systemd services.*

Unit files

Systemd unit files as records of initialization instructions for each service in declarative style are a great idea. The declarative approach has a huge advantage of replacing the traditionally used per-daemon startup shell scripts. It mitigates differences between distributions and also maintenance on the time scale. See my overview over Unit types.

Services

A service unit describes how a service is managed. This will include how to start or stop, restart the service, under which circumstances it should be automatically started, and the dependency and ordering information for related software.The easiest way to determine the type of a unit is with its type suffix, which is appended to the end of the resource name.

Most interesting commands for systemctl

systemctl start | stop | restart | reload | status | reload-or-restart | is-active [name.service]
systemctl list-units --type service --all

and

systemctl status [name.service]

is your friend. It shows basic status information and several loglines of journald logs for the service.

If you are unsure whether the service has the functionality to reload its configuration, you can use the reload-or-restart command. This will reload the configuration in place if available. Otherwise, it will restart the service so the new configuration is picked up.

Unit file locations

Let’s go beyond basic usage. The first thing we should be familiar with, is where unit files are located on the system and in which order they are loaded.

TL;DR

  • /usr/lib/systemd/system/ - units provided by installed packages
  • /etc/systemd/system/ - units installed by the system administrator

or list the state with:

systemd-analyze unit-paths 
# and
systemd-analyze --user unit-paths

Unit file Locations: system mode

Maybe it needs some more words of explanation. When systemd starts its run in system mode and check a particular set of (system-mode)locations. Here is an example of this cascade from my Fedora installation:

#systemd-analyze unit-paths

/etc/systemd/system.control
/run/systemd/system.control
/run/systemd/transient
/run/systemd/generator.early
/etc/systemd/system
/run/systemd/system
/run/systemd/generator
/usr/local/lib/systemd/system
/usr/lib/systemd/system
/run/systemd/generator.late

Therefore you may put unit files in /etc/systemd/system/ and it will override something that is coming by default with the package into /usr/lib/systemd/system or is autogenerated from legacy scripts. However, that is only the beginning. If systemd is started in user mode another set of locations is processed. Usually, systemd in user mode instance (systemd --user) is launched when the user logs in for the first time. We can also check out these locations with systemd-analyze --user unit-paths. However, it’s out of scope a little bit. Check man page for more details.

Listing Service Units

We can list all service units present on the system with the status by

systemctl list-unit-files --type service

for loaded services:

systemctl list-units --type service

It might be usefull to list failed units as well

systemctl list-units --failed

Edit Unit Files

There are two approaches

  • replacement
  • drop-in

Replacement can be done e.g with

systemctl edit --full my-unit.service

This opens /etc/systemd/system/my-unit.service in your editor, copying the installed version (if it does not exist yet) from /usr/lib/systemd/system/my-unit.service. Finally on save it automatically reloads the unit.

Alternatively, to create drop-in files for the unit file /usr/lib/systemd/system/my-unit.service, create the directory /etc/systemd/system/my-unit.service.d/ and place .conf files there to override or add new options. Systemd will parse and apply these files on top of the original unit by mergin directives.

The easiest way to do this is to run:

systemctl edit my-unit.service

This opens the file /etc/systemd/system/my-unit.service.d/override.conf in your text editor (creating it if necessary) and automatically reloads the unit when you are done editing.

However, you can always revert status to default by:

systemctl revert my-unit.service

and check the overall units delta by systemd-delta, here an example from my machine:

# systemd-delta
[REDIRECTED] /etc/systemd/system/dbus-org.freedesktop.timedate1.service → /usr/lib/systemd/system/dbus-org.freedesktop.timedate1.service
[EQUIVALENT] /etc/systemd/system/default.target → /usr/lib/systemd/system/default.target
[MASKED]     /etc/systemd/system/systemd-timedated.service → /usr/lib/systemd/system/systemd-timedated.service
[EXTENDED]   /usr/lib/systemd/system/systemd-udev-trigger.service → /usr/lib/systemd/system/systemd-udev-trigger.service.d/systemd-udev-trigger-no-reload.conf
[EXTENDED]   /usr/lib/systemd/user/dbus.service → /usr/lib/systemd/user/dbus.service.d/flatpak.conf

at this point, we should understand where to find what, let’s take a look at some service units.

Service unit example

Here is an example of system service sshd. With command:

systemctl cat sshd.service

on my Fedora 28 linux i get

# /usr/lib/systemd/system/sshd.service
[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.target
Wants=sshd-keygen.target

[Service]
Type=notify
EnvironmentFile=-/etc/crypto-policies/back-ends/opensshserver.config
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS $CRYPTO_POLICY
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

[Unit] section is more or less self-explanatory. But important directives are After and Wants. These two define dependencies on other units. Means sshd should run after network and sshd-keygen targets. Also, sshd-keygen should be started, however, sshd can start also if the start of sshd-keygen has failed (Check Wants vs. Required) Of course a lot more options existing in this section.

[Service] section controls the behavior of the service. Here we already see important parts. E.g. ExecStart defines which command starts the service.

ExecStart=/usr/sbin/sshd -D $OPTIONS $CRYPTO_POLICY

that command reload the configuration

ExecReload=/bin/kill -HUP $MAINPID

and Type=notify means systemd waits until sshd notifies it about successful start over Notify service manager, while

KillMode=process means that only the main process should be killed, a remaining process in the same control-group continuing to live. If not set this property defaults to control-group, which means everything in the same control-group is killed.