Services in systemd

Previosly (meanwhile years ago) i've wrote an intro to systemd. Recently i had close contact with it, i want to write down some practical informations with focus on Systemd services.

Unit files

Systemd unit files as records of initialization instructions for each service in declarative style are great idea. Declarative approach has a huge advantage of replacing the traditionally used per-daemon startup shell scripts. It mitigates differences between distributions and also maintenance on the time scale. See my overview over Unit types.

Services

A service unit describes how a service is managed. This will include how to start or stop, restart the service, under which circumstances it should be automatically started, and the dependency and ordering information for related software.The easiest way to determine the type of a unit is with its type suffix, which is appended to the end of the resource name.

Most interesting commands for systemctl

systemctl start | stop | restart | reload | status | reload-or-restart | is-active [name.service]  
systemctl list-units --type service --all  

and

systemctl status [name.service]  

is your friend. It shows basic status information and several loglines of journald logs for the service.

If you are unsure whether the service has the functionality to reload its configuration, you can use the reload-or-restart command. This will reload the configuration in-place if available. Otherwise, it will restart the service so the new configuration is picked up.

Unit file locations

Let's go beyund basic usage. First thing we should be familiar with, is where unit files are located on the system and in which order they are loaded.

TL;DR

  • /usr/lib/systemd/system/ - units provided by installed packages
  • /etc/systemd/system/ - units installed by the system administrator

or list the cascate with:

systemd-analyze unit-paths  
# and
systemd-analyze --user unit-paths  

Unit file Locations: system mode

Maybe it needs some more words of explanation. When systemd starts its run in system mode and check a particular set of (system-mode)locations. Here is example of this cascade from my Fedora installation:

#systemd-analyze unit-paths

/etc/systemd/system.control
/run/systemd/system.control
/run/systemd/transient
/run/systemd/generator.early
/etc/systemd/system
/run/systemd/system
/run/systemd/generator
/usr/local/lib/systemd/system
/usr/lib/systemd/system
/run/systemd/generator.late

Therefore you may put unit files in /etc/systemd/system/ and it will override somethig that is comming by default with package into /usr/lib/systemd/system or is autogenerated from legacy scripts.
However that is only the beginning. If systemd is started in user mode other set of location is processed.
Usually systemd in user mode instance (systemd --user) is launched when the user logs in for the first time. We can also check out these locations with systemd-analyze --user unit-paths. However it's out of scope a little bit. Check man page for more details.

Listing Service Units

We can list all service units present on the system with the status by

systemctl list-unit-files --type service  

for loaded services:

systemctl list-units --type service  

It might be usefull to list failed units as well

systemctl list-units --failed  

Edit Unit Files

There are two approaches

  • replacement
  • drop-in

Replacement can be done e.g with

systemctl edit --full my-unit.service  

This opens /etc/systemd/system/my-unit.service in your editor, copying the installed version (if it does not exist yet) from /usr/lib/systemd/system/my-unit.service. Finally on save it automatically reloads the unit.

Alternatively, to create drop-in files for the unit file /usr/lib/systemd/system/my-unit.service, create the directory /etc/systemd/system/my-unit.service.d/ and place .conf files there to override or add new options. Systemd will parse and apply these files on top of the original unit by mergin directives.

The easiest way to do this is to run:

systemctl edit my-unit.service  

This opens the file /etc/systemd/system/my-unit.service.d/override.conf in your text editor (creating it if necessary) and automatically reloads the unit when you are done editing.

However you can allways revert status to default by:

systemctl revert my-unit.service  

and check the overal units delta by systemd-delta, here an example from my machine:

# systemd-delta
[REDIRECTED] /etc/systemd/system/dbus-org.freedesktop.timedate1.service → /usr/lib/systemd/system/dbus-org.freedesktop.timedate1.service
[EQUIVALENT] /etc/systemd/system/default.target → /usr/lib/systemd/system/default.target
[MASKED]     /etc/systemd/system/systemd-timedated.service → /usr/lib/systemd/system/systemd-timedated.service
[EXTENDED]   /usr/lib/systemd/system/systemd-udev-trigger.service → /usr/lib/systemd/system/systemd-udev-trigger.service.d/systemd-udev-trigger-no-reload.conf
[EXTENDED]   /usr/lib/systemd/user/dbus.service → /usr/lib/systemd/user/dbus.service.d/flatpak.conf

at this point we should understand where to find what, let's take a look at some service units.

Service unit example

Here is an example of system service sshd. With command:

systemctl cat sshd.service  

on my Fedora 28 linux i get

# /usr/lib/systemd/system/sshd.service
[Unit]
Description=OpenSSH server daemon  
Documentation=man:sshd(8) man:sshd_config(5)  
After=network.target sshd-keygen.target  
Wants=sshd-keygen.target

[Service]
Type=notify  
EnvironmentFile=-/etc/crypto-policies/back-ends/opensshserver.config  
EnvironmentFile=-/etc/sysconfig/sshd  
ExecStart=/usr/sbin/sshd -D $OPTIONS $CRYPTO_POLICY  
ExecReload=/bin/kill -HUP $MAINPID  
KillMode=process  
Restart=on-failure  
RestartSec=42s

[Install]
WantedBy=multi-user.target  

Here [Unit] section is more or less self-explanatory. But important directives are After and Wants. These two define dependencies to other units. Means sshd should run after network and sshd-keygen targets. Also sshd-keygen should be started, however sshd can start also if start of sshd-keygen has failed (Check Wants vs. Required) Of course a lot more options existing in this section.

[Service] section controls the behaviour of the service. Here we already see important parts. E.g. ExecStart defines which command starts the service.

ExecStart=/usr/sbin/sshd -D $OPTIONS $CRYPTO_POLICY  

which command reload configuration

ExecReload=/bin/kill -HUP $MAINPID  

and Type=notify means systemd waits untill sshd notifies it about successful start over Notify service manager, while

KillMode=process means that only main process should be killed, remaining process in the same control-group continuig to live. If not set this property defaults to control-group, which means averything in the same control-group is killed.