A few weeks ago I needed to create a daemon for a school project. I had never really dealt with daemons before so I browsed on the Internet for what they were and how they worked. After reading some pages, I found out how hard it seemed to be working with daemons: you need to deal with the correct forking of a process, prevent core dump generation, change root and working directories, change process and file umasks and ownership, and quite a lot of other OS-related stuff you can find listed here. By the way, at this time I’m still not quite sure of how these conditions came to be.
Wait. What is a daemon?
A little side note. Although the word “daemon” might sound like something diabolical, it has nothing to do with Satan, in fact it comes from the Greek word δαίμων, which refers to the spirits people have inside and which eventually define them. A daemon is a process. No more and no less then your browser’s process is. The key difference is, though, that a daemon is a process that doesn’t need user input to work. Think, for instance, about a web server which isn’t waiting for its own user to perform some action, but rather it’s waiting for some other host on the network to perform a request. Such request needs to be processed without any human taking part in it.
So a daemon is a process that performs what you could think of as a background task. May it be a server like a web server or an ssh server, or something more complicated like systemd. OS-wise, daemons are specific to the world of Unix. If you’re running Windows or Mac OS X, you should keep in mind that Unix daemons do have their respective counter-part in other OSes: Windows has its so-called services; OSX sometimes calls them “daemons”, sometimes “agents” and they still pretty much work just as Unix daemons, but it sort of expects you to make them compatible with launchd.
How can I create a daemon?
There are a lot of ways you can create a daemon in Unix, since nobody enforces or supports one in favor of the others. Within the shell of your choice, type in the command you were trying to daemonize and add ‘&’ at the end of it.
And that’s it. You have a daemon.
Now, while this is a very fast way to spawn a daemon, it might not be the most reasonable choice for a number of reasons: the process will output anything to your current shell (using ‘&’ doesn’t close the stdout and stderr file descriptors), you can’t assign the daemon any PID lock file so multiple daemons might be running at the same time (I’ll come back to the lock file later) and often that makes it generally harder to control the daemon.
Nonetheless this remains the fastest way to create a daemon in Unix. Just a character away. And sometimes speed and simplicity are exactly what you need. There are a couple recipes online that will do the job for you. I won’t go through the details of all of them, I’ll just link a couple of them here:
This does have some consequences: you cannot pip install them and you cannot have a simple way to know if there is going to be an update for any of them (crucial in a security-aware context), but still these recipes will get you the job done. Use “daemon“. Here’s a nice article about it. This is a pretty nice utility, which will do most of the job for you, similarly to python-daemon, but if you’re relying on python code for running your daemon, it may not be the best choice. Anyway, it’s simple enough that you might want to look at it if you need to spawn deamons in C or any other language.
While quite a lot of people (like me) find it tedious to re-invent the wheel, others sometimes feel like they need to re-invent the whole TCP/IP stack from the ground up. I won’t judge you for this. You’re going to have a lot of fun with that. Just a couple of modules I found a lot of people were using while writing their homemade daemons: os.fork, os.setsid, signal, subprocess.
Ok, here we are. This is the way I create daemons and the way I would recommend to most people by using python-daemon.
It doesn’t require a lot of knowledge about the underlying machinery that gets the daemon to run, its source code is rather readable, its APIs are very very simple (we’ll go through these later), you can pip install it, it is still maintainedand it was going to be the standard way to create daemons in Python. Its main missing feature is its lack of documentation: the documentation you’ll find online is sparse and you’ll often need to look at its source code if you encounter any bugs.
So what is python-daemon?
Back in the first weeks of 2009 PEP 3143 was created. Its aim was to create “a package [in] the Python standard library that provides a simple interface to the task of becoming a daemon process.” While the goal was not an impossible task and quite some people were interested in seeing this project succeed, it didn’t make it. The guy that was in charge of doing it simply didn’t have enough time anymore and no one stepped in to save the project. Such a sad death for such a nice project.
This tragedy didn’t affect the functionality of python-daemon too much (it does include basically anything you need for a daemon), but rather its documentation. As I said earlier, its weakest point is documentation: you’ll find some inside the PEP and some within the code itself.
And how do you make it work?
Ok, I guess I got you interested in python-daemon since you’re still reading. First, let me start off by telling you what you shouldn’t use in this library: DaemonRunner. Googling python-daemon will find some pages that will point you to the DaemonRunner object to handle your daemon, but it is a deprecated part of the library.
Instead, you want to use the DaemonContext API which is used inside of DaemonRunner. It’s true that DaemonRunner extends DaemonContext functionality, but it does so in a very old-fashioned way (doesn’t use argparse for instance). This is probably the reason why it ended up being deprecated.
Without further ado, DaemonContext makes it super simple to start your daemon with just a context manager:
Dealing with the file system
A daemon is a fairly peculiar process: since it is unbound from human interaction, a daemon will have its own keys to be identified user-wise. This means that, regardless of the user that started a daemon, the daemon will have its own UID, GID (User/Group ID), its own root and working directories, and its own umask.
Don’t be afraid, DaemonContext will take care of this stuff for you, even with just the default configuration, but let’s say that you need to customize this stuff. To change the root directory, useful for confining your daemon, simply set the chroot_directory argument to a valid directory on your file system. The same goes for the working directory, which is a more usual thing to do, under the argument working_directory. By default, DaemonContext will set your working directory to root “/”. An example as below:
One thing to take into account when creating a daemon is that on start DaemonContext will close any open files you have around. This is normal and it’s what it is supposed to do. Now, even though this is the behavior we should expect from the library, you might still need some files to be opened in your program. You can do this by declaring what files you won’t need to be closed through the files_preserve argument. For instance:
Signals coming from the OS are important, regardless of whether you’re switching your program to be daemonized. Furthermore, this makes it even more important for you to take care of such signals, since it might become the only way a human interacts with your process. DaemonContext will conveniently let you define a dictionary in the signal_map argument that will be linked to the signals you might want to configure. Some popular ones are: SIGINT, SIGKILL, SIGTERM, SIGTSTP. You can find further details here.
More than often daemons will use resources, such as a TCP port for a listening server or some files on disk. You’ll probably want to make sure that there aren’t multiple daemons conflicting for these resources. To make sure that only one of your daemons is running at the same time, you can use a PID lock file, which is a file containing the PID of a process that will prevent the same program from running on more than one instance. Please note that it is the duty of the newly spawned process (handled within DaemonContext) to check the lock file and abort the start procedure. If you’re already familiar with threading.Lock the concept is basically the same.
You can set a lock file like this:
A common pattern for a daemon to interact with its administrator is to provide a start/stop/reload behavior which is usually implemented as a set of command line arguments. This is particularly useful if you’re planning to support initd. DaemonContext, though, will not take care of this for you. DaemonRunner does have code in regard to this behavior, but I wouldn’t advise you to use it directly since it is deprecated. Anyway you can still use its source code as a reference, for further details take a look at the _start and _stop methods.
The package python-daemon is absolutely not the only way you can create a daemon for a Python program, you should carefully consider every possibility you have. If your choice is python-daemon, we have gone through pretty much all of the configurations of DaemonContext. I haven’t covered all of them though, if you’re still looking for more options you should look in the PEP; if you can’t find enough information there, have a look at DaemonContext’s source code.
* PEP 3143 -- Standard daemon process library