Appearance
2.8.4 — Users, Permissions & Services
Until now the photo-upload service has been started by hand, from a terminal, as whatever user happened to be logged in. 2.8.2 showed that it dies when that terminal goes away. 2.8.3 showed that any program running as the same user can read its secrets. And if that user is the administrator, a single bug in the service can hand an attacker the whole machine.
A real service needs three things:
- its own user, holding only the permissions it needs;
- a supervisor, which starts it at boot, restarts it if it crashes, and stops it gracefully;
- somewhere to put its logs. On Linux, those are users and groups, the permission tools around them, and systemd. This page covers all three, and ends with a complete, production-shaped setup for the upload service.
1. Users and groups
Every process runs as some user, and the kernel checks that user against a file's permission bits on every open (2.6.1). Here is how a program finds out who it is running as.
ts
// who-am-i.ts — which user is the upload service running as?
import { userInfo } from "node:os";
const me = userInfo(); // (1)
console.log(`user: ${me.username} uid: ${me.uid} gid: ${me.gid} home: ${me.homedir}`);
if (process.getuid?.() === 0) // (2)
console.error("refusing to run as root: start the service as its own user");
// On Windows, which has no Unix user numbers, it prints:
// user: Indranil uid: -1 gid: -1 home: C:\Users\Indranil
//
// On a Linux server, running as the service's own user, it prints something like:
// user: photos uid: 998 gid: 998 home: /var/lib/photos(1)userInfoasks the operating system for the current user's name, numbers and home folder.(2)process.getuidexists only on Unix-like systems, so the?.skips the check on Windows. On Linux, it returns the user number, and 0 is special, as section 2 explains.
The kernel does not really care about names. It works with numbers.
- A user ID, or UID, is the number that identifies a user. Names are only labels, looked up in the file
/etc/passwd, which has one line per user giving the name, the UID, the home folder and the login shell. - A group ID, or GID, identifies a group, a named set of users listed in
/etc/group. Groups let you give several users the same access to a file without giving it to everyone: put them in a group, and give the group permission.
Password hashes are not in /etc/passwd, which every user can read. They are in /etc/shadow, which only the administrator can read. On a server, many "users" are not people at all. They are system users created just to run one service, with no password and no login shell, such as www-data for a web server or postgres for the database.
2. Root, and why not to be root
User number 0, called root, is the administrator, and the kernel treats it differently. For root, the permission checks of 2.6.1 are skipped almost entirely. Root can read every file, kill every process, load kernel modules and change any setting in /proc/sys.
That power is exactly why services should never run as root. If the upload service ran as root and had one bug, say the command injection of 2.8.1 or a path that lets a user write a file anywhere, the attacker would get everything root can do. If it runs as its own system user, photos, the same bug gives the attacker only what photos can do: read and write its own photo folder, and nothing else. This idea is called the principle of least privilege: give each program only the permissions its job needs. It turns a catastrophe into an incident.
sudo: borrowing root for one command
People need root occasionally, to install software or restart a service. Logging in as root for a whole session is risky: every mistake is made with full power, and the record of who did what is lost. So Linux systems use sudo, short for "substitute user, do".
sh
sudo systemctl restart photo-upload # run just this one command as rootsudo checks the rules in /etc/sudoers, which say which users may run which commands as root. It usually asks for the user's own password, runs the one command, and records it in the system log. The rest of the time, you work as an ordinary user. A mistake in an ordinary command cannot damage the system.
3. Doing one privileged thing without being root
Sometimes a program needs one specific power that normally only root has. Linux has two ways to grant exactly that power and no more.
The setuid bit
The passwd command lets any user change their own password. But passwords are stored in /etc/shadow, which only root can write. How does an ordinary user's passwd change it?
The answer is a special permission bit called setuid, "set user ID". When a program file has the setuid bit and is owned by root, it runs as root, whoever starts it. ls -l /usr/bin/passwd shows it as an s where the owner's x would be: -rwsr-xr-x. The passwd program has been very carefully written to change only the caller's own password, and nothing else.
Setuid programs are a classic security risk, because any bug in one runs with root's power. That is why there are very few of them, and why setting the bit on your own programs is almost never the right answer.
Capabilities: splitting root into pieces
Modern Linux divides root's power into about forty separate capabilities, each a single privilege. For example:
| Capability | Allows |
|---|---|
CAP_NET_BIND_SERVICE | listening on ports below 1024, such as 80 and 443 |
CAP_NET_ADMIN | changing network settings |
CAP_SYS_ADMIN | a large mix of administrative actions |
CAP_KILL | sending signals to any process |
A program can be given one capability without becoming root. Suppose the upload service must listen on port 443, the standard port for secure web traffic. Ports below 1024 are traditionally reserved for root. Rather than run the whole service as root, you grant it only CAP_NET_BIND_SERVICE, and it can open port 443 while still running as the unprivileged photos user. Section 5 does this in the service's unit file. In practice, many deployments avoid the question entirely by having a reverse proxy, or a load balancer, listen on 443 and pass requests to the service on a high port such as 8080.
Containers lean heavily on capabilities. A container's "root" user is usually given only a small set of them, so even root inside the container cannot do most of what real root can (14.1.3).
4. systemd: starting, supervising and stopping services
2.1.3 introduced systemd, the program that runs as PID 1 on most Linux systems and starts everything else at boot. It is also the service manager: it starts services, watches them, restarts them when they crash, and stops them in order at shutdown.
Each thing systemd manages is described by a small text file called a unit. A unit that runs a long-lived program is a service unit, and its file name ends in .service.
The everyday commands
sh
sudo systemctl start photo-upload # start it now
sudo systemctl stop photo-upload # stop it (SIGTERM, then SIGKILL after a timeout)
sudo systemctl restart photo-upload # stop, then start
sudo systemctl enable photo-upload # start it automatically at every boot
systemctl status photo-upload # running or not, its PID, and its last log linesstatus is the first command to run when a service misbehaves. It shows whether the service is running, how long it has been up, its main PID, its memory use, and the last few lines it logged.
Logs: journald
Every service started by systemd has its stdout and stderr (2.8.1) captured automatically by a companion program, journald, which stores them in a structured log. The upload service does not need to manage log files at all. It simply prints, and journalctl reads the result.
sh
journalctl -u photo-upload # all logs from this service
journalctl -u photo-upload -f # follow new lines as they arrive
journalctl -u photo-upload --since "10 min ago"
journalctl -u photo-upload -p err # only lines logged as errorsThe -p err option works because journald records the priority of each line. Output written to stderr is usually stored with a higher priority than output to stdout, which is one more reason to send errors to console.error.
5. The upload service as a proper systemd service
Before the file itself, here is what it sets up. systemd starts the service, watches it and stops it, and journald keeps everything it prints.
photos. ② If the service crashes, systemd starts it again. ③ To stop it, systemd sends SIGTERM, waits up to 30 seconds, and only then sends SIGKILL. Everything the service prints goes to journald.Every arrow in the figure comes from one or two lines of the file below. Arrow ① is User= and ExecStart=. Arrow ② is Restart= and RestartSec=. Step ③ is KillSignal= and TimeoutStopSec=, which give the graceful shutdown of 2.2.3 its 30 seconds. The amber arrow to journald needs no setting at all, because systemd collects every service's output automatically.
Here is a complete, production-shaped unit file for the upload service. It brings together almost every idea in Part 2.
ini
# /etc/systemd/system/photo-upload.service
# (systemd only allows comments on their own lines, so the markers sit above each setting)
[Unit]
Description=Photo upload service
# (1)
After=network-online.target
Wants=network-online.target
[Service]
# (2)
User=photos
Group=photos
WorkingDirectory=/opt/photo-upload
# (3)
ExecStart=/usr/bin/node /opt/photo-upload/dist/server.js
Environment=NODE_ENV=production
# (4)
Restart=on-failure
RestartSec=2
# (5)
KillSignal=SIGTERM
TimeoutStopSec=30
# (6)
LimitNOFILE=65536
# (7)
AmbientCapabilities=CAP_NET_BIND_SERVICE
# (8)
NoNewPrivileges=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/photos
PrivateTmp=yes
[Install]
# (9)
WantedBy=multi-user.targetOn the server, systemd-analyze verify /etc/systemd/system/photo-upload.service checks a unit file for mistakes before you load it.
Walk through it.
(1)Start this service only after the network is up, because the upload service listens for connections. This is the "units declare what they depend on" idea from 2.1.3.(2)Run as the dedicated system userphotos, never as root: least privilege. The user is created once withsudo useradd --system --home /var/lib/photos --shell /usr/sbin/nologin photos, which makes a user with no password and no login shell.(3)The exact command to run. systemd starts it directly with fork and exec, with no shell in between (2.8.1), so the Node process is the service's main process and receives signals directly. That avoids the "shell as PID 1" trap of 2.2.3.(4)If the process exits with a failure, restart it two seconds later. systemd also stops trying after too many restarts in a short time, so a service that crashes immediately does not restart forever in a tight loop.(5)To stop the service, sendSIGTERMand wait up to 30 seconds before sendingSIGKILL. That is exactly the window the graceful-shutdown handler of 2.2.3 was built for.(6)Raise the open-files limit from the usual 1,024 to 65,536, for a service holding many connections. This prevents theEMFILEerror of 2.8.3 when the load is genuine, though it is not a fix for a real descriptor leak.(7)Grant only the capability to listen on low ports, so the service can use port 443 without being root.(8)A set of sandboxing options.NoNewPrivilegesstops the service, and anything it starts, from ever gaining more privileges, even through a setuid program.ProtectSystem=strictmakes the whole file system read-only for this service, except the folders listed inReadWritePaths.PrivateTmpgives it its own private/tmp. If the service is compromised, the attacker can write only to the photo folder.(9)WantedBy=multi-user.targetmeans "start this as part of normal boot". This is whatsystemctl enablehooks into.
After writing the file, sudo systemctl daemon-reload tells systemd to read it, and sudo systemctl enable --now photo-upload starts the service and enables it at boot. From then on, the service starts at boot, restarts after crashes, shuts down gracefully on deploys, logs to the journal, and runs with the least privilege it needs. That is the difference between a program and a service.
systemd can also run jobs on a schedule, with timer units, which are a more manageable replacement for the older cron scheduler. A nightly thumbnail clean-up is a .service unit plus a .timer unit saying when to run it, and its output goes to the same journal.
What the interviewer will push on
"Why shouldn't a service run as root?" They are checking that you understand the blast radius. Root bypasses permission checks, so any bug in a root service gives an attacker the whole machine. A dedicated system user limits a compromise to what that user can touch: the principle of least privilege.
"How can a program listen on port 80 without running as root?" Grant it the single capability CAP_NET_BIND_SERVICE, in its systemd unit or with setcap, or put a reverse proxy or load balancer on port 80 and run the service on a high port. Explaining capabilities as root split into pieces shows depth.
"What is the setuid bit?" A permission bit that makes a program run as its file's owner, usually root, whoever starts it, as passwd does to write /etc/shadow. The strong answer adds why setuid programs are dangerous and rare.
"How do you run a Node app as a proper Linux service?" A systemd unit with a dedicated User, ExecStart running node directly, Restart=on-failure, a stop timeout matching the graceful shutdown, a raised LimitNOFILE, and sandboxing options. Logs go to journald, read with journalctl -u.
"Where are a systemd service's logs, and how do you read them?" journald captures its stdout and stderr; journalctl -u name -f follows them, with --since and -p err to filter.
Something to volunteer that nobody asks for: ProtectSystem=strict with a short ReadWritePaths list is one of the cheapest security wins on Linux. It costs one line, and it turns "the attacker can write anywhere the service user can" into "the attacker can write only to the photo folder".
Recall
- Every process runs as a user. The kernel uses numbers: user ID and group ID, named in
/etc/passwdand/etc/group; password hashes are in root-only/etc/shadow. Services run as system users with no login. - root (UID 0) bypasses permission checks, so services must not run as root: the principle of least privilege. sudo runs a single command as root, checked against
/etc/sudoersand logged. - The setuid bit makes a program run as its owner, as
passwddoes; it is rare and risky. Capabilities split root into pieces, such asCAP_NET_BIND_SERVICEfor ports below 1024. - systemd manages services through units:
systemctl start,stop,restart,enableandstatus. journald captures stdout and stderr; read it withjournalctl -u name -f. - A production unit sets a dedicated
User, runs node directly,Restart=on-failure,TimeoutStopSecfor graceful shutdown, a raisedLimitNOFILE, one capability, and sandboxing such asNoNewPrivilegesandProtectSystem=strict.
Self-test: What changes about a bug's damage when the service runs as photos instead of root? How does passwd write a file only root can write? What single capability lets a service use port 443? In the unit file, which line makes deploys wait for graceful shutdown, and which line prevents EMFILE under real load? Where do the service's console.error lines end up?
Quiz Bank
FoundationalWhat is root, and why should services not run as root?
root is the user with user ID 0, the administrator. The kernel skips almost all permission checks for root, so root can read and change any file, signal any process and change kernel settings. A service running as root therefore turns any bug, such as a command injection or a path traversal, into full control of the machine. Running each service as its own unprivileged system user follows the principle of least privilege: a compromise is limited to what that user can reach.
FoundationalWhat does sudo do?
sudo runs a single command as root, or as another user, if the rules in /etc/sudoers allow the calling user to do so. It normally asks for the caller's own password, and records each use in the system log. It lets administrators work as ordinary users most of the time, borrowing root only for the specific commands that need it, which limits the damage of mistakes and keeps a record of who did what.
AppliedHow can a non-root program listen on port 443?
Ports below 1024 traditionally need root. Linux divides root's powers into capabilities, and the one needed here is CAP_NET_BIND_SERVICE. Grant it to the program alone, either in its systemd unit with AmbientCapabilities=CAP_NET_BIND_SERVICE or on the program file with setcap, and it can bind to port 443 while running as an unprivileged user. Many deployments avoid the issue entirely by placing a reverse proxy or load balancer on port 443 and running the application on a high port.
InterviewWhat is the setuid bit, and why is it a security concern?
The setuid bit on an executable file makes it run with the user ID of the file's owner rather than the user who started it. If the owner is root, the program runs as root for everyone. passwd uses it so ordinary users can update their own entry in the root-only /etc/shadow. It is a concern because any bug in a setuid-root program is a bug with root's power, available to every user on the machine. That is why systems keep very few of them, why they are written with extreme care, and why options such as NoNewPrivileges exist to stop services from gaining privileges through them.
InterviewHow would you run a Node application as a reliable service on Linux?
Create a dedicated system user with no login shell, and write a systemd unit that runs the application as that user. Use ExecStart to run node directly, without a shell, so signals reach it. Set Restart=on-failure so crashes are recovered. Set TimeoutStopSec to match the application's graceful-shutdown time, since systemd sends SIGTERM and then SIGKILL. Raise LimitNOFILE if it holds many connections, and add sandboxing such as NoNewPrivileges=yes and ProtectSystem=strict with only the needed ReadWritePaths. Enable it with systemctl enable --now. Its stdout and stderr go to journald, read with journalctl -u.
StaffA security review finds that a compromised dependency in a web service could read SSH keys and other services' config files on the same host. Using this page, propose layered mitigations without changing the application code.
The service can read those files because it runs as a user with broad access, possibly root or a shared account. Mitigations, in layers:
- Identity. Run it as its own dedicated system user, so files owned by other users and services are not readable under ordinary permissions.
- File system view. In its systemd unit, set
ProtectSystem=strictandProtectHome=yesso system folders are read-only and home folders, where SSH keys live, are invisible, with only the photo folder inReadWritePaths; addPrivateTmp=yes. - Privilege.
NoNewPrivileges=yes, and grant only the capabilities it needs, ideally none. - Secrets. Move secrets out of environment variables and world-readable files into files readable only by the service user, or fetch them from a secrets manager at startup.
- Isolation. Run each service in its own container or virtual machine, so a compromise of one host process cannot see the others' files at all.
Each layer limits what the compromised dependency can reach even if the others fail.
Scenario Drill
DrillThe photo-upload service works when started by hand but fails as a systemd service with 'EACCES: permission denied' when saving photos. Using this page, find the likely causes and fix them.
EACCES means the kernel refused a file operation because of permissions. Under systemd the service runs with a different identity and a tighter sandbox than in your terminal, so check both.
- The user. The unit runs it as
photos, but the photo folder is probably owned by your own user, since you created it by hand. Fix the ownership withsudo chown -R photos:photos /var/lib/photos, and make sure the folder's permission bits allow the owner to write, for example750. - The sandbox. With
ProtectSystem=strict, everything outside theReadWritePathslist is read-only for the service. If the code writes somewhere else, such as a temporary folder inside/opt/photo-upload, either add that path toReadWritePathsor change the code to write under/var/lib/photos. - The working directory. A relative path such as
./uploadsis resolved againstWorkingDirectory, which may differ from where you ran it by hand.
journalctl -u photo-upload shows the exact path in the error message, which tells you which of the three it is. Do not fix it by running the service as root: that hides the problem and removes the protection.
Next: the service is now running properly. The last skill is knowing what it is doing when something goes wrong. Page 2.8.5 is a field guide to the tools that look inside a running Linux system, what each number really means, and a method for finding a bottleneck instead of guessing.