Appearance
2.8.1 — The Shell, Streams & Pipes
The photo-upload service has started failing some uploads. Its log file has thousands of lines like these:
text
10.0.0.5 POST /upload 200 812ms
10.0.0.9 POST /upload 500 30ms
10.0.0.5 GET /thumbs/7 200 4ms
10.0.0.7 POST /upload 500 28msEach line holds the client's address, the request, the result code and the time taken. A result of 500 means the server failed. Which clients are hitting the most failures? On a Linux machine, one line answers it:
sh
grep " 500 " access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn | head -3
# ① keep failures ② keep column 1 ③ sort ④ count repeats ⑤ biggest first ⑥ top 3
# Prints:
# 3 10.0.0.9
# 2 10.0.0.7Six small programs, each doing one tiny job, joined by the | character into a custom log analyser you wrote in ten seconds. None of the programs knows the others exist. This page explains exactly how that works: what the shell is, what the three standard streams are, how redirection and pipes connect them, and how exit codes let commands make decisions. Every one of those ideas sits directly on the processes, file descriptors and pipes of the earlier chapters.
1. The shell is a fork–exec machine
When you open a terminal, you are talking to a shell: bash, zsh or fish. A shell is not part of the kernel. It is an ordinary user-mode program, part of the userland from 2.1.1. Its whole job is a loop called a REPL, for read, evaluate, print, loop:
- Print a prompt and read a line you type.
- Work out what it means, and run it.
- Show any result, and loop back to step 1.
And "run it" is exactly the fork–exec pattern from 2.2.2. When you type ls:
- The shell calls
fork(), making a child copy of itself. - The child calls
exec()and becomes thelsprogram. - The shell calls
wait()until the child finishes, then prints the next prompt.
Every command you type is a new process the shell starts. That is not a metaphor. It is literally the mechanism, and it explains everything else on this page.
A few commands are different. cd changes the current directory. If the shell ran cd as a child process, the child's directory would change, and the child would exit, leaving the shell exactly where it was. So cd is a built-in, run inside the shell itself. export, exit and alias are built-ins for the same reason: they change the shell's own state.
How the shell finds a program: PATH
When you type node, how does the shell know which file to run? It looks in the directories listed in an environment variable called PATH.
An environment variable is a named piece of text that a process carries, and that every child it starts inherits a copy of (2.9.3 covers them in full, including Windows's two kinds). PATH holds a list of directories separated by colons, such as /usr/local/bin:/usr/bin:/bin. The shell tries each directory in order, and runs the first file called node that it finds. The command which node tells you which one that is.
Two consequences matter in practice. A new program you installed is not found until its directory is in PATH. And because children inherit a copy of the environment when they start, changing PATH in one terminal does not affect terminals that were already open.
How the shell treats what you type
The shell changes your command line before the program ever sees it. Knowing this prevents many puzzling bugs.
- Word splitting. The shell splits the line into separate arguments at spaces.
rm my photo.jpgpasses two arguments,myandphoto.jpg. Quotes stop the split:rm "my photo.jpg"passes one. - Globbing. The shell replaces patterns such as
*.jpgwith the matching file names. The program receives the list of names, not the*. In a folder holdinga.jpgandb.jpg,echo *.jpgprintsa.jpg b.jpg, whileecho '*.jpg', in single quotes, prints*.jpgunchanged. - Variables.
$HOMEis replaced with the value of theHOMEvariable. Single quotes prevent this; double quotes allow it.
Starting a program without a shell, on purpose
That rewriting is useful when you type. It is dangerous when a program builds a command line from data it does not control. Here is the upload service passing a file name chosen by a user to a thumbnail tool, two ways.
ts
// shell-or-not.ts — the same upload name passed through a shell, and passed directly
import { execSync, execFileSync } from "node:child_process";
const uploadedName = "beach.jpg & echo INJECTED"; // (1) a file name chosen by a user
const viaShell = execSync(`echo resizing ${uploadedName}`, { encoding: "utf8" }); // (2)
console.log("through a shell:\n" + viaShell);
const direct = execFileSync(process.execPath, // (3) no shell in between
["-e", "console.log('resizing ' + process.argv[1])", uploadedName], { encoding: "utf8" });
console.log("no shell:\n" + direct);
// Printed on Windows, where the shell is cmd.exe; /bin/sh on Linux behaves the same way:
// through a shell:
// resizing beach.jpg
// INJECTED
//
// no shell:
// resizing beach.jpg & echo INJECTED(1)A user uploads a photo with a deliberately strange name. In both common shells,&separates two commands.(2)execSyncruns the string through a shell. The shell sees&and runs a second command the attacker wrote. Here it only printed a word. It could just as easily have deleted files or downloaded malware. This is command injection, one of the classic security holes (8.5.1).(3)execFileSyncstarts the program directly with fork and exec, and passes each argument as a separate, untouched string. No shell ever reads the name, so&is just a character in a file name.
The rule: when a command includes data from outside, never pass it through a shell. Use execFile or spawn with an argument list.
2. Three standard streams
Every process starts with three file descriptors already open (2.1.2 explained why the first file you open is usually number 3). They are the standard streams:
| Descriptor | Name | Used for | Default |
|---|---|---|---|
| 0 | stdin, standard input | reading input | the keyboard |
| 1 | stdout, standard output | normal results | the terminal |
| 2 | stderr, standard error | error messages and warnings | the terminal |
A program does not know or care what is really behind them. It simply reads descriptor 0 and writes descriptors 1 and 2. The shell decides what they are connected to, in the gap between fork and exec (2.2.2). That is the whole secret of redirection and pipes.
Why have a separate stream for errors? So that error messages do not get mixed into the real output. If you save a program's results to a file, you still want to see its errors on the screen, and you do not want error text corrupting the saved data.
In Node, console.log writes to stdout and console.error writes to stderr. That is why a server should log errors with console.error: deployment tools often collect the two streams separately.
3. Redirection: pointing the streams at files
The shell can point any stream at a file instead of the terminal.
sh
node report.js > report.txt # stdout into a file, replacing it
node report.js >> report.txt # stdout appended to the end of the file
node import.js < photos.csv # stdin read from a file
node report.js 2> errors.txt # stderr (descriptor 2) into a file
node report.js > all.txt 2>&1 # stdout into all.txt, then stderr to the same place
node noisy.js > /dev/null # throw stdout awayThe number before > says which descriptor to redirect, and 1 is the default. /dev/null is a special device file that accepts anything written to it and keeps nothing (2.8.3).
Why 2>&1 must come after >
2>&1 means "make descriptor 2 point wherever descriptor 1 points right now". The shell processes redirections from left to right, so the order changes the meaning. Here is ls asked to list one file that exists and one that does not.
sh
ls access.log missing.txt > out1.txt 2>&1
cat out1.txt
# ls: cannot access 'missing.txt': No such file or directory
# access.log
ls access.log missing.txt 2>&1 > out2.txt
# ls: cannot access 'missing.txt': No such file or directory ← appeared on the screen
cat out2.txt
# access.logIn the first command, the shell first points stdout at out1.txt, then points stderr at "wherever stdout points", which is now out1.txt. Both end up in the file.
In the second, the shell first points stderr at "wherever stdout points", which is still the terminal, and only then points stdout at the file. So the error went to the screen. Once you see that 2>&1 copies a pointer at that moment, the order makes complete sense.
4. Pipes: one program's output is the next one's input
The | character connects the stdout of one program to the stdin of the next, through a pipe (2.2.4). For grep " 500 " access.log | cut -d' ' -f1, the shell:
- creates a pipe, getting a read end and a write end;
- forks a child for
grep, and beforeexec, points that child's stdout at the pipe's write end; - forks a child for
cut, and beforeexec, points that child's stdin at the pipe's read end; - closes its own copies of the pipe's ends, and waits for both children.
Both programs run at the same time. As grep writes matching lines, cut reads them. No temporary file is created. And the pipe brings the behaviour you met in 2.2.4. If cut falls behind, the pipe fills and grep is put to sleep, which is backpressure. If a later program exits early, as head does after its few lines, the programs before it get SIGPIPE and stop, instead of reading the rest of a huge file for nothing.
Now read the opening pipeline again, stage by stage, on the sample log:
grep " 500 " access.logkeeps only lines containing500: the failures.cut -d' ' -f1splits each line at spaces and keeps field 1, the client address.sortsorts the addresses, so identical ones sit next to each other.uniq -ccollapses each run of identical lines into one, with a count in front. It only merges adjacent duplicates, which is whysorthad to come first.sort -rnsorts by that count, as a number (-n), biggest first (-r, reversed).head -3keeps the first three lines.
The Unix philosophy
That pipeline is the Unix philosophy in action, summed up by Doug McIlroy, who invented pipes at Bell Labs in 1973: write programs that do one thing well, write programs to work together, and write programs to handle text streams, because that is a universal interface. Small tools, each reading stdin and writing stdout, can be combined into solutions nobody planned for. The same idea reappears as single-responsibility design in code (9.3.5) and as small services joined by networks (Part 10).
5. Exit codes: how commands report success
Every process ends with an exit code, a number from 0 to 255 that its parent collects with wait (2.2.2). By convention, 0 means success and anything else means failure. Different non-zero numbers can say what went wrong, and 128 or above means a signal killed the process (2.2.3).
The shell keeps the last command's exit code in the variable $?, and uses exit codes to make decisions:
sh
grep -q " 503 " access.log; echo "exit code: $?"
# exit code: 1 ← grep found nothing, which grep reports as 1
grep -q " 500 " access.log && echo "found 500s"
# found 500s ← && runs the second command only if the first succeeded
false || echo "false failed, so this ran"
# false failed, so this ran ← || runs the second command only if the first failed&& means "and then, if that worked". || means "or else, if that failed". Scripts are built from these. A deploy script might say npm test && npm run build && ./deploy.sh, so it stops at the first failure.
In a script, the line set -e near the top makes the script stop as soon as any command fails, instead of carrying on after an error. Most careful shell scripts start with it, often written set -euo pipefail. The extra parts make it an error to use a variable that was never set, and make a pipeline fail if any stage fails, not just the last.
In Node, process.exit(1), or an uncaught error, ends the process with a non-zero code. That is how a failing test runner or build script tells the tool that started it that something went wrong.
6. Scripts and the shebang line
A shell script is a text file of shell commands. To run it like a program, two things are needed.
- Its execute permission bit must be set, with
chmod +x deploy.sh(2.6.1). - Its first line must say which program should run it. That line is called the shebang, from the two characters it starts with,
#!("hash" and "bang").
sh
#!/usr/bin/env bash
set -euo pipefail
echo "deploying the photo-upload service"When you run ./deploy.sh, the kernel itself reads the first two bytes. Seeing #!, it runs the program named on that line and passes it the script's path. /usr/bin/env bash means "find bash using PATH", which works whether bash lives in /bin or /usr/local/bin. The same trick runs a TypeScript or JavaScript file directly: a first line of #!/usr/bin/env node makes a file behave like any other command.
What the interviewer will push on
"How does the shell run a command?" They are checking that you connect it to processes. The shell forks a child, the child sets up its standard streams and calls exec to become the program, and the shell waits for it. The tell is explaining why cd must be a built-in: a child cannot change its parent's directory.
"What are stdin, stdout and stderr, and what does a pipe do?" Descriptors 0, 1 and 2: input, normal output and errors, kept separate so errors do not corrupt output. A pipe connects one process's stdout to the next one's stdin through a kernel buffer, and both processes run at the same time.
"What is the difference between > file 2>&1 and 2>&1 > file?" Redirections are processed left to right, and 2>&1 copies where stdout points at that moment. Only the first sends both streams to the file.
"What do exit codes mean, and how do && and || use them?" 0 is success, non-zero is failure, 128 + n means killed by signal n. && runs the next command only on success, || only on failure. Mention set -euo pipefail for scripts.
"Why is it dangerous to build a shell command from user input?" Command injection: the shell interprets characters like ;, & and | in the data. Pass arguments as a list to execFile or spawn, never through a shell string.
Something to volunteer that nobody asks for: the shell expands *.jpg before the program runs. That is why rm *.jpg in a huge folder can fail with "argument list too long": the shell built an argument list larger than the kernel accepts. find . -name '*.jpg' -delete avoids it by never building the list.
Recall
- The shell is an ordinary program running a REPL; every command is a fork, an exec and a wait. Built-ins such as
cdrun inside the shell because a child cannot change its parent. - The shell finds programs through
PATH, an environment variable that children inherit a copy of. It rewrites your line first: word splitting, globbing and variables. Pass outside data withexecFileorspawn, never through a shell, to avoid command injection. - Every process has stdin (0), stdout (1) and stderr (2). Redirection points them at files;
2>&1copies where stdout points at that moment, so order matters. - A pipe joins one program's stdout to the next one's stdin; the programs run together, with backpressure and
SIGPIPE. The Unix philosophy: small tools joined by text streams. - An exit code of 0 means success;
&&,||andset -euo pipefailbuild scripts on them. The shebang line#!tells the kernel which program runs a script.
Self-test: Why would cd not work if the shell ran it as a child process? What does the shell do between fork and exec to make grep x | sort work? Why did the error appear on screen with 2>&1 > out2.txt? Why must sort come before uniq -c? What would a user's file name containing ; rm -rf ~ do in a shell command string?
Quiz Bank
FoundationalWhat is a shell, and how does it run the commands you type?
A shell, such as bash or zsh, is an ordinary user-mode program running a read–evaluate–print loop: it reads a command, runs it, shows the result and repeats. It runs a command with fork–exec: it forks a child process, the child points its standard streams where the command line says and then calls exec to become the program, and the shell waits for the child to exit before printing the next prompt. So every command is a child process. Commands that must change the shell itself, such as cd, are built into the shell.
FoundationalWhat are stdin, stdout and stderr, and what does a pipe do?
Every process starts with three open file descriptors: stdin (0) for input, stdout (1) for normal output, and stderr (2) for errors, kept separate so errors do not mix into the data. By default they are the keyboard and the terminal, but the shell can redirect them to files with <, > and 2>. A pipe, written |, connects one process's stdout to the next process's stdin through a small kernel buffer, so a | b | c streams data through a chain of programs running at the same time, with no temporary files.
AppliedExplain the difference between `cmd > out.txt 2>&1` and `cmd 2>&1 > out.txt`.
The shell applies redirections from left to right, and 2>&1 means "make stderr point wherever stdout points right now". In the first form, stdout is pointed at out.txt first, then stderr is pointed at the same place, so both streams go to the file. In the second form, stderr is pointed at stdout's current target, the terminal, before stdout is moved to the file, so errors still appear on the screen and only normal output goes to the file.
InterviewWhat are exit codes, and how do shell scripts use them?
Every process ends with an exit code, 0 to 255, collected by its parent. Zero means success; any other value means failure, with specific values sometimes meaning specific errors, and 128 plus n meaning the process was killed by signal n. The shell stores the last code in $?. a && b runs b only if a succeeded; a || b runs b only if a failed. Scripts commonly start with set -euo pipefail, which stops the script at the first failing command, treats unset variables as errors, and makes a pipeline fail if any of its stages fails.
InterviewWhy is it dangerous to run a command built from user input through a shell, and what should you do instead?
A shell interprets special characters in the command string: ;, & and | separate commands, $(...) runs a command, and * expands to file names. If user-controlled text such as a file name is pasted into the string, the user can add their own commands. That is command injection, and it runs with the program's full permissions.
Instead, start the program directly, without a shell, and pass each argument as a separate item in a list: execFile or spawn in Node, subprocess.run([...]) in Python. The kernel's exec receives the arguments untouched, so special characters are just characters.
StaffA nightly cleanup script runs `rm /data/thumbs/*.jpg` and has started failing with 'argument list too long'. Explain the cause and a robust fix.
The shell expands *.jpg into the full list of matching names before running rm, and passes them all as arguments to exec. The kernel limits the total size of the arguments and environment passed to a new program. With hundreds of thousands of thumbnails, the expanded list exceeds that limit, so exec fails and rm never runs. The script also ignores the failure unless it uses set -e.
A robust fix avoids building one giant argument list: find /data/thumbs -name '*.jpg' -delete walks the directory itself, or find ... -print0 | xargs -0 rm passes names in safe batches. Add set -euo pipefail so failures stop the script and are noticed. Longer term, a directory holding that many files is itself a design smell; spreading files across subdirectories avoids both this and slow listings.
Scenario Drill
DrillThe photo-upload service's deploy script runs `npm ci; npm run build; ./restart.sh`. A build failure last week still restarted the service with half-built files. Using this page, explain and fix it.
The ; separator runs the next command regardless of whether the previous one succeeded. npm run build failed and exited with a non-zero exit code, but ; ignored it, so ./restart.sh ran with an incomplete build.
Fix it by making the script stop on failure:
- Join the steps with
&&, as innpm ci && npm run build && ./restart.sh, so each runs only if the previous succeeded. - Better, start the script with
#!/usr/bin/env bashandset -euo pipefail, so any failing command, unset variable or failing pipeline stage stops the script at once. - Build into a new directory and switch to it only after a successful build, so a failure never leaves the running service half-updated.
Finally, make sure the deploy tool checks the script's own exit code, so a failed deploy is reported instead of silently ignored.
Next: the shell talks to you through something called a "terminal", and on Linux you will see the letters "TTY" everywhere. Page 2.8.2 explains where those names come from, what Ctrl+C and Ctrl+Z really do, why closing a terminal kills the programs started from it, and how to keep a program running after you log out.