series · 9 articles · 23k words
Mastering sed
Nine articles on sed for real work: the one-liners worth memorising, the GNU vs BSD traps, multiline patterns and the hold space, and how to use it safely in a pipeline.
sed is a stream editor: it reads input one line at a time, applies your
commands to that line, prints the result, and forgets the line. That is the
entire model, and it is worth stating plainly, because nearly every surprise in
this series follows from it.
One line at a time is why matching a pattern across two lines needs a second buffer and four commands most people have never used. It is why sed is a poor fit for JSON, which does not care where the newlines are. And it is why sed is astonishingly fast on a two-gigabyte log file — it never holds more than one line in memory.
The four things that bite
Across nine articles the same failures keep showing up. If you take nothing else from this hub, take these.
There is no “sed”. There are at least three. GNU sed on Linux, BSD sed on
macOS, and busybox sed in your Alpine container. They disagree about the most
commonly used flag there is: sed -i edits in place on GNU, while on BSD -i
takes a mandatory backup suffix, so sed -i 's/a/b/' file on a Mac silently
creates a file called -e or errors, depending on the day. The portable form
is sed -i.bak on both, then delete the backup. This is the single most common
production sed bug and it has its own article.
sed does not fail when it matches nothing. A substitution that matches zero lines exits 0 and prints the file unchanged. At a prompt you notice. In a pipeline you get a green build and an unmodified config, and you find out in production. Checking that sed did something is your job — diff the output, or grep for the expected result afterwards and fail the step yourself.
Run it twice. Any sed in automation will eventually run twice: a retried job, a re-run pipeline, a script someone invokes by hand to check. An edit that appends a line is not safe to repeat; an edit that replaces a matched line is. Writing the idempotent version takes a few more characters and removes a whole category of incident.
Quoting and delimiters. s/.../.../ is a convention, not a rule — the
character after s is the delimiter, so use s|...|...| the moment your pattern
contains a path. And a shell variable interpolated into a sed expression is an
injection: if the value contains the delimiter, or an &, sed will do something
you did not ask for.
When to stop using sed
This series is unusually keen to tell you to use something else, which is the mark of knowing a tool rather than liking it:
- Finding lines →
grep. If you are writingsed -n '/x/p', that is grep. - Columns and arithmetic →
awk. If you are counting fields, you have left sed’s territory. - JSON →
jq. The JSON article exists for exactly one situation — a locked-down box where you cannot install anything and something is broken right now — and it opens by telling you to use jq. - Anything nested or recursive → a real language. Config files with blocks, YAML with meaningful indentation, HTML: sed can be forced to do these and the result is unmaintainable.
The rule of thumb the decision matrix settles on: grep to find, sed to change, awk to compute. Most arguments about which one to reach for dissolve once you say what you are doing out loud.
How to read this
If you already use sed daily, go to GNU vs BSD and sed in CI/CD — that is where the material is that experience does not teach you, because both failure modes are silent.
If you are starting, read sed vs awk vs grep for the shape of the problem, then keep the cheat sheet open and learn the patterns as you hit them. The 30 one-liners in it are the ones that came out of real logs, not the ones that demonstrate features.
And if you have ever needed to match across two lines and given up: the multiline article explains the hold space, which is the piece of sed that makes the rest of the manual make sense.
What you need
A shell. That is it — sed is on every Unix machine you will ever touch, which is the whole reason to learn it. The articles note where GNU and BSD differ, so the examples work on macOS as well as Linux.
All 9 articles
Start here
Two questions come before everything else: is sed the right tool for this at all, and which handful of patterns cover most of what you will ever do?
- Sed vs Awk vs Grep: When to Use Which (with Decision Matrix) Sed vs awk vs grep: a decision matrix for picking the right tool, with common DevOps tasks side by side and the rule of thumb that settles most of them.
- Sed Cheat Sheet: 30 One-Liners from Real Production Logs A working sed cheat sheet: 30 one-liners for logs and config files, covering substitution, deletion, line selection, and multiline patterns.
The three jobs sed is actually for
Changing the same string across a tree of files, editing config, and cutting a log down to the part you care about. Most sed you will ever write is one of these.
- How to Replace Text in Multiple Files with Sed Master replacing text across multiple files with sed. This step-by-step guide covers basic substitutions to advanced pattern matching with practical examples.
- Mastering sed for YAML, JSON, TOML Config Files Become an expert at using sed to precisely manipulate config files. Covers advanced techniques for updating YAML, JSON, TOML, and properties with real examples.
- Sed for Log Analysis: Errors, Time Filters, Patterns Practical sed patterns for log analysis: extract errors, filter time ranges, anonymize PII, parse multi-line stack traces, and merge sorted logs.
Where it gets hard
The two things that turn a working one-liner into a bug report: patterns that need to span lines, and the fact that there is no single sed.
- Sed Multiline Patterns: How to Match Across Lines Sed multiline patterns explained: the hold space, the N/D/P commands, and address ranges. Production examples for stack traces, YAML blocks, and SQL statements.
- Sed Gotchas: GNU vs BSD and Safe In-Place Editing The sed gotchas that bite in production: GNU vs BSD differences, in-place editing safety, escape sequence traps, and portable patterns that work everywhere.
sed in a pipeline
Running sed unattended is a different discipline from running it at a prompt. Nobody is watching, and sed does not complain when it does nothing.
- Sed in CI/CD: Safe Patterns for GitHub Actions and Jenkins Use sed safely in CI/CD pipelines: idempotent edits, exit-code checks, dry-run patterns, and the gotchas that break Jenkins, GitHub Actions, and CodeBuild jobs.
- Sed for JSON: Emergency Patterns When jq Is Unavailable When jq is unavailable, sed can manipulate JSON in emergencies, such as on a locked-down server with no install permission. Use jq whenever you can.