# Bash sed & awk Cheatsheet

- URL: https://karandeepsingh.ca/cheatsheets/bash/bash-sed-awk/
- Author: Karandeep Singh
- Published: 2025-04-25
- Tags: bash, sed
- Summary: My working reference for sed and awk. Substitution, addresses, capture groups, fields, BEGIN/END blocks, associative arrays, and the one-liners I actually use.

Half of DevOps is reshaping text: log files, CSVs, config files, command output. `sed` and `awk` are the two tools I reach for constantly, and somehow I still re-derive the syntax from scratch every time. So I wrote it down. If your shell fundamentals are rusty, start with the [Bash Basics Cheatsheet](https://karandeepsingh.ca/cheatsheets/bash/bash-basics/) first.

## sed: Substitution

> **Docs:** [GNU sed manual](https://www.gnu.org/software/sed/manual/sed.html)

```bash
sed 's/foo/bar/' file            # replace first foo per line
sed 's/foo/bar/g' file           # replace ALL per line
sed 's/foo/bar/2' file           # only the 2nd occurrence per line
sed 's/foo/bar/gi' file          # global + case-insensitive
sed 's|/var/log|/mnt/log|g' file # any delimiter, handy for paths
sed -E 's/[0-9]+/N/g' file       # -E: extended regex (no backslash escaping)
sed -n 's/foo/bar/p' file        # print only lines where a change happened
```

## sed: Deletion & Insertion

> **Docs:** [sed(1) man page](https://man7.org/linux/man-pages/man1/sed.1.html)

```bash
sed '/^#/d' config.conf          # delete comment lines
sed '/^$/d' file                 # delete blank lines
sed '3d' file                    # delete line 3
sed '2,5d' file                  # delete lines 2-5
sed '$d' file                    # delete last line

sed '3i\inserted before line 3' file
sed '3a\appended after line 3'  file
sed '/^\[nginx\]/a\worker_processes auto;' app.ini   # append after a match
sed '/pattern/c\replacement line' file               # change whole line
```

## sed: In-Place Editing (GNU vs BSD Gotcha)

This is the one that breaks a script when it moves between a Mac and a build
agent. BSD `sed` needs an argument after `-i`; GNU `sed` reads that argument
as the filename:

```text
$ sed -i '' 's/ERROR/FATAL/' app.log
# exits with status 2
sed: can't read s/ERROR/FATAL/: No such file or directory
# GNU took '' as the backup suffix and 's/ERROR/FATAL/' as the file to edit.
# app.log was never touched.
```

Write it so both agree — `-i.bak` is accepted by each, and the backup is
something you can delete:

```bash
sed -i.bak 's/ERROR/FATAL/' app.log && rm -f app.log.bak
```

> **Docs:** [GNU sed manual](https://www.gnu.org/software/sed/manual/sed.html)

```bash
sed -i 's/foo/bar/g' file        # GNU sed (Linux): edit in place
sed -i.bak 's/foo/bar/g' file    # in place + keep file.bak backup
```

**warning:**
On macOS (BSD sed), `-i` **requires** an argument: `sed -i '' 's/foo/bar/g' file`. GNU's `sed -i 's/...'` fails on BSD with "invalid command code", and BSD's `sed -i ''` breaks GNU. For cross-platform scripts, use `sed -i.bak ... && rm file.bak`, or install `gsed` via Homebrew.

## sed: Addresses & Ranges

> **Docs:** [GNU sed manual](https://www.gnu.org/software/sed/manual/sed.html)

| Address | Applies command to... |
|---------|----------------------|
| `5` | Line 5 |
| `$` | Last line |
| `2,10` | Lines 2 through 10 |
| `/regex/` | Lines matching regex |
| `/start/,/end/` | From a `start` match to the next `end` match |
| `0~3` | Every 3rd line (GNU) |
| `5,$` | Line 5 to end of file |
| `/regex/!` | Lines NOT matching |

```bash
sed -n '10,20p' file                    # print lines 10-20
sed -n '/BEGIN CERT/,/END CERT/p' pem   # extract a certificate block
sed '/^#/!s/foo/bar/' file              # substitute only on non-comments
sed '1,/^$/d' mail.txt                  # delete headers (up to first blank line)
```

## sed: Capture Groups & Backreferences

> **Docs:** [GNU sed manual](https://www.gnu.org/software/sed/manual/sed.html)

```bash
sed -E 's/(user)-([0-9]+)/\2-\1/' file        # swap: user-42 -> 42-user
sed -E 's/^([a-z]+)=(.*)$/export \1="\2"/'    # key=val -> export key="val"
sed -E 's/(error)/[\1]/gi' app.log            # wrap matches; \1 refers back
sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/'  # date reshuffle
sed 's/.*"id": *"\([^"]*\)".*/\1/' resp.json  # basic regex needs \( \)
```

`&` in the replacement means "the whole match": `sed -E 's/[0-9]+/<&>/g'`.

## awk: Fields & Built-in Variables

> **Docs:** [GNU gawk manual](https://www.gnu.org/software/gawk/manual/gawk.html)

awk splits each line into fields: `$1`, `$2`, ... `$NF`. `$0` is the whole line.

| Variable | Meaning |
|----------|---------|
| `NR` | Current line (record) number |
| `NF` | Number of fields on this line |
| `FS` | Input field separator (default: runs of whitespace) |
| `OFS` | Output field separator (default: single space) |
| `RS` / `ORS` | Input / output record separator |
| `FILENAME` | Current input file |
| `FNR` | Line number within the current file |

```bash
awk '{print $1, $3}' access.log         # fields 1 and 3
awk '{print $NF}' file                  # last field
awk '{print $(NF-1)}' file              # second-to-last
awk -F: '{print $1}' /etc/passwd        # -F sets FS: usernames
awk -F',' -v OFS='\t' '{$1=$1; print}' data.csv   # CSV -> TSV
awk 'NR==5' file                        # print line 5
awk 'NR>1' data.csv                     # skip the header row
```

## awk: Patterns & Actions

> **Docs:** [gawk(1) man page](https://man7.org/linux/man-pages/man1/gawk.1.html)

The model is `pattern { action }`. The action runs only on matching lines, and a pattern alone means `{ print }`.

```bash
awk '/ERROR/' app.log                       # grep-like
awk '$3 > 500' access.log                   # numeric comparison on a field
awk '$1 == "nginx" && $4 ~ /5[0-9][0-9]/'   # combine conditions, regex on field
awk 'NF == 0 {next} {print $1}' file        # skip blank lines
awk 'length($0) > 120' script.sh            # lines longer than 120 chars
awk '!seen[$0]++' file                      # dedupe, preserving order
```

## awk: BEGIN / END

> **Docs:** [GNU gawk manual](https://www.gnu.org/software/gawk/manual/gawk.html)

```bash
awk 'BEGIN {FS=","; print "host,status"} $2=="down" {print $1","$2}' hosts.csv
awk '{total += $5} END {print "bytes:", total}' access.log
awk 'END {print NR}' file                   # line count, like wc -l
awk 'BEGIN {for (i=1; i<=5; i++) print "srv" i}'   # no input file needed
```

## awk: Associative Arrays

```text
$ cat app.log
2025-06-01 ERROR web-1 timeout contacting db
2025-06-01 INFO  web-1 request served
2025-06-01 ERROR web-2 timeout contacting db
2025-06-01 WARN  lb-1  upstream slow
2025-06-01 ERROR web-1 connection refused
$ awk '{ count[$2]++ } END { for (lvl in count) print count[lvl], lvl }' app.log | sort -rn
3 ERROR
1 WARN
1 INFO
$ awk '$2=="ERROR" { host[$3]++ } END { for (h in host) print host[h], h }' app.log | sort -rn
2 web-1
1 web-2
# One pass, no sort of the whole file, and the key can be any field.
```

> **Docs:** [GNU gawk manual](https://www.gnu.org/software/gawk/manual/gawk.html)

```bash
# count requests per IP, sorted by count
awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn | head

# sum bytes per status code
awk '{bytes[$9] += $10} END {for (s in bytes) print s, bytes[s]}' access.log

# join two files on field 1 (like a lookup table)
awk 'NR==FNR {owner[$1]=$2; next} {print $0, owner[$1]}' owners.txt servers.txt
```

The `NR==FNR` idiom is worth memorizing: true only while awk reads the *first* file, so you load it into an array, then enrich the second.

## awk: printf

> **Docs:** [GNU gawk manual](https://www.gnu.org/software/gawk/manual/gawk.html)

```bash
awk '{printf "%-20s %8d\n", $1, $2}' file      # left-pad name, right-pad number
awk '{printf "%.2f%%\n", $1/$2*100}' file      # floats; awk does real math
awk 'BEGIN {printf "%s\t%s\n", "col1", "col2"}'
```

`%s` string, `%d` int, `%f` float, `%x` hex, `%-10s` left-justified width 10. `\n` is not automatic, so add it.

## Common One-Liners

> **Docs:** [GNU gawk manual](https://www.gnu.org/software/gawk/manual/gawk.html)

| Task | Command |
|------|---------|
| Sum column 3 | `awk '{s+=$3} END {print s}' file` |
| Average column 2 | `awk '{s+=$2} END {print s/NR}' file` |
| Dedupe keeping order | `awk '!seen[$0]++' file` |
| Extract between markers | `sed -n '/START/,/END/p' file` |
| Pick CSV columns 1 & 4 | `awk -F',' '{print $1","$4}' file.csv` |
| Strip trailing whitespace | `sed -i 's/[ \t]*$//' file` |
| Number all lines | `awk '{print NR": "$0}' file` |
| Print file between line 100-110 | `sed -n '100,110p' file` |
| Comment out a config line | `sed -i '/^listen 80/s/^/#/' nginx.conf` |
| Top talkers in a log | `awk '{print $1}' log \| sort \| uniq -c \| sort -rn \| head` |

More pipeline building blocks (`sort`, `uniq`, `xargs`, `jq`) live in the [Bash CLI One-Liners Cheatsheet](https://karandeepsingh.ca/cheatsheets/bash/bash-cli/).

## When to Use What

| Tool | Reach for it when... |
|------|---------------------|
| `grep` | You only need to *find* lines (fastest, simplest) |
| `cut` | Fixed single-char delimiter, simple column extraction |
| `sed` | Line-oriented *editing*: substitute, delete, insert, in-place |
| `awk` | Fields, arithmetic, aggregation, conditions across columns |
| Python | Nested data, JSON, anything a one-liner can't express cleanly |

Rule of thumb: `grep` to find, `sed` to change, `awk` to compute. If your awk program grows past a few lines, it's a script. Commit it with a shebang and tests.

**Question:** You have a 2 GB log and need a count per host. Is your first instinct `grep | sort | uniq -c`, and how much of that file does it read compared to one awk pass?

## Related Cheatsheets

- [Bash Basics](https://karandeepsingh.ca/cheatsheets/bash/bash-basics/): quoting, conditionals, loops, and here-docs, the scripting frame these sed and awk snippets slot into
- [Bash CLI One-Liners](https://karandeepsingh.ca/cheatsheets/bash/bash-cli/): the rest of the pipeline toolkit (find, `grep -o`, `sort | uniq -c`, jq, and curl recipes)
- [Ubuntu Terminal](https://karandeepsingh.ca/cheatsheets/terminal/ubuntu-terminal/): apt, systemd, journalctl, and ufw, which is where the logs and config files you'll run these one-liners against actually live

The rest of the shell pages are in the [Bash group](https://karandeepsingh.ca/cheatsheets/bash/); the full [cheatsheet library](https://karandeepsingh.ca/cheatsheets/) has everything else.