# Sed for Log Analysis: Errors, Time Filters, Patterns

- URL: https://karandeepsingh.ca/posts/sed-for-log-analysis/
- Author: Karandeep Singh
- Published: 2026-05-05
- Updated: 2026-07-16
- Series: Mastering sed
- Tags: bash, sed
- Summary: Practical sed recipes for working with production logs — error extraction, time-range filtering, multi-line stack trace handling, anonymization, and pre-pipeline cleanup before grep/awk.

Sooner or later a problem lands on a box where the logs are all you have. No Splunk, no Datadog forwarder, and a strict "no installing tools" policy. Just the GNU coreutils that shipped with the AMI. For me that box is usually a Jenkins build machine — I've spent years digging through build and application logs with nothing fancier than a shell.

You have `grep`. You have `awk`. And you have `sed`. Every pattern in this article is the kind you reach for in exactly that situation, and the kind worth folding into a reusable script afterward.

The thing nobody tells you about log analysis is that `grep` runs out of road quickly. As soon as your problem touches a multi-line stack trace, a time window, or PII redaction before sharing logs with a vendor, you need stream editing — and sed is the tool that's already there. This is a companion to the [Sed Cheat Sheet: 30 One-Liners from Real Production Logs](https://karandeepsingh.ca/posts/sed-command-cheat-sheet-30-essential-one-liners/), but focused entirely on log work.

## 1. Extract Errors from Mixed-Severity Logs

The simplest log task: pull only the ERROR lines from a file that mixes INFO, WARN, DEBUG, and ERROR.

The naive approach is `grep ERROR app.log`. That works until you realize sed gives you something grep can't: combine extraction with edits in a single pass. No re-piping, no buffering twice.

**Input (`app.log`):**

```
2024-10-22 14:02:11 INFO  request_id=ab12 status=200 path=/health
2024-10-22 14:02:14 ERROR request_id=cd34 db connection refused (host=db-primary.local)
2024-10-22 14:02:14 INFO  heartbeat ok
2024-10-22 14:02:18 WARN  request_id=ef56 slow query 1240ms
2024-10-22 14:02:22 ERROR heartbeat probe failed (host=svc-b.local)
2024-10-22 14:02:25 ERROR request_id=gh78 payment gateway 502
```

**Command:**

```bash
sed -n '/ERROR/p' app.log
```

**Output:**

```
2024-10-22 14:02:14 ERROR request_id=cd34 db connection refused (host=db-primary.local)
2024-10-22 14:02:22 ERROR heartbeat probe failed (host=svc-b.local)
2024-10-22 14:02:25 ERROR request_id=gh78 payment gateway 502
```

The `-n` suppresses default printing, and `/ERROR/p` prints only matching lines. Now the bug: heartbeat probe failures are noise. A flaky internal health check fires constantly and drowns out the real signal.

**Fixed command (exclude noisy errors):**

```bash
sed -n '/ERROR/{/heartbeat/!p;}' app.log
```

The `{...}` groups commands. Inside, `/heartbeat/!p` means "if the line does not match heartbeat, print it." So you get ERROR lines that aren't heartbeat noise.

**Output:**

```
2024-10-22 14:02:14 ERROR request_id=cd34 db connection refused (host=db-primary.local)
2024-10-22 14:02:25 ERROR request_id=gh78 payment gateway 502
```

For incident triage, a count of distinct errors per minute is often what you want. Sed pairs with `cut` and `uniq` to do this without awk:

```bash
sed -n '/ERROR/{/heartbeat/!p;}' app.log \
  | cut -c1-16 \
  | uniq -c \
  | sort -rn \
  | head -20
```

That gives you the top 20 minutes by error volume, which is usually the first thing on-call wants to see.

_Use this when:_ you need extraction plus inline filtering in one pass, or when grep's `-v` would chain awkwardly with other transforms.

## 2. Filter Logs by Time Range

Most of the time you don't need the entire log — you need a narrow window around the failure.

Sed handles this with an _address range_ using two regex patterns. The first match opens the range, the second closes it. Everything between (inclusive) prints.

**Command:**

```bash
sed -n '/2024-10-22 02:00/,/2024-10-22 02:20/p' app.log > window.log
```

**Input (excerpt):**

```
2024-10-22 01:58:42 INFO  cron job_id=4101 done
2024-10-22 02:00:11 INFO  request_id=zz01 status=200
2024-10-22 02:08:14 ERROR request_id=zz77 retry budget exhausted
2024-10-22 02:14:02 ERROR request_id=zz92 dns lookup failed: payments.local
2024-10-22 02:20:00 INFO  cron job_id=4102 start
2024-10-22 02:25:11 INFO  request_id=ab44 status=200
```

**Output (`window.log`):**

```
2024-10-22 02:00:11 INFO  request_id=zz01 status=200
2024-10-22 02:08:14 ERROR request_id=zz77 retry budget exhausted
2024-10-22 02:14:02 ERROR request_id=zz92 dns lookup failed: payments.local
2024-10-22 02:20:00 INFO  cron job_id=4102 start
```

The first pattern that matches opens the range; the first match _after that_ closes it. If the closing pattern never matches, sed reads to EOF — useful when you want "everything since 02:00" and don't have a known end.

The common bug with this pattern: timestamp formats vary across services. One service might log ISO 8601 with millisecond precision (`2024-10-22T02:08:14.331Z`), a legacy worker might log syslog format (`Oct 22 02:08:14`), and Kubernetes events use RFC3339 with timezone (`2024-10-22T02:08:14-06:00`). A naive range pattern matches one and misses the others.

For mixed sources, anchor on the time portion only and accept any prefix:

```bash
sed -n '/02:0[0-9]:/,/02:2[0-9]:/p' merged.log
```

The character classes `0[0-9]` and `2[0-9]` match minutes 00-09 and 20-29 respectively. Less precise, but resilient when you can't normalize timestamps upstream.

For live tailing within a known window, pipe `tail -f` into sed but use line buffering or you'll wait forever for the buffer to flush:

```bash
tail -f app.log | stdbuf -oL sed -n '/14:00/,/15:00/p'
```

`stdbuf -oL` forces line-buffered output so each match prints immediately. Without it, sed buffers before flushing — which means in a slow log stream, your "live" filter is anything but.

_Use this when:_ you have a known incident window and want to scope analysis to it, or when a downstream tool (less, awk, jq) would otherwise have to scan the entire file.

## 3. Parse Multi-Line Stack Traces

This is where sed earns its keep. A Java exception spans many lines. A Python traceback spans several. Grep treats every line independently, so when you grep for `NullPointerException`, you get the message line — but not the stack frames you need to see.

The classic approach is sed's `N` command, which appends the next line to the pattern space, joined by an embedded newline. With a pattern that loops `N` while the next line looks like a continuation, you can collapse a whole stack trace onto one line.

**Input (`error.log`, Java):**

```
2024-10-22 02:14:14 ERROR request_id=cd34 NullPointerException at PaymentService
    at com.example.PaymentService.charge(PaymentService.java:142)
    at com.example.BillingHandler.process(BillingHandler.java:88)
    at com.example.RequestRouter.dispatch(RequestRouter.java:201)
Caused by: java.lang.IllegalStateException: connection pool exhausted
    at com.example.db.Pool.acquire(Pool.java:55)
2024-10-22 02:14:15 INFO  request_id=ef56 status=200
```

**Command:**

```bash
sed -E ':a; N; /\n([[:space:]]+at |Caused by:)/{ s/\n/ | /; ba }; P; D' error.log
```

Breaking this down:

- `:a` defines a label called `a`
- `N` appends the next line to pattern space
- `/\n([[:space:]]+at |Caused by:)/{...}` tests whether the line just appended starts with whitespace+`at ` OR `Caused by:` — i.e. whether it's a continuation frame
- inside that block, `s/\n/ | /` joins the continuation onto the line above with a `| ` separator, and `ba` branches back to `a` to pull in the next line
- when the appended line is *not* a continuation, `P` prints the completed first line and `D` deletes it and restarts the cycle with the leftover line still in pattern space — so the next record begins cleanly instead of getting glued onto the previous one

The `P;D` pair is what keeps the next log record (a fresh timestamped line) from being absorbed into the stack trace above it. Use `[[:space:]]` here instead of `\s` because `\s` is a GNU extension. On macOS or Alpine busybox the script will silently fail to match. The portable POSIX class works on every sed.

**Output:**

```
2024-10-22 02:14:14 ERROR request_id=cd34 NullPointerException at PaymentService | 	at com.example.PaymentService.charge(PaymentService.java:142) | 	at com.example.BillingHandler.process(BillingHandler.java:88) | 	at com.example.RequestRouter.dispatch(RequestRouter.java:201) | Caused by: java.lang.IllegalStateException: connection pool exhausted | 	at com.example.db.Pool.acquire(Pool.java:55)
2024-10-22 02:14:15 INFO  request_id=ef56 status=200
```

Now `grep NullPointerException` returns the whole stack trace as one line. Pipe it into log aggregators or feed it to `sort | uniq -c` to find duplicate exceptions.

For Python tracebacks, the continuation pattern is different. Python uses `Traceback (most recent call last):` followed by indented `File "..."` lines:

**Input (Python):**

```
2024-10-22 02:14:14 ERROR request_id=cd34
Traceback (most recent call last):
  File "/srv/app/billing.py", line 142, in charge
    response = api.post(url, json=payload)
  File "/srv/app/api.py", line 88, in post
    raise ConnectionError("dns lookup failed")
ConnectionError: dns lookup failed
2024-10-22 02:14:15 INFO  request_id=ef56 status=200
```

**Command:**

```bash
sed -E ':a; N; /\n(Traceback|  File|    |[A-Za-z_.]+(Error|Exception):)/{ s/\n/ | /; ba }; P; D' error.log
```

This loops `N` while the appended line is a traceback continuation — `Traceback`, two-space-`File`, a four-space indent (the actual source line), or the final `SomethingError:`/`SomethingException:` message that closes the traceback. As soon as a non-continuation line shows up, `P;D` flushes the collapsed traceback and restarts cleanly, so the next record isn't glued on. The output collapses the whole traceback onto one line, ready for `grep` or aggregation.

For deeper coverage of these patterns including hold-space tricks, see [Sed Multiline Patterns: How to Match Across Lines](https://karandeepsingh.ca/posts/sed-multiline-patterns/).

_Use this when:_ you need to grep, count, or alert on exceptions and the message line alone isn't enough — you need the frames too.

## 4. Anonymize PII Before Sharing Logs

Sooner or later you have to share log excerpts with a vendor to debug a third-party issue. A support ticket is not the place to leak email addresses, card numbers, or source IPs.

The pattern: sed substitution with capture groups, applied as a redaction pass before the file ever leaves your machine.

**Input (`raw.log`):**

```
2024-10-22 14:02:14 INFO  user=alice.smith@acme.example.com login from 142.103.18.55
2024-10-22 14:02:18 INFO  charge attempted card=4532-1488-0343-6467 amount=$129.00
2024-10-22 14:02:22 ERROR user=bob_42@gmail.com 3ds verification failed from 99.21.4.211
```

**Email redaction:**

```bash
sed -E 's/[A-Za-z0-9._+-]+@([A-Za-z0-9.-]+)/***@\1/g' raw.log
```

The capture group `\1` keeps the domain (useful for support: they can still see "the bug only hits gmail addresses") while erasing the local part.

**Credit card redaction (16-digit, dashed or spaced):**

```bash
sed -E 's/([0-9]{4})[ -]?([0-9]{4})[ -]?([0-9]{4})[ -]?([0-9]{4})/XXXX-XXXX-XXXX-\4/g' raw.log
```

PCI-DSS allows the last four digits to remain; this preserves them for support correlation while masking the rest.

**IP redaction (keep first two octets for ASN/region context):**

```bash
sed -E 's/([0-9]{1,3}\.[0-9]{1,3})\.[0-9]{1,3}\.[0-9]{1,3}/\1.x.x/g' raw.log
```

**Combined into a single redaction pass:**

```bash
sed -E -e 's/[A-Za-z0-9._+-]+@([A-Za-z0-9.-]+)/***@\1/g' \
       -e 's/([0-9]{4})[ -]?([0-9]{4})[ -]?([0-9]{4})[ -]?([0-9]{4})/XXXX-XXXX-XXXX-\4/g' \
       -e 's/([0-9]{1,3}\.[0-9]{1,3})\.[0-9]{1,3}\.[0-9]{1,3}/\1.x.x/g' \
       raw.log > redacted.log
```

**Output:**

```
2024-10-22 14:02:14 INFO  user=***@acme.example.com login from 142.103.x.x
2024-10-22 14:02:18 INFO  charge attempted card=XXXX-XXXX-XXXX-6467 amount=$129.00
2024-10-22 14:02:22 ERROR user=***@gmail.com 3ds verification failed from 99.21.x.x
```

One portability gotcha: BSD sed (the macOS default) doesn't always handle `-E` the same way GNU sed does, and word boundaries (`\b`) are inconsistent. If you're sharing redaction scripts across a team that mixes Linux and macOS, run them with `--posix` on GNU sed to catch compatibility issues early:

```bash
sed --posix -E -e '...' raw.log
```

For pipelines that must work on both, use `[[:space:]]` instead of `\s`, avoid `\b`, and stick to BRE (basic regex) when possible.

_Use this when:_ you're about to attach logs to a support ticket, paste them into a Slack channel, or store them in a system with broader access than the source.

## 5. Pre-Clean Logs for grep/awk Pipelines

A real chunk of "my pipeline doesn't work" bugs come down to two things: ANSI color codes from a tool that thought it was talking to a terminal, and inconsistent whitespace.

When you tail a Docker container's logs through a CI runner, then pipe them to a log aggregator, you'll often see this:

**Input (`ci.log`, with hidden ANSI codes):**

```
2024-10-22 14:02:14 \x1b[31mERROR\x1b[0m   build failed
2024-10-22 14:02:15 \x1b[33mWARN\x1b[0m    deprecation: foo()
   2024-10-22 14:02:16 INFO	step complete
```

The escape sequences (shown here as `\x1b[31m` for red) are invisible in your terminal but real bytes in the file. When you pipe to grep, your "exact match" on `ERROR` works. When you pipe to awk and split on whitespace, the third field is `\x1b[31mERROR\x1b[0m` instead of `ERROR`, and your downstream conditions silently never match.

**ANSI strip:**

```bash
sed 's/\x1b\[[0-9;]*m//g' ci.log
```

`\x1b` is the escape character (ESC, 0x1B), followed by `[`, then any number of digits and semicolons, then `m`. That's the SGR (Select Graphic Rendition) escape sequence. The `g` flag handles multiple per line.

**Trim leading and trailing whitespace:**

```bash
sed -E 's/^[[:space:]]+//;s/[[:space:]]+$//' ci.log
```

Two substitutions in one sed call, separated by `;`. The first strips leading whitespace; the second strips trailing.

**Normalize tabs to single spaces (so awk's default FS works predictably):**

```bash
sed 's/\t/ /g' ci.log
```

**A reusable normalizer** to keep in `~/bin/log-normalize.sed`:

```bash
# log-normalize.sed - clean log streams before pipelining
s/\x1b\[[0-9;]*m//g
s/\r$//
s/^[[:space:]]+//
s/[[:space:]]+$//
s/\t/ /g
s/  +/ /g
```

Used as:

```bash
kubectl logs deploy/billing-api --tail=10000 \
  | sed -E -f ~/bin/log-normalize.sed \
  | grep ERROR \
  | awk '{print $3, $4}'
```

Now grep and awk work on a known, clean stream. Normalizing the stream this way pays back many times over in "why is my pipeline returning empty?" debugging sessions.

For a deeper look at when sed beats grep and when it doesn't, see [Sed vs Awk vs Grep](https://karandeepsingh.ca/posts/sed-vs-awk-vs-grep/). The short version: if you need to _transform_ before filtering, sed comes first.

_Use this when:_ your downstream tool gives empty or inconsistent results despite the input "looking right" in your terminal.

## Putting It Together: One Pipeline

Suppose you have a large log set across multiple services, a suspected impact window that's only a narrow band of minutes, and no idea yet which errors matter.

Here's a pipeline for exactly that situation, with patterns 1-5 stacked:

```bash
# Step 1: scope to the customer-impact window
sed -n '/2024-10-22 02:0[0-9]:/,/2024-10-22 02:3[0-9]:/p' app.log > window.log

# Step 2: collapse Java stack traces to one line each
sed -E ':a; N; /\n([[:space:]]+at |Caused by:)/{ s/\n/ | /; ba }; P; D' window.log > collapsed.log

# Step 3: extract errors, drop heartbeat noise
sed -n '/ERROR/{/heartbeat/!p;}' collapsed.log > errors.log

# Step 4: redact PII before sharing with the upstream API vendor
sed -E -e 's/[A-Za-z0-9._+-]+@([A-Za-z0-9.-]+)/***@\1/g' \
       -e 's/([0-9]{1,3}\.[0-9]{1,3})\.[0-9]{1,3}\.[0-9]{1,3}/\1.x.x/g' \
       errors.log > shareable.log

# Step 5: count distinct exception types
grep -oE 'Caused by: [A-Za-z.]+Exception' shareable.log \
  | sort | uniq -c | sort -rn
```

That last `sort | uniq -c` gives you a ranked list of exception types inside the window. If one type dominates and clusters around the impact window, that's your breadcrumb — follow it into the service that throws it.

A large log set scoped down to a short ranked list in five commands. From `sed -n '/.../,/.../p'` to a concrete lead in a fraction of the time line-by-line grepping would have taken.

This kind of work is also why getting these patterns right in [Sed in CI/CD Pipelines: Safe Patterns](https://karandeepsingh.ca/posts/sed-in-ci-cd-pipelines/) matters — the line between "useful log filter" and "accidental data corruption" is one missing escape character.

## Related

- [Sed Cheat Sheet: 30 One-Liners from Real Production Logs](https://karandeepsingh.ca/posts/sed-command-cheat-sheet-30-essential-one-liners/) — the pillar reference with all 30 patterns indexed by use case
- [Sed Multiline Patterns: How to Match Across Lines](https://karandeepsingh.ca/posts/sed-multiline-patterns/) — deeper coverage of `N`, `P`, `D`, and hold space for stack traces
- [Sed in CI/CD Pipelines: Safe Patterns](https://karandeepsingh.ca/posts/sed-in-ci-cd-pipelines/) — when in-place sed edits are safe and when they corrupt your repo
- [Sed vs Awk vs Grep](https://karandeepsingh.ca/posts/sed-vs-awk-vs-grep/) — picking the right tool when your pipeline could use any of them