# Log Aggregator From Scratch in Go

- URL: https://karandeepsingh.ca/posts/log-aggregator-from-scratch-go/
- Author: Karandeep Singh
- Published: 2026-02-16
- Tags: golang, linux, logging, devops
- Summary: Build a log aggregator from scratch in Go. Tail files with inotify, survive log rotation, parse syslog and JSON, fan in from multiple sources, and wire up graceful shutdown.

Every production system writes logs. Your web server writes access logs, your application writes error logs, systemd writes journal logs. These files pile up across dozens of paths under `/var/log/`, and when something breaks at 3 AM, you need one tool that watches all of them, filters out the noise, and gives you structured output you can pipe into whatever comes next.

That's what a log aggregator does. Tools like Fluentd, Logstash, and Vector all solve this problem, but they're large, complex systems with plugin architectures and YAML configs. The core of what they do is surprisingly simple: tail files, parse lines, filter, and forward.

We're going to build that core from scratch in Go. About 200 lines, three files, zero external dependencies beyond one inotify wrapper. By the end, you'll understand how log aggregation actually works at the Linux kernel level: inotify watches, file descriptors surviving rotation, fan-in concurrency, and signal handling.

![Network infrastructure cables in a server room for log data collection](https://karandeepsingh.ca/images/posts/log-aggregator-go.jpg)

## What We're Building

A CLI tool called **logtail** that tails multiple log files in real-time, parses them, filters by severity, and outputs structured JSON Lines.

**The journey:**
1. Tail a single file with polling
2. Watch with inotify via fsnotify (zero CPU when idle)
3. Survive log rotation (logrotate `create` mode)
4. Parse syslog and JSON log formats
5. Fan in from multiple files with goroutines
6. Filter by severity and output JSON Lines
7. Graceful shutdown with signal handling

## Prerequisites

- **Go 1.21+** installed
- **Linux** (inotify is a Linux kernel feature)
- A terminal with `logger` command available (standard on most distros)

> **macOS note:** fsnotify uses kqueue on macOS, so the code runs there too, but the inotify-specific concepts (like `strace` demos) require Linux.

## Step 1: Tail a Single File with Polling

**What:** Read new lines from the end of a file in a loop, like `tail -f`.

**Why:** This is the most basic building block of any log aggregator: watch a file and print new lines as they appear. Every tool from `tail` to Fluentd starts here.

When you open a file and call `Seek(0, io.SeekEnd)`, the read cursor jumps to the end. From that point, any new data written to the file appears after the cursor. A polling loop just checks for new data every few hundred milliseconds.

```bash
mkdir logtail && cd logtail
go mod init logtail
```

Create a test log directory we'll use throughout the article:

```bash
mkdir -p /tmp/testlogs
echo '{"level":"info","msg":"service started"}' > /tmp/testlogs/app.log
```

**`main.go`**
```go
package main

import (
	"bufio"
	"fmt"
	"io"
	"os"
	"time"
)

func main() {
	path := "/tmp/testlogs/app.log"

	f, err := os.Open(path)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(1)
	}
	defer f.Close()

	// Jump to end — we only want new lines
	f.Seek(0, io.SeekEnd)

	scanner := bufio.NewScanner(f)
	for {
		for scanner.Scan() {
			fmt.Println(scanner.Text())
		}
		time.Sleep(200 * time.Millisecond)
	}
}
```

`os.Open` gives us a file descriptor. `Seek(0, io.SeekEnd)` moves the cursor to the end of the file so we skip existing content. The inner `for scanner.Scan()` reads all available lines, then we sleep and try again.

Build and run:

```bash
go build -o logtail . && ./logtail
```

In another terminal, append lines:

```bash
echo '{"level":"error","msg":"connection refused"}' >> /tmp/testlogs/app.log
echo '{"level":"info","msg":"retrying"}' >> /tmp/testlogs/app.log
```

You'll see each line printed by `logtail` after a brief delay. It works, but watch what happens when you check CPU usage:

```bash
top -p $(pgrep logtail)
```

The process wakes up every 200ms even when nothing is being written. On a busy server with 50 log files, that's 250 wakeups per second doing nothing. We need the kernel to tell us when data arrives instead of constantly asking.

## Step 2: Watch with inotify via fsnotify

**What:** Replace the polling loop with Linux's inotify subsystem, so we wake up only when the file is actually modified.

**Why:** inotify is a Linux kernel feature that lets a process subscribe to filesystem events. Instead of polling, the kernel sends an event the instant a file is written to. CPU usage drops to zero when logs are quiet.

You can see inotify in action with `strace`:

```bash
strace -e inotify_add_watch,read tail -f /tmp/testlogs/app.log
```

You'll see `inotify_add_watch` register the file, then `read` blocks until an event arrives. No polling.

Install the fsnotify package (a thin wrapper around inotify that Hugo itself uses):

```bash
go get github.com/fsnotify/fsnotify
```

The sleep loop gives way to an event loop, and time drops out of the imports:

@@ -5,7 +5,8 @@
 	"fmt"
 	"io"
 	"os"
-	"time"
+
+	"github.com/fsnotify/fsnotify"
 )
 
 func main() {
@@ -18,14 +19,23 @@
 	}
 	defer f.Close()
 
-	// Jump to end — we only want new lines
 	f.Seek(0, io.SeekEnd)
+	scanner := bufio.NewScanner(f)
 
-	scanner := bufio.NewScanner(f)
-	for {
-		for scanner.Scan() {
-			fmt.Println(scanner.Text())
+	watcher, err := fsnotify.NewWatcher()
+	if err != nil {
+		fmt.Println("Error:", err)
+		os.Exit(1)
+	}
+	defer watcher.Close()
+
+	watcher.Add(path)
+
+	for event := range watcher.Events {
+		if event.Has(fsnotify.Write) {
+			for scanner.Scan() {
+				fmt.Println(scanner.Text())
+			}
 		}
-		time.Sleep(200 * time.Millisecond)
 	}
 }
```

The full listing, for reference:

**`main.go`**
```go
package main

import (
	"bufio"
	"fmt"
	"io"
	"os"

	"github.com/fsnotify/fsnotify"
)

func main() {
	path := "/tmp/testlogs/app.log"

	f, err := os.Open(path)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(1)
	}
	defer f.Close()

	f.Seek(0, io.SeekEnd)
	scanner := bufio.NewScanner(f)

	watcher, err := fsnotify.NewWatcher()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(1)
	}
	defer watcher.Close()

	watcher.Add(path)

	for event := range watcher.Events {
		if event.Has(fsnotify.Write) {
			for scanner.Scan() {
				fmt.Println(scanner.Text())
			}
		}
	}
}
```

`fsnotify.NewWatcher()` creates an inotify instance. `watcher.Add(path)` registers the file. The `for event := range watcher.Events` loop blocks until the kernel delivers an event, meaning zero CPU while waiting.

Build and run:

```bash
go build -o logtail . && ./logtail
```

Append lines in another terminal and they appear instantly. Check CPU with `top` and it's 0.0% when idle. That's inotify doing its job.

But there's a problem. Simulate log rotation:

```bash
mv /tmp/testlogs/app.log /tmp/testlogs/app.log.1
echo '{"level":"error","msg":"this line is lost"}' > /tmp/testlogs/app.log
```

Nothing prints. The watcher is still pointing at the old file (now `app.log.1`). The new `app.log` is a completely different inode, and our watcher doesn't know about it.

## Step 3: Survive Log Rotation

**What:** Watch the directory instead of the file, so we detect when logrotate creates a new file.

**Why:** `logrotate` is the standard Linux tool for rotating logs. In `create` mode (the default), it renames the current file and creates a fresh one. Any process watching the old file descriptor keeps reading from the renamed file. The new file gets missed entirely. This is why every production log aggregator watches directories.

The fix: watch `/tmp/testlogs/` for `Create` events. When we see a new file matching our target name, re-open it.

**`main.go`**
```go
package main

import (
	"bufio"
	"context"
	"fmt"
	"io"
	"os"
	"path/filepath"

	"github.com/fsnotify/fsnotify"
)

func tailFile(ctx context.Context, path string, out chan<- string) {
	dir := filepath.Dir(path)
	base := filepath.Base(path)

	watcher, err := fsnotify.NewWatcher()
	if err != nil {
		fmt.Println("Error:", err)
		return
	}
	defer watcher.Close()

	watcher.Add(dir) // Watch directory, not file

	f, scanner := openAndSeekEnd(path)
	if f != nil {
		defer f.Close()
	}

	for {
		select {
		case <-ctx.Done():
			return
		case event := <-watcher.Events:
			if event.Has(fsnotify.Write) && filepath.Base(event.Name) == base {
				for scanner.Scan() {
					out <- scanner.Text()
				}
			}
			if event.Has(fsnotify.Create) && filepath.Base(event.Name) == base {
				// Log was rotated — new file appeared
				if f != nil {
					f.Close()
				}
				f, scanner = openAndSeekEnd(path)
			}
		case err := <-watcher.Errors:
			fmt.Println("Watcher error:", err)
		}
	}
}

func openAndSeekEnd(path string) (*os.File, *bufio.Scanner) {
	f, err := os.Open(path)
	if err != nil {
		return nil, bufio.NewScanner(os.Stdin)
	}
	f.Seek(0, io.SeekEnd)
	return f, bufio.NewScanner(f)
}

func main() {
	ctx := context.Background()
	out := make(chan string, 100)

	go tailFile(ctx, "/tmp/testlogs/app.log", out)

	for line := range out {
		fmt.Println(line)
	}
}
```

Now we watch the directory. When `fsnotify.Create` fires for our filename, we close the old file descriptor and open the new one. The `context.Context` parameter and `out` channel will become important in Step 5 when we fan in from multiple files.

Build and test rotation:

```bash
go build -o logtail . && ./logtail
```

In another terminal:

```bash
echo '{"level":"info","msg":"before rotation"}' >> /tmp/testlogs/app.log
mv /tmp/testlogs/app.log /tmp/testlogs/app.log.1
echo '{"level":"error","msg":"after rotation"}' > /tmp/testlogs/app.log
echo '{"level":"info","msg":"still working"}' >> /tmp/testlogs/app.log
```

All three lines print. Log rotation survived.

## Step 4: Parse Syslog and JSON Formats

**What:** Parse raw log lines into a structured format, supporting both syslog (RFC 3164) and JSON.

**Why:** Raw log lines are strings. To filter by severity or output structured JSON Lines, we need to parse them into fields. Most Linux services write syslog format, while modern applications write JSON. A real log aggregator handles both.

Syslog RFC 3164 looks like this:

```bash
# Generate a syslog entry
logger -p user.err "database connection failed"
# Output: Feb 16 14:30:01 myhost root: database connection failed
```

The format is: `<timestamp> <hostname> <tag>: <message>`, with severity encoded in the facility/priority byte.

**`parser.go`**
```go
package main

import (
	"encoding/json"
	"regexp"
	"strings"
	"time"
)

type ParsedLine struct {
	Timestamp string `json:"timestamp"`
	Host      string `json:"host"`
	Level     string `json:"level"`
	Message   string `json:"message"`
	Source    string `json:"source"`
}

var syslogRegex = regexp.MustCompile(
	`^(?P<timestamp>\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2})\s+` +
		`(?P<host>\S+)\s+` +
		`(?P<tag>[^:]+):\s*` +
		`(?P<message>.+)$`,
)

func parseLine(raw string, source string) *ParsedLine {
	// Try JSON first — cheaper than regex
	if strings.HasPrefix(strings.TrimSpace(raw), "{") {
		return parseJSON(raw, source)
	}
	return parseSyslog(raw, source)
}

func parseJSON(raw string, source string) *ParsedLine {
	var fields map[string]interface{}
	if err := json.Unmarshal([]byte(raw), &fields); err != nil {
		return fallbackParse(raw, source)
	}

	p := &ParsedLine{
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		Source:    source,
		Level:     "info",
	}

	if v, ok := fields["level"].(string); ok {
		p.Level = strings.ToLower(v)
	}
	if v, ok := fields["msg"].(string); ok {
		p.Message = v
	} else if v, ok := fields["message"].(string); ok {
		p.Message = v
	}
	if v, ok := fields["host"].(string); ok {
		p.Host = v
	}

	return p
}

func parseSyslog(raw string, source string) *ParsedLine {
	matches := syslogRegex.FindStringSubmatch(raw)
	if matches == nil {
		return fallbackParse(raw, source)
	}

	return &ParsedLine{
		Timestamp: matches[1],
		Host:      matches[2],
		Level:     "info", // syslog priority requires the PRI header to determine level
		Message:   matches[4],
		Source:    source,
	}
}

func fallbackParse(raw string, source string) *ParsedLine {
	return &ParsedLine{
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		Level:     "info",
		Message:   raw,
		Source:    source,
	}
}
```

`strings.HasPrefix` sniffs the format. If the line starts with `{`, try JSON first. This is fast because JSON parsing only happens when the prefix matches. The syslog regex uses named groups for readability.

Notice the `fallbackParse` function. Without it, a malformed line like `not a valid log` would cause `parseSyslog` to return `nil`, and any caller doing `parsed.Level` would panic with a nil pointer dereference. The fallback ensures every line produces a valid `ParsedLine`. Malformed lines just get the raw text as the message.

Test the parser by updating `main()` temporarily:

```go
func main() {
	ctx := context.Background()
	out := make(chan string, 100)

	go tailFile(ctx, "/tmp/testlogs/app.log", out)

	for line := range out {
		parsed := parseLine(line, "/tmp/testlogs/app.log")
		fmt.Printf("[%s] %s: %s\n", parsed.Level, parsed.Source, parsed.Message)
	}
}
```

```bash
go build -o logtail . && ./logtail
```

In another terminal:

```bash
echo '{"level":"error","msg":"disk full"}' >> /tmp/testlogs/app.log
echo 'Feb 16 14:30:01 myhost sshd: Failed password for root' >> /tmp/testlogs/app.log
echo 'some garbage line' >> /tmp/testlogs/app.log
```

All three lines parse without crashing. JSON, syslog, and the garbage line all produce output.

## Step 5: Fan-in from Multiple Files

**What:** Tail multiple log files concurrently and merge their output into a single stream.

**Why:** A real server has dozens of log files: `/var/log/syslog`, `/var/log/auth.log`, `/var/log/nginx/error.log`, application logs. A log aggregator watches all of them simultaneously. In Go, the fan-in pattern uses one goroutine per file, all sending to a shared channel.

Create a second test log:

```bash
echo '{"level":"warn","msg":"high latency"}' > /tmp/testlogs/access.log
```

**Update `main.go`:**
```go
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"
	"sync"
)

func main() {
	files := []string{
		"/tmp/testlogs/app.log",
		"/tmp/testlogs/access.log",
	}

	ctx := context.Background()
	out := make(chan string, 100)
	var wg sync.WaitGroup

	for _, path := range files {
		wg.Add(1)
		go func(p string) {
			defer wg.Done()
			tailFile(ctx, p, out)
		}(path)
	}

	// Close the channel when all tailers finish
	go func() {
		wg.Wait()
		close(out)
	}()

	for line := range out {
		parsed := parseLine(line, "")
		b, _ := json.Marshal(parsed)
		fmt.Println(string(b))
	}
}
```

Each file gets its own goroutine running `tailFile`. All goroutines send to the same `out` channel. That's the fan-in. `sync.WaitGroup` tracks how many goroutines are running.

The goroutine that calls `wg.Wait()` followed by `close(out)` is critical. Without it, `main()` would exit immediately after spawning the goroutines, and they'd be orphaned. The `for line := range out` loop blocks `main()` until the channel is closed, and the channel closes only when all tailers finish.

Try removing the `wg.Wait()` / `close(out)` goroutine and the `range out` loop, replacing them with just `select {}`. Then run the program and hit Ctrl+C. You'll see the goroutines were running, but there was no clean coordination. We'll fix this properly in Step 7 with signal handling.

Build and test:

```bash
go build -o logtail . && ./logtail
```

In another terminal:

```bash
echo '{"level":"error","msg":"app crashed"}' >> /tmp/testlogs/app.log
echo '{"level":"warn","msg":"slow query"}' >> /tmp/testlogs/access.log
```

Both lines appear in the output stream as JSON Lines.

## Step 6: Filter by Severity and Output JSON Lines

**What:** Add a `--level` flag that filters out log lines below a given severity threshold, and write output through a buffered writer.

**Why:** When an incident happens, you don't want to wade through thousands of info lines. Severity filtering (like `*.err` in rsyslog.conf) lets you focus on what matters. Buffered I/O reduces system calls. Instead of one `write()` per line, the buffer batches them.

You can see the difference with `strace`:

```bash
# Unbuffered: one write() syscall per line
strace -c echo "line" > /dev/null

# Buffered: far fewer write() calls for the same data
strace -c cat /var/log/syslog > /dev/null
```

**`filter.go`**
```go
package main

var severityRank = map[string]int{
	"emergency": 0,
	"alert":     1,
	"critical":  2,
	"error":     3,
	"warn":      4,
	"warning":   4,
	"notice":    5,
	"info":      6,
	"debug":     7,
}

func meetsLevel(lineLevel string, threshold string) bool {
	lineRank, ok := severityRank[lineLevel]
	if !ok {
		return true // Unknown levels always pass
	}
	threshRank, ok := severityRank[threshold]
	if !ok {
		return true
	}
	return lineRank <= threshRank
}
```

Syslog severity levels range from 0 (emergency) to 7 (debug). Lower numbers are more severe. `meetsLevel` returns true if the line's severity is at or above the threshold.

**Update `main.go` with flag parsing and buffered output:**
```go
package main

import (
	"bufio"
	"context"
	"encoding/json"
	"flag"
	"fmt"
	"os"
	"sync"
)

func main() {
	level := flag.String("level", "debug", "Minimum severity level (debug, info, warn, error, critical)")
	flag.Parse()

	files := flag.Args()
	if len(files) == 0 {
		fmt.Fprintln(os.Stderr, "Usage: logtail [--level LEVEL] FILE [FILE...]")
		os.Exit(1)
	}

	ctx := context.Background()
	out := make(chan string, 100)
	var wg sync.WaitGroup

	for _, path := range files {
		wg.Add(1)
		go func(p string) {
			defer wg.Done()
			tailFile(ctx, p, out)
		}(path)
	}

	go func() {
		wg.Wait()
		close(out)
	}()

	bw := bufio.NewWriterSize(os.Stdout, 32*1024)

	for line := range out {
		parsed := parseLine(line, "")
		if !meetsLevel(parsed.Level, *level) {
			continue
		}
		b, _ := json.Marshal(parsed)
		fmt.Fprintln(bw, string(b))
	}
}
```

Now the tool accepts file paths as arguments and a `--level` flag. The `bufio.NewWriterSize` creates a 32KB buffer. Lines are written to the buffer, and the buffer flushes to stdout when it fills up or when... wait.

Build and run:

```bash
go build -o logtail . && ./logtail --level error /tmp/testlogs/app.log /tmp/testlogs/access.log
```

In another terminal:

```bash
echo '{"level":"info","msg":"routine check"}' >> /tmp/testlogs/app.log
echo '{"level":"error","msg":"disk full"}' >> /tmp/testlogs/app.log
echo '{"level":"warn","msg":"high memory"}' >> /tmp/testlogs/access.log
```

Only the error line appears. Info and warn are filtered out. But try killing the process with Ctrl+C right after sending a few error lines. Some lines might be missing from the output. The buffer hasn't been flushed. Without `defer bw.Flush()`, the last lines sitting in the 32KB buffer are silently lost when the process exits. We'll fix this in the next step.

## Step 7: Graceful Shutdown with Signal Handling

**What:** Catch SIGINT and SIGTERM, cancel all goroutines, and flush the output buffer before exiting.

**Why:** In production, a log aggregator runs as a systemd service. When you `systemctl restart` it, systemd sends SIGTERM. If the process doesn't handle that signal, any buffered data is lost. For a log forwarder, that means missing log lines, exactly the lines from the moment something went wrong.

```bash
# See all signals
kill -l

# SIGINT (2) = Ctrl+C
# SIGTERM (15) = kill <pid>, systemd stop
```

Signal handling and one deferred flush are the whole change:

@@ -7,7 +7,9 @@
 	"flag"
 	"fmt"
 	"os"
+	"os/signal"
 	"sync"
+	"syscall"
 )
 
 func main() {
@@ -20,7 +22,17 @@
 		os.Exit(1)
 	}
 
-	ctx := context.Background()
+	ctx, cancel := context.WithCancel(context.Background())
+
+	// Catch SIGINT and SIGTERM
+	sigCh := make(chan os.Signal, 1)
+	signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
+	go func() {
+		sig := <-sigCh
+		fmt.Fprintf(os.Stderr, "\nReceived %s, shutting down...\n", sig)
+		cancel()
+	}()
+
 	out := make(chan string, 100)
 	var wg sync.WaitGroup
 
@@ -38,6 +50,7 @@
 	}()
 
 	bw := bufio.NewWriterSize(os.Stdout, 32*1024)
+	defer bw.Flush() // Flush remaining data on exit
 
 	for line := range out {
 		parsed := parseLine(line, "")
```

The full listing, for reference:

**Final `main.go`:**
```go
package main

import (
	"bufio"
	"context"
	"encoding/json"
	"flag"
	"fmt"
	"os"
	"os/signal"
	"sync"
	"syscall"
)

func main() {
	level := flag.String("level", "debug", "Minimum severity level (debug, info, warn, error, critical)")
	flag.Parse()

	files := flag.Args()
	if len(files) == 0 {
		fmt.Fprintln(os.Stderr, "Usage: logtail [--level LEVEL] FILE [FILE...]")
		os.Exit(1)
	}

	ctx, cancel := context.WithCancel(context.Background())

	// Catch SIGINT and SIGTERM
	sigCh := make(chan os.Signal, 1)
	signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
	go func() {
		sig := <-sigCh
		fmt.Fprintf(os.Stderr, "\nReceived %s, shutting down...\n", sig)
		cancel()
	}()

	out := make(chan string, 100)
	var wg sync.WaitGroup

	for _, path := range files {
		wg.Add(1)
		go func(p string) {
			defer wg.Done()
			tailFile(ctx, p, out)
		}(path)
	}

	go func() {
		wg.Wait()
		close(out)
	}()

	bw := bufio.NewWriterSize(os.Stdout, 32*1024)
	defer bw.Flush() // Flush remaining data on exit

	for line := range out {
		parsed := parseLine(line, "")
		if !meetsLevel(parsed.Level, *level) {
			continue
		}
		b, _ := json.Marshal(parsed)
		fmt.Fprintln(bw, string(b))
	}
}
```

`signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)` registers our channel to receive these signals instead of the default behavior (immediate termination). When a signal arrives, we call `cancel()` on the context. That context was passed to every `tailFile` goroutine, and they all check `ctx.Done()` in their select loop and exit cleanly. Once all goroutines exit, `wg.Wait()` returns, the channel closes, the `range out` loop ends, and `defer bw.Flush()` writes any remaining buffered data.

Build and test:

```bash
go build -o logtail . && ./logtail --level info /tmp/testlogs/app.log /tmp/testlogs/access.log
```

In another terminal, send some lines then immediately kill the process:

```text
$ for i in $(seq 1 5); do echo "{\"level\":\"error\",\"msg\":\"line $i\"}" >> /tmp/testlogs/app.log; done
> kill -TERM $(pgrep logtail)
Received terminated, shutting down...
{"timestamp":"2026-02-16T14:00:00Z","host":"","level":"error","message":"line 1","source":""}
{"timestamp":"2026-02-16T14:00:00Z","host":"","level":"error","message":"line 2","source":""}
{"timestamp":"2026-02-16T14:00:00Z","host":"","level":"error","message":"line 3","source":""}
{"timestamp":"2026-02-16T14:00:00Z","host":"","level":"error","message":"line 4","source":""}
{"timestamp":"2026-02-16T14:00:00Z","host":"","level":"error","message":"line 5","source":""}
```

All five lines present. The `defer bw.Flush()` saved them. Without it, some or all would have been lost in the buffer.

## What We Built

Starting from a `time.Sleep` polling loop, we incrementally built a real log aggregator:

1. **Polling tail:** read new lines from the end of a file in a loop
2. **inotify watching:** zero-CPU file monitoring via the Linux kernel
3. **Rotation survival:** watch directories, detect new files, re-open seamlessly
4. **Format parsing:** syslog RFC 3164 and JSON with automatic format detection
5. **Fan-in concurrency:** one goroutine per file, shared output channel
6. **Severity filtering:** filter by level with buffered output for performance
7. **Graceful shutdown:** signal handling, context cancellation, buffer flushing

All of this in about 200 lines of Go across three files. That's the core of what Fluentd, Logstash, and Vector do under the hood.

## Cleanup

```bash
rm -rf /tmp/testlogs/ && rm ./logtail
```

## Next Steps

This log aggregator covers the fundamentals. Production-grade tools add:

- **TCP/UDP forwarding:** ship JSON Lines to a remote syslog server or Elasticsearch
- **Glob patterns:** watch `/var/log/*.log` instead of listing files individually
- **Backpressure:** slow down reading when the output destination can't keep up
- **Metrics:** count lines processed, bytes read, errors per file
- **Systemd journal:** read from journald's binary format via the sd-journal API
- **Multi-line joining:** merge Java stack traces into a single log entry

Check out [Containers From Scratch in Go](https://karandeepsingh.ca/posts/containers-from-scratch-go/) for another from-scratch project that uses the same Linux kernel features (namespaces instead of inotify), or [Go + Nginx Reverse Proxy](https://karandeepsingh.ca/posts/go-nginx-reverse-proxy/) for the same graceful shutdown pattern applied to an HTTP reverse proxy.

## Cheat Sheet

Quick reference for log aggregation concepts in Go.

**Tail from end of file:**
```go
f, _ := os.Open(path)
f.Seek(0, io.SeekEnd)
scanner := bufio.NewScanner(f)
```

**Watch with fsnotify (inotify):**
```go
watcher, _ := fsnotify.NewWatcher()
watcher.Add("/var/log/")
for event := range watcher.Events {
    if event.Has(fsnotify.Write) { /* read new lines */ }
    if event.Has(fsnotify.Create) { /* file rotated, re-open */ }
}
```

**Fan-in from multiple files:**
```go
out := make(chan string, 100)
var wg sync.WaitGroup
for _, path := range files {
    wg.Add(1)
    go func(p string) {
        defer wg.Done()
        tailFile(ctx, p, out)
    }(path)
}
go func() { wg.Wait(); close(out) }()
```

**Severity filtering:**
```go
rank := map[string]int{"emergency": 0, "error": 3, "warn": 4, "info": 6, "debug": 7}
func meetsLevel(line, threshold string) bool {
    return rank[line] <= rank[threshold]
}
```

**Buffered JSON Lines output:**
```go
bw := bufio.NewWriterSize(os.Stdout, 32*1024)
defer bw.Flush()
b, _ := json.Marshal(parsed)
fmt.Fprintln(bw, string(b))
```

**Graceful shutdown:**
```go
ctx, cancel := context.WithCancel(context.Background())
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
go func() { <-sigCh; cancel() }()
// All goroutines check ctx.Done() in their select loop
```

**Key concepts to remember:**
- **inotify** wakes your process only when a file changes, so CPU usage is zero when logs are quiet
- **Log rotation** breaks file watchers because the inode changes. Watch directories, not files
- **Fan-in** in Go means multiple goroutines sending to one channel. The receiver doesn't know or care how many senders exist
- **`sync.WaitGroup`** tracks goroutine lifecycle: `Add` before launching, `Done` when finished, `Wait` to block until all complete
- **`context.WithCancel`** propagates shutdown to all goroutines. One `cancel()` call stops everything
- **Buffered writers** reduce syscalls but require explicit `Flush()`. `defer bw.Flush()` prevents data loss
- **`signal.Notify`** intercepts OS signals. Without it, SIGTERM kills immediately and buffers are lost
- Syslog severity goes from 0 (emergency) to 7 (debug). Lower numbers are more severe

## References and Further Reading

- Kerrisk, M. (2010). [The Linux Programming Interface](https://man7.org/tlpi/). Chapter 19 covers inotify in depth
- [fsnotify documentation](https://pkg.go.dev/github.com/fsnotify/fsnotify), the Go inotify wrapper used in this article
- [RFC 3164: The BSD Syslog Protocol](https://datatracker.ietf.org/doc/html/rfc3164), the syslog format we parse in Step 4
- Donovan, A. & Kernighan, B. (2015). [The Go Programming Language](https://www.gopl.io/). Chapters 8-9 cover goroutines and channels

**Question:** What would you add to this log aggregator next: TCP forwarding, glob patterns, or multi-line joining?