Learn Terraform with AWS from scratch. Start with a single S3 bucket, hit real errors, fix them, …
Log Aggregator From Scratch in Go Log Aggregator From Scratch in Go

Summary
Every production system writes logs. Your web server writes access logs, your application writes error logs, systemd writes journal logs. These files pile up across dozens of paths under /var/log/, and when something breaks at 3 AM, you need one tool that watches all of them, filters out the noise, and gives you structured output you can pipe into whatever comes next.
That’s what a log aggregator does. Tools like Fluentd, Logstash, and Vector all solve this problem, but they’re large, complex systems with plugin architectures and YAML configs. The core of what they do is surprisingly simple: tail files, parse lines, filter, and forward.
We’re going to build that core from scratch in Go. About 200 lines, three files, zero external dependencies beyond one inotify wrapper. By the end, you’ll understand how log aggregation actually works at the Linux kernel level: inotify watches, file descriptors surviving rotation, fan-in concurrency, and signal handling.
What We’re Building
A CLI tool called logtail that tails multiple log files in real-time, parses them, filters by severity, and outputs structured JSON Lines.
The journey:
Expand your knowledge with Building a URL Shortener: From Linux Networking to Go
- Tail a single file with polling
- Watch with inotify via fsnotify (zero CPU when idle)
- Survive log rotation (logrotate
createmode) - Parse syslog and JSON log formats
- Fan in from multiple files with goroutines
- Filter by severity and output JSON Lines
- Graceful shutdown with signal handling
Prerequisites
- Go 1.21+ installed
- Linux (inotify is a Linux kernel feature)
- A terminal with
loggercommand available (standard on most distros)
macOS note: fsnotify uses kqueue on macOS, so the code runs there too, but the inotify-specific concepts (like
stracedemos) require Linux.Deepen your understanding in Docker Log Management: From docker logs to a Go Log Collector
Step 1: Tail a Single File with Polling
What: Read new lines from the end of a file in a loop, like tail -f.
Why: This is the most basic building block of any log aggregator: watch a file and print new lines as they appear. Every tool from tail to Fluentd starts here.
When you open a file and call Seek(0, io.SeekEnd), the read cursor jumps to the end. From that point, any new data written to the file appears after the cursor. A polling loop just checks for new data every few hundred milliseconds.
mkdir logtail && cd logtail
go mod init logtail
Create a test log directory we’ll use throughout the article:
mkdir -p /tmp/testlogs
echo '{"level":"info","msg":"service started"}' > /tmp/testlogs/app.log
main.go
package main
import (
"bufio"
"fmt"
"io"
"os"
"time"
)
func main() {
path := "/tmp/testlogs/app.log"
f, err := os.Open(path)
if err != nil {
fmt.Println("Error:", err)
os.Exit(1)
}
defer f.Close()
// Jump to end — we only want new lines
f.Seek(0, io.SeekEnd)
scanner := bufio.NewScanner(f)
for {
for scanner.Scan() {
fmt.Println(scanner.Text())
}
time.Sleep(200 * time.Millisecond)
}
}
os.Open gives us a file descriptor. Seek(0, io.SeekEnd) moves the cursor to the end of the file so we skip existing content. The inner for scanner.Scan() reads all available lines, then we sleep and try again.
Build and run:
go build -o logtail . && ./logtail
In another terminal, append lines:
echo '{"level":"error","msg":"connection refused"}' >> /tmp/testlogs/app.log
echo '{"level":"info","msg":"retrying"}' >> /tmp/testlogs/app.log
You’ll see each line printed by logtail after a brief delay. It works, but watch what happens when you check CPU usage:
top -p $(pgrep logtail)
The process wakes up every 200ms even when nothing is being written. On a busy server with 50 log files, that’s 250 wakeups per second doing nothing. We need the linux-kernel/">kernel to tell us when data arrives instead of constantly asking.
Explore this further in How to Replace Text in Multiple Files with Sed: A Step-by-Step Guide
Step 2: Watch with inotify via fsnotify
What: Replace the polling loop with Linux’s inotify subsystem, so we wake up only when the file is actually modified.
Why: inotify is a Linux kernel feature that lets a process subscribe to filesystem events. Instead of polling, the kernel sends an event the instant a file is written to. CPU usage drops to zero when logs are quiet.
You can see inotify in action with strace:
strace -e inotify_add_watch,read tail -f /tmp/testlogs/app.log
You’ll see inotify_add_watch register the file, then read blocks until an event arrives. No polling.
Install the fsnotify package (a thin wrapper around inotify that Hugo itself uses):
go get github.com/fsnotify/fsnotify
main.go
package main
import (
"bufio"
"fmt"
"io"
"os"
"github.com/fsnotify/fsnotify"
)
func main() {
path := "/tmp/testlogs/app.log"
f, err := os.Open(path)
if err != nil {
fmt.Println("Error:", err)
os.Exit(1)
}
defer f.Close()
f.Seek(0, io.SeekEnd)
scanner := bufio.NewScanner(f)
watcher, err := fsnotify.NewWatcher()
if err != nil {
fmt.Println("Error:", err)
os.Exit(1)
}
defer watcher.Close()
watcher.Add(path)
for event := range watcher.Events {
if event.Has(fsnotify.Write) {
for scanner.Scan() {
fmt.Println(scanner.Text())
}
}
}
}
fsnotify.NewWatcher() creates an inotify instance. watcher.Add(path) registers the file. The for event := range watcher.Events loop blocks until the kernel delivers an event, meaning zero CPU while waiting.
Build and run:
go build -o logtail . && ./logtail
Append lines in another terminal and they appear instantly. Check CPU with top and it’s 0.0% when idle. That’s inotify doing its job.
But there’s a problem. Simulate log rotation:
mv /tmp/testlogs/app.log /tmp/testlogs/app.log.1
echo '{"level":"error","msg":"this line is lost"}' > /tmp/testlogs/app.log
Nothing prints. The watcher is still pointing at the old file (now app.log.1). The new app.log is a completely different inode, and our watcher doesn’t know about it.
Discover related concepts in How to Replace Text in Multiple Files with Sed: A Step-by-Step Guide
Step 3: Survive Log Rotation
What: Watch the directory instead of the file, so we detect when logrotate creates a new file.
Why: logrotate is the standard Linux tool for rotating logs. In create mode (the default), it renames the current file and creates a fresh one. Any process watching the old file descriptor keeps reading from the renamed file. The new file gets missed entirely. This is why every production log aggregator watches directories.
The fix: watch /tmp/testlogs/ for Create events. When we see a new file matching our target name, re-open it.
main.go
package main
import (
"bufio"
"context"
"fmt"
"io"
"os"
"path/filepath"
"github.com/fsnotify/fsnotify"
)
func tailFile(ctx context.Context, path string, out chan<- string) {
dir := filepath.Dir(path)
base := filepath.Base(path)
watcher, err := fsnotify.NewWatcher()
if err != nil {
fmt.Println("Error:", err)
return
}
defer watcher.Close()
watcher.Add(dir) // Watch directory, not file
f, scanner := openAndSeekEnd(path)
if f != nil {
defer f.Close()
}
for {
select {
case <-ctx.Done():
return
case event := <-watcher.Events:
if event.Has(fsnotify.Write) && filepath.Base(event.Name) == base {
for scanner.Scan() {
out <- scanner.Text()
}
}
if event.Has(fsnotify.Create) && filepath.Base(event.Name) == base {
// Log was rotated — new file appeared
if f != nil {
f.Close()
}
f, scanner = openAndSeekEnd(path)
}
case err := <-watcher.Errors:
fmt.Println("Watcher error:", err)
}
}
}
func openAndSeekEnd(path string) (*os.File, *bufio.Scanner) {
f, err := os.Open(path)
if err != nil {
return nil, bufio.NewScanner(os.Stdin)
}
f.Seek(0, io.SeekEnd)
return f, bufio.NewScanner(f)
}
func main() {
ctx := context.Background()
out := make(chan string, 100)
go tailFile(ctx, "/tmp/testlogs/app.log", out)
for line := range out {
fmt.Println(line)
}
}
Now we watch the directory. When fsnotify.Create fires for our filename, we close the old file descriptor and open the new one. The context.Context parameter and out channel will become important in Step 5 when we fan in from multiple files.
Build and test rotation:
go build -o logtail . && ./logtail
In another terminal:
echo '{"level":"info","msg":"before rotation"}' >> /tmp/testlogs/app.log
mv /tmp/testlogs/app.log /tmp/testlogs/app.log.1
echo '{"level":"error","msg":"after rotation"}' > /tmp/testlogs/app.log
echo '{"level":"info","msg":"still working"}' >> /tmp/testlogs/app.log
All three lines print. Log rotation survived.
Uncover more details in Terraform From Scratch: Provision AWS Infrastructure Step by Step
Step 4: Parse Syslog and JSON Formats
What: Parse raw log lines into a structured format, supporting both syslog (RFC 3164) and JSON.
Why: Raw log lines are strings. To filter by severity or output structured JSON Lines, we need to parse them into fields. Most Linux services write syslog format, while modern applications write JSON. A real log aggregator handles both.
Syslog RFC 3164 looks like this:
# Generate a syslog entry
logger -p user.err "database connection failed"
# Output: Feb 16 14:30:01 myhost root: database connection failed
The format is: <timestamp> <hostname> <tag>: <message>, with severity encoded in the facility/priority byte.
parser.go
package main
import (
"encoding/json"
"regexp"
"strings"
"time"
)
type ParsedLine struct {
Timestamp string `json:"timestamp"`
Host string `json:"host"`
Level string `json:"level"`
Message string `json:"message"`
Source string `json:"source"`
}
var syslogRegex = regexp.MustCompile(
`^(?P<timestamp>\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2})\s+` +
`(?P<host>\S+)\s+` +
`(?P<tag>[^:]+):\s*` +
`(?P<message>.+)$`,
)
func parseLine(raw string, source string) *ParsedLine {
// Try JSON first — cheaper than regex
if strings.HasPrefix(strings.TrimSpace(raw), "{") {
return parseJSON(raw, source)
}
return parseSyslog(raw, source)
}
func parseJSON(raw string, source string) *ParsedLine {
var fields map[string]interface{}
if err := json.Unmarshal([]byte(raw), &fields); err != nil {
return fallbackParse(raw, source)
}
p := &ParsedLine{
Timestamp: time.Now().UTC().Format(time.RFC3339),
Source: source,
Level: "info",
}
if v, ok := fields["level"].(string); ok {
p.Level = strings.ToLower(v)
}
if v, ok := fields["msg"].(string); ok {
p.Message = v
} else if v, ok := fields["message"].(string); ok {
p.Message = v
}
if v, ok := fields["host"].(string); ok {
p.Host = v
}
return p
}
func parseSyslog(raw string, source string) *ParsedLine {
matches := syslogRegex.FindStringSubmatch(raw)
if matches == nil {
return fallbackParse(raw, source)
}
return &ParsedLine{
Timestamp: matches[1],
Host: matches[2],
Level: "info", // syslog priority requires the PRI header to determine level
Message: matches[4],
Source: source,
}
}
func fallbackParse(raw string, source string) *ParsedLine {
return &ParsedLine{
Timestamp: time.Now().UTC().Format(time.RFC3339),
Level: "info",
Message: raw,
Source: source,
}
}
strings.HasPrefix sniffs the format. If the line starts with {, try JSON first. This is fast because JSON parsing only happens when the prefix matches. The syslog regex uses named groups for readability.
Notice the fallbackParse function. Without it, a malformed line like not a valid log would cause parseSyslog to return nil, and any caller doing parsed.Level would panic with a nil pointer dereference. The fallback ensures every line produces a valid ParsedLine. Malformed lines just get the raw text as the message.
Test the parser by updating main() temporarily:
func main() {
ctx := context.Background()
out := make(chan string, 100)
go tailFile(ctx, "/tmp/testlogs/app.log", out)
for line := range out {
parsed := parseLine(line, "/tmp/testlogs/app.log")
fmt.Printf("[%s] %s: %s\n", parsed.Level, parsed.Source, parsed.Message)
}
}
go build -o logtail . && ./logtail
In another terminal:
echo '{"level":"error","msg":"disk full"}' >> /tmp/testlogs/app.log
echo 'Feb 16 14:30:01 myhost sshd: Failed password for root' >> /tmp/testlogs/app.log
echo 'some garbage line' >> /tmp/testlogs/app.log
All three lines parse without crashing. JSON, syslog, and the garbage line all produce output.
Journey deeper into this topic with Sed for JSON Manipulation: Parsing Without jq in 5 Simple Patterns
Step 5: Fan-in from Multiple Files
What: Tail multiple log files concurrently and merge their output into a single stream.
Why: A real server has dozens of log files: /var/log/syslog, /var/log/auth.log, /var/log/nginx/error.log, application logs. A log aggregator watches all of them simultaneously. In Go, the fan-in pattern uses one goroutine per file, all sending to a shared channel.
Create a second test log:
echo '{"level":"warn","msg":"high latency"}' > /tmp/testlogs/access.log
Update main.go:
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"sync"
)
func main() {
files := []string{
"/tmp/testlogs/app.log",
"/tmp/testlogs/access.log",
}
ctx := context.Background()
out := make(chan string, 100)
var wg sync.WaitGroup
for _, path := range files {
wg.Add(1)
go func(p string) {
defer wg.Done()
tailFile(ctx, p, out)
}(path)
}
// Close the channel when all tailers finish
go func() {
wg.Wait()
close(out)
}()
for line := range out {
parsed := parseLine(line, "")
b, _ := json.Marshal(parsed)
fmt.Println(string(b))
}
}
Each file gets its own goroutine running tailFile. All goroutines send to the same out channel. That’s the fan-in. sync.WaitGroup tracks how many goroutines are running.
The goroutine that calls wg.Wait() followed by close(out) is critical. Without it, main() would exit immediately after spawning the goroutines, and they’d be orphaned. The for line := range out loop blocks main() until the channel is closed, and the channel closes only when all tailers finish.
Try removing the wg.Wait() / close(out) goroutine and the range out loop, replacing them with just select {}. Then run the program and hit Ctrl+C. You’ll see the goroutines were running, but there was no clean coordination. We’ll fix this properly in Step 7 with signal handling.
Build and test:
go build -o logtail . && ./logtail
In another terminal:
echo '{"level":"error","msg":"app crashed"}' >> /tmp/testlogs/app.log
echo '{"level":"warn","msg":"slow query"}' >> /tmp/testlogs/access.log
Both lines appear in the output stream as JSON Lines.
Enrich your learning with How to Replace Text in Multiple Files with Sed: A Step-by-Step Guide
Step 6: Filter by Severity and Output JSON Lines
What: Add a --level flag that filters out log lines below a given severity threshold, and write output through a buffered writer.
Why: When an incident happens, you don’t want to wade through thousands of info lines. Severity filtering (like *.err in rsyslog.conf) lets you focus on what matters. Buffered I/O reduces system calls. Instead of one write() per line, the buffer batches them.
You can see the difference with strace:
# Unbuffered: one write() syscall per line
strace -c echo "line" > /dev/null
# Buffered: far fewer write() calls for the same data
strace -c cat /var/log/syslog > /dev/null
filter.go
package main
var severityRank = map[string]int{
"emergency": 0,
"alert": 1,
"critical": 2,
"error": 3,
"warn": 4,
"warning": 4,
"notice": 5,
"info": 6,
"debug": 7,
}
func meetsLevel(lineLevel string, threshold string) bool {
lineRank, ok := severityRank[lineLevel]
if !ok {
return true // Unknown levels always pass
}
threshRank, ok := severityRank[threshold]
if !ok {
return true
}
return lineRank <= threshRank
}
Syslog severity levels range from 0 (emergency) to 7 (debug). Lower numbers are more severe. meetsLevel returns true if the line’s severity is at or above the threshold.
Update main.go with flag parsing and buffered output:
package main
import (
"bufio"
"context"
"encoding/json"
"flag"
"fmt"
"os"
"sync"
)
func main() {
level := flag.String("level", "debug", "Minimum severity level (debug, info, warn, error, critical)")
flag.Parse()
files := flag.Args()
if len(files) == 0 {
fmt.Fprintln(os.Stderr, "Usage: logtail [--level LEVEL] FILE [FILE...]")
os.Exit(1)
}
ctx := context.Background()
out := make(chan string, 100)
var wg sync.WaitGroup
for _, path := range files {
wg.Add(1)
go func(p string) {
defer wg.Done()
tailFile(ctx, p, out)
}(path)
}
go func() {
wg.Wait()
close(out)
}()
bw := bufio.NewWriterSize(os.Stdout, 32*1024)
for line := range out {
parsed := parseLine(line, "")
if !meetsLevel(parsed.Level, *level) {
continue
}
b, _ := json.Marshal(parsed)
fmt.Fprintln(bw, string(b))
}
}
Now the tool accepts file paths as arguments and a --level flag. The bufio.NewWriterSize creates a 32KB buffer. Lines are written to the buffer, and the buffer flushes to stdout when it fills up or when… wait.
Build and run:
go build -o logtail . && ./logtail --level error /tmp/testlogs/app.log /tmp/testlogs/access.log
In another terminal:
echo '{"level":"info","msg":"routine check"}' >> /tmp/testlogs/app.log
echo '{"level":"error","msg":"disk full"}' >> /tmp/testlogs/app.log
echo '{"level":"warn","msg":"high memory"}' >> /tmp/testlogs/access.log
Only the error line appears. Info and warn are filtered out. But try killing the process with Ctrl+C right after sending a few error lines. Some lines might be missing from the output. The buffer hasn’t been flushed. Without defer bw.Flush(), the last lines sitting in the 32KB buffer are silently lost when the process exits. We’ll fix this in the next step.
Gain comprehensive insights from Sed for JSON Manipulation: Parsing Without jq in 5 Simple Patterns
Step 7: Graceful Shutdown with Signal Handling
What: Catch SIGINT and SIGTERM, cancel all goroutines, and flush the output buffer before exiting.
Why: In production, a log aggregator runs as a systemd service. When you systemctl restart it, systemd sends SIGTERM. If the process doesn’t handle that signal, any buffered data is lost. For a log forwarder, that means missing log lines, exactly the lines from the moment something went wrong.
# See all signals
kill -l
# SIGINT (2) = Ctrl+C
# SIGTERM (15) = kill <pid>, systemd stop
Final main.go:
package main
import (
"bufio"
"context"
"encoding/json"
"flag"
"fmt"
"os"
"os/signal"
"sync"
"syscall"
)
func main() {
level := flag.String("level", "debug", "Minimum severity level (debug, info, warn, error, critical)")
flag.Parse()
files := flag.Args()
if len(files) == 0 {
fmt.Fprintln(os.Stderr, "Usage: logtail [--level LEVEL] FILE [FILE...]")
os.Exit(1)
}
ctx, cancel := context.WithCancel(context.Background())
// Catch SIGINT and SIGTERM
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
go func() {
sig := <-sigCh
fmt.Fprintf(os.Stderr, "\nReceived %s, shutting down...\n", sig)
cancel()
}()
out := make(chan string, 100)
var wg sync.WaitGroup
for _, path := range files {
wg.Add(1)
go func(p string) {
defer wg.Done()
tailFile(ctx, p, out)
}(path)
}
go func() {
wg.Wait()
close(out)
}()
bw := bufio.NewWriterSize(os.Stdout, 32*1024)
defer bw.Flush() // Flush remaining data on exit
for line := range out {
parsed := parseLine(line, "")
if !meetsLevel(parsed.Level, *level) {
continue
}
b, _ := json.Marshal(parsed)
fmt.Fprintln(bw, string(b))
}
}
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) registers our channel to receive these signals instead of the default behavior (immediate termination). When a signal arrives, we call cancel() on the context. That context was passed to every tailFile goroutine, and they all check ctx.Done() in their select loop and exit cleanly. Once all goroutines exit, wg.Wait() returns, the channel closes, the range out loop ends, and defer bw.Flush() writes any remaining buffered data.
Build and test:
go build -o logtail . && ./logtail --level info /tmp/testlogs/app.log /tmp/testlogs/access.log
In another terminal, send some lines then immediately kill the process:
for i in $(seq 1 5); do echo "{\"level\":\"error\",\"msg\":\"line $i\"}" >> /tmp/testlogs/app.log; done
kill -TERM $(pgrep logtail)
Expected output:
Received terminated, shutting down...
{"timestamp":"2026-02-16T14:00:00Z","host":"","level":"error","message":"line 1","source":""}
{"timestamp":"2026-02-16T14:00:00Z","host":"","level":"error","message":"line 2","source":""}
{"timestamp":"2026-02-16T14:00:00Z","host":"","level":"error","message":"line 3","source":""}
{"timestamp":"2026-02-16T14:00:00Z","host":"","level":"error","message":"line 4","source":""}
{"timestamp":"2026-02-16T14:00:00Z","host":"","level":"error","message":"line 5","source":""}
All five lines present. The defer bw.Flush() saved them. Without it, some or all would have been lost in the buffer.
Master this concept through Bulletproof Bash Scripts: Mastering Error Handling for Reliable Automation
What We Built
Starting from a time.Sleep polling loop, we incrementally built a real log aggregator:
- Polling tail: read new lines from the end of a file in a loop
- inotify watching: zero-CPU file monitoring via the Linux kernel
- Rotation survival: watch directories, detect new files, re-open seamlessly
- Format parsing: syslog RFC 3164 and JSON with automatic format detection
- Fan-in concurrency: one goroutine per file, shared output channel
- Severity filtering: filter by level with buffered output for performance
- Graceful shutdown: signal handling, context cancellation, buffer flushing
All of this in about 200 lines of Go across three files. That’s the core of what Fluentd, Logstash, and Vector do under the hood.
Delve into specifics at Deploy Jenkins on Amazon EKS: Complete Tutorial for Pods and Deployments
Cleanup
rm -rf /tmp/testlogs/ && rm ./logtail
Next Steps
This log aggregator covers the fundamentals. Production-grade tools add:
- TCP/UDP forwarding: ship JSON Lines to a remote syslog server or Elasticsearch
- Glob patterns: watch
/var/log/*.loginstead of listing files individually - Backpressure: slow down reading when the output destination can’t keep up
- Metrics: count lines processed, bytes read, errors per file
- Systemd journal: read from journald’s binary format via the sd-journal API
- Multi-line joining: merge Java stack traces into a single log entry
Check out Containers From Scratch in Go for another from-scratch project that uses the same Linux kernel features (namespaces instead of inotify), or Go + Nginx Reverse Proxy for the same graceful shutdown pattern applied to an HTTP reverse proxy.
Deepen your understanding in Docker Log Management: From docker logs to a Go Log Collector
Cheat Sheet
Quick reference for log aggregation concepts in Go.
Tail from end of file:
f, _ := os.Open(path)
f.Seek(0, io.SeekEnd)
scanner := bufio.NewScanner(f)
Watch with fsnotify (inotify):
watcher, _ := fsnotify.NewWatcher()
watcher.Add("/var/log/")
for event := range watcher.Events {
if event.Has(fsnotify.Write) { /* read new lines */ }
if event.Has(fsnotify.Create) { /* file rotated, re-open */ }
}
Fan-in from multiple files:
out := make(chan string, 100)
var wg sync.WaitGroup
for _, path := range files {
wg.Add(1)
go func(p string) {
defer wg.Done()
tailFile(ctx, p, out)
}(path)
}
go func() { wg.Wait(); close(out) }()
Severity filtering:
rank := map[string]int{"emergency": 0, "error": 3, "warn": 4, "info": 6, "debug": 7}
func meetsLevel(line, threshold string) bool {
return rank[line] <= rank[threshold]
}
Buffered JSON Lines output:
bw := bufio.NewWriterSize(os.Stdout, 32*1024)
defer bw.Flush()
b, _ := json.Marshal(parsed)
fmt.Fprintln(bw, string(b))
Graceful shutdown:
ctx, cancel := context.WithCancel(context.Background())
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
go func() { <-sigCh; cancel() }()
// All goroutines check ctx.Done() in their select loop
Key concepts to remember:
Deepen your understanding in Docker Log Management: From docker logs to a Go Log Collector
- inotify wakes your process only when a file changes, so CPU usage is zero when logs are quiet
- Log rotation breaks file watchers because the inode changes. Watch directories, not files
- Fan-in in Go means multiple goroutines sending to one channel. The receiver doesn’t know or care how many senders exist
sync.WaitGrouptracks goroutine lifecycle:Addbefore launching,Donewhen finished,Waitto block until all completecontext.WithCancelpropagates shutdown to all goroutines. Onecancel()call stops everything- Buffered writers reduce syscalls but require explicit
Flush().defer bw.Flush()prevents data loss signal.Notifyintercepts OS signals. Without it, SIGTERM kills immediately and buffers are lost- Syslog severity goes from 0 (emergency) to 7 (debug). Lower numbers are more severe
References and Further Reading
- Kerrisk, M. (2010). The Linux Programming Interface. Chapter 19 covers inotify in depth
- fsnotify documentation, the Go inotify wrapper used in this article
- RFC 3164: The BSD Syslog Protocol, the syslog format we parse in Step 4
- Donovan, A. & Kernighan, B. (2015). The Go Programming Language. Chapters 8-9 cover goroutines and channels
What would you add to this log aggregator next: TCP forwarding, glob patterns, or multi-line joining?
Similar Articles
Related Content
More from devops
Learn nginx log analysis step by step — start with grep and awk one-liners for quick answers, then …
Learn AWS automation step by step. Start with AWS CLI commands for S3, EC2, and IAM, then build the …
You Might Also Like
Learn config templating step by step: start with envsubst for simple variable substitution, then …
Contents
- What We’re Building
- Prerequisites
- Step 1: Tail a Single File with Polling
- Step 2: Watch with inotify via fsnotify
- Step 3: Survive Log Rotation
- Step 4: Parse Syslog and JSON Formats
- Step 5: Fan-in from Multiple Files
- Step 6: Filter by Severity and Output JSON Lines
- Step 7: Graceful Shutdown with Signal Handling
- What We Built
- Cleanup
- Next Steps
- Cheat Sheet
- References and Further Reading
