Filename Extraction: basename to a Production File Pipeline Filename Extraction: basename to a Production File Pipeline

A high-volume log pipeline is where filename handling quietly breaks. Picture services all writing logs to /var/log/<service>/<timestamp>-<hostname>.log, with hundreds of thousands of files landing every day, and a job that has to extract just the service name from each path — fast enough to keep up with the incoming files.

High-volume file processing teaches filename extraction through edge cases: filenames with spaces, symlinks, performance bottlenecks, multiple extensions. This article walks through the same progression: start simple, hit the bugs, fix them, and end with benchmarks showing which approach actually performs at scale.

Step 1: The Simple Approach (basename)

The requirement: given /var/log/auth-api/2024-03-15-prod01.log, extract auth-api.

The obvious approach is basename:

path="/var/log/auth-api/2024-03-15-prod01.log"
service=$(basename $(dirname "$path"))
echo "$service"

Output:

auth-api

This works. dirname gives /var/log/auth-api, then basename extracts auth-api. Simple, readable, done.

But with a dozen services writing logs, you need it to work for all of them. Test across the set:

for path in /var/log/*/2024-03-15-*.log; do
    service=$(basename $(dirname "$path"))
    echo "$service"
done

Output:

auth-api
payment-svc
user-svc
notification-svc

This works in dev. It works in staging. It can still break in production.

Step 2: The Symlink Deploy Bug

A common failure report looks like this: “Service name extraction broken for auth-api on prod03.”

Check the log paths on the affected host:

ls -la /var/log/ | grep auth

Output:

lrwxrwxrwx  1 root root   28 Mar 15 10:30 auth-api -> /mnt/shared-logs/auth-api

Symlink. The log directory is a symlink to shared NFS storage. On most servers, /var/log/auth-api is a real directory. On some it is a symlink — and that is enough to break extraction inconsistently across a fleet.

Try the extraction on a symlink:

path="/var/log/auth-api/2024-03-15-prod03.log"
service=$(basename $(dirname "$path"))
echo "$service"

Output:

auth-api

That worked. Try a more realistic test:

# Create a symlink test
mkdir -p /tmp/real-logs/service-a
ln -s /tmp/real-logs/service-a /tmp/logs-link
path="/tmp/logs-link/2024-03-15.log"

service=$(basename $(dirname "$path"))
echo "$service"

Output:

logs-link

There’s the bug. basename gives the symlink name, not the target. In production, some paths look like:

/var/log/auth-api -> /mnt/shared/logs/auth-api

The dirname gave /var/log/auth-api, basename gave auth-api. But for resolved paths:

/mnt/shared/logs/auth-api/2024-03-15.log

The dirname gave /mnt/shared/logs/auth-api, basename gave auth-api. Same result.

But the actual problem is more subtle. Suppose the log path on the affected host isn’t /var/log/auth-api/file.log but the resolved real path: /mnt/shared/logs/auth-api/file.log. That happens when something normalizes paths with readlink -f before processing:

path=$(readlink -f "/var/log/auth-api/2024-03-15.log")
# path is now /mnt/shared/logs/auth-api/2024-03-15.log

service=$(basename $(dirname "$path"))
# service is "auth-api" — correct!

That case still resolves correctly. So trace what actually breaks. Consider a monitoring script doing:

for log_file in /var/log/*/2024-03-15-*.log; do
    # log_file here is the symlink path: /var/log/auth-api/2024-03-15-prod03.log
    service=$(basename $(dirname "$log_file"))
    echo "$service"
done

That should still work because log_file contains the literal path. The breakage shows up in a different shape of the monitoring code:

find /var/log -name "*.log" | while read -r path; do
    service=$(basename $(dirname "$path"))
    echo "$service"
done

And here’s the problem. On an affected host:

find /var/log -name "*.log" -print

Output (truncated):

/var/log/syslog
/var/log/auth-api/2024-03-15-prod03.log

But with -L (follow symlinks), which can be set globally via alias find='find -L' on the affected servers:

find -L /var/log -name "*.log" -print

Output:

/mnt/shared/logs/auth-api/2024-03-15-prod03.log

The symlink got resolved by find -L, the path became /mnt/shared/logs/..., and the extraction logic expected /var/log/....

Step 3: The Fix — Use Basename on the Parent Directory Name

The real issue: you need the directory name directly under /var/log, not the resolved path. The fix:

path="/mnt/shared/logs/auth-api/2024-03-15.log"

# Extract the service name from the original symlink location
# by not using readlink and not using find -L
service=$(basename $(dirname "$path"))

But if the paths are already resolved because of a global alias, that isn’t enough. One option is to remove alias find='find -L' from the server configs — but that risks breaking other scripts. A safer option is to change the extraction logic to look for the pattern directly:

# Match /var/log/<service>/ or /mnt/shared/logs/<service>/
service=$(echo "$path" | sed 's|.*/log[s]*/\([^/]*\)/.*|\1|')
echo "$service"

Test:

echo "/var/log/auth-api/2024-03-15.log" | sed 's|.*/log[s]*/\([^/]*\)/.*|\1|'
echo "/mnt/shared/logs/auth-api/2024-03-15.log" | sed 's|.*/log[s]*/\([^/]*\)/.*|\1|'

Output:

auth-api
auth-api

Fixed. Both paths now extract correctly.

Step 4: Handling Files with Spaces

This sed fix works — until a service appears with a space in the directory name, e.g. reporting service.

Its logs go to /var/log/reporting service/2024-04-01.log.

The extraction breaks:

path="/var/log/reporting service/2024-04-01.log"
service=$(echo "$path" | sed 's|.*/log[s]*/\([^/]*\)/.*|\1|')
echo "$service"

Output:

reporting service

That part works. But a downstream script that consumes the value can still break:

service="reporting service"
mkdir "/backup/$service"

Error:

mkdir: cannot create directory '/backup/reporting': No such file or directory
mkdir: cannot create directory 'service': No such file or directory

No quotes around $service would split the space into two arguments — a classic bash mistake. But here the quotes are already present, so the real problem is elsewhere. The extraction is piped into xargs:

echo "/var/log/reporting service/2024-04-01.log" | \
    sed 's|.*/log[s]*/\([^/]*\)/.*|\1|' | \
    xargs -I {} mkdir "/backup/{}"

Output:

mkdir: cannot create directory '/backup/reporting': No such file or directory

xargs split on whitespace. The {} got replaced with reporting only. The solution:

echo "/var/log/reporting service/2024-04-01.log" | \
    sed 's|.*/log[s]*/\([^/]*\)/.*|\1|' | \
    xargs -d '\n' -I {} mkdir "/backup/{}"

-d '\n' tells xargs to split on newlines only, not whitespace. Now:

ls /backup/

Output:

reporting service/

Fixed. But this exposed another question: is there a faster way than spawning sed for every path?

Step 5: Performance — basename vs Parameter Expansion vs awk

At hundreds of thousands of files per day — call it several files per second, around the clock — the per-file cost adds up. Not huge, but enough that spawning a process for each filename matters.

Four approaches:

1. basename (spawn process)
2. Parameter expansion (pure bash)
3. awk (spawn process)
4. sed (already tested)

Benchmark them. The numbers below come from a rough benchmark over ~10k iterations on my laptop — they are illustrative and will vary by machine, not production telemetry. What matters is the relative ordering, which is consistent: pure-bash parameter expansion avoids the per-call process spawn, so it comes out roughly an order of magnitude ahead of the tools that fork a process per path.

Test Setup

#!/bin/bash

# Generate 10,000 sample paths
paths=()
for i in $(seq 1 10000); do
    paths+=("/var/log/service-$((i % 50))/2024-03-15-host$i.log")
done

echo "Testing 10,000 paths..."

Approach 1: basename + dirname

start=$(date +%s%N)
for path in "${paths[@]}"; do
    service=$(basename $(dirname "$path"))
done
end=$(date +%s%N)
elapsed=$(( (end - start) / 1000000 ))
echo "basename: ${elapsed}ms"

Output:

basename: 4523ms

Approach 2: Parameter Expansion

start=$(date +%s%N)
for path in "${paths[@]}"; do
    service="${path%/*}"     # Remove everything after last /
    service="${service##*/}" # Remove everything before last /
done
end=$(date +%s%N)
elapsed=$(( (end - start) / 1000000 ))
echo "parameter expansion: ${elapsed}ms"

Output:

parameter expansion: 187ms

24x faster. No process spawning.

Approach 3: awk

start=$(date +%s%N)
for path in "${paths[@]}"; do
    service=$(echo "$path" | awk -F'/' '{print $(NF-1)}')
done
end=$(date +%s%N)
elapsed=$(( (end - start) / 1000000 ))
echo "awk: ${elapsed}ms"

Output:

awk: 5201ms

Slower than basename.

Approach 4: sed

start=$(date +%s%N)
for path in "${paths[@]}"; do
    service=$(echo "$path" | sed 's|.*/\([^/]*\)/[^/]*$|\1|')
done
end=$(date +%s%N)
elapsed=$(( (end - start) / 1000000 ))
echo "sed: ${elapsed}ms"

Output:

sed: 5104ms

Similar to awk.

Results Summary

Approximate figures from one such run (10K paths, single laptop run — representative, not exact):

Winner: parameter expansion. Pure bash, no process spawning, handles spaces correctly when quoted.

The final production code:

for log_file in /var/log/*/202*.log; do
    dir="${log_file%/*}"      # /var/log/auth-api/2024-03-15.log -> /var/log/auth-api
    service="${dir##*/}"      # /var/log/auth-api -> auth-api

    # Process with proper quoting
    mkdir -p "/backup/$service"
    cp "$log_file" "/backup/$service/"
done

This handles a high-volume stream without issues. Parameter expansion deals with spaces, symlinks (as long as you don’t resolve them), and runs 24x faster than basename.

Step 6: Removing Extensions (The Next Requirement)

After filename extraction was stable, the next requirement: strip file extensions.

Input: 2024-03-15-prod01.log Output: 2024-03-15-prod01

Approach 1: basename with Suffix

filename="2024-03-15-prod01.log"
base=$(basename "$filename" .log)
echo "$base"

Output:

2024-03-15-prod01

Works. But what if the extension varies? Some files were .log, others .log.gz.

basename "2024-03-15-prod01.log.gz" .log.gz  # Works
basename "2024-03-15-prod01.log.gz" .gz      # Gives "2024-03-15-prod01.log"

You need to know the exact extension.

Approach 2: Parameter Expansion

filename="2024-03-15-prod01.log"
base="${filename%.*}"
echo "$base"

Output:

2024-03-15-prod01

Removes everything after the last .. Handles any extension:

filename="2024-03-15-prod01.log.gz"
base="${filename%.*}"
echo "$base"

Output:

2024-03-15-prod01.log

Removes only .gz. To remove all extensions:

filename="2024-03-15-prod01.log.gz"
base="${filename%%.*}"
echo "$base"

Output:

2024-03-15-prod01

%% removes the longest match, so it strips everything after the first .. But this breaks filenames with dots:

filename="service.v2.log"
base="${filename%%.*}"
echo "$base"

Output:

service

Lost v2. The correct pattern depends on your naming convention.

For logs that always end with .log or .log.gz, the correct pattern:

filename="${filename%.log.gz}"
filename="${filename%.log}"
echo "$filename"

Apply both removals in sequence. If .log.gz exists, remove it. Otherwise remove .log.

The Bug: Order Matters

Try this:

filename="2024-03-15.log"
filename="${filename%.log}"
filename="${filename%.log.gz}"
echo "$filename"

Output:

2024-03-15

Works. Now reverse the order:

filename="2024-03-15.log.gz"
filename="${filename%.log.gz}"
filename="${filename%.log}"
echo "$filename"

Output:

2024-03-15

Still works. Both orders work because:

• .log.gz file: first pattern removes .log.gz, second pattern finds no .log, does nothing
• .log file: first pattern finds no .log.gz, does nothing, second pattern removes .log

Good. Final version:

# Extract service name and remove extensions
path="/var/log/auth-api/2024-03-15-prod01.log.gz"
dir="${path%/*}"
service="${dir##*/}"
filename="${path##*/}"
filename="${filename%.log.gz}"
filename="${filename%.log}"

echo "Service: $service"
echo "Base filename: $filename"

Output:

Service: auth-api
Base filename: 2024-03-15-prod01

What We Built: Production Log Pipeline

Starting from a simple requirement (extract service names from log paths), we hit every real-world edge case:

1. Simple basename — worked in dev, failed in prod due to symlinks
2. Symlink bug — find -L resolved paths, breaking extraction assumptions
3. Sed fix — pattern matching worked regardless of symlink resolution
4. Spaces in filenames — broke xargs, fixed with -d '\n'
5. Performance — in a rough laptop benchmark, parameter expansion came out roughly 24x faster than basename (about 187ms vs 4523ms for 10K files)
6. Extension removal — multiple approaches, parameter expansion won again

The final production pipeline:

#!/bin/bash

# Process all logs, extract service name, archive by service
for log_file in /var/log/*/202*.log*; do
    [ -f "$log_file" ] || continue

    # Extract service name (pure bash, fast)
    dir="${log_file%/*}"
    service="${dir##*/}"

    # Extract and clean filename
    filename="${log_file##*/}"
    filename="${filename%.log.gz}"
    filename="${filename%.log}"

    # Archive (with proper quoting for spaces)
    mkdir -p "/backup/$service"
    cp "$log_file" "/backup/$service/"
done

This runs around the clock at high volume with zero issues once the fixes are in place.

Cheat Sheet

Extract directory from path:

dir="${path%/*}"              # /var/log/auth-api/file.log → /var/log/auth-api

Extract filename from path:

filename="${path##*/}"        # /var/log/auth-api/file.log → file.log

Extract parent directory name:

dir="${path%/*}"             # Get directory
service="${dir##*/}"          # Get last component
# /var/log/auth-api/file.log → auth-api

Remove file extension:

base="${filename%.*}"         # file.log → file (last extension)
base="${filename%%.*}"        # file.tar.gz → file (all extensions)

Remove specific extension:

filename="${filename%.log.gz}"  # Try .log.gz first
filename="${filename%.log}"     # Then try .log

Process spawning comparison (10K files, rough laptop benchmark — illustrative):

• Parameter expansion: ~187ms
• basename: ~4523ms (~24x slower)
• sed: ~5104ms (~27x slower)
• awk: ~5201ms (~28x slower)

Key Rules

1. Use parameter expansion for performance — 24x faster than spawning processes
2. Quote everything — "$variable" prevents word splitting on spaces
3. Use xargs -d '\n' when piping filenames with spaces
4. Don’t use find -L unless you actually need symlink resolution
5. % removes from end, # removes from beginning — %% and ## are longest match
6. Test with edge cases — spaces, symlinks, dots in names, multiple extensions

FAQ

Q: When should I use basename vs parameter expansion? A: Use parameter expansion (${var##*/}) for scripts that process many files. It’s 24x faster. Use basename for interactive one-liners where readability matters more than performance.

Q: How do I handle filenames with spaces? A: Always quote variables: "$filename". When using xargs, add -d '\n' to split on newlines instead of whitespace.

Q: What’s the difference between % and %%? A: Both strip from the end, but they differ by how much of a glob pattern they match: % removes the shortest trailing match, %% removes the longest. For a literal suffix like .log (no wildcards) there’s nothing to match short vs long, so ${var%.log} and ${var%%.log} behave identically. The difference only shows up with a pattern: on file.tar.gz, ${var%.*} removes the shortest match (.gz, giving file.tar), while ${var%%.*} removes the longest (.tar.gz, giving file). For trimming a single extension, you usually want %.* (shortest).

Q: Why did symlinks break my script? A: find -L and readlink -f resolve symlinks to real paths. If your extraction logic expects the symlink path, disable symlink resolution or adjust the pattern to work with both.

Q: Which is faster: awk or sed? A: For filename extraction, they’re roughly the same speed (both slow compared to parameter expansion). Both spawn processes. In a rough 10K-file benchmark on my laptop, awk took ~5201ms, sed ~5104ms, and parameter expansion ~187ms — illustrative figures, but the ordering holds.

Keep Reading

• Mastering Bash: The Ultimate Guide to Command Line Productivity — more bash patterns and productivity techniques
• Linux Automation: From Cron to a Go Task Runner — apply these filename patterns in automation pipelines
• Sed Cheat Sheet: 30 Essential One-Liners — when you need more power than parameter expansion