“I need to update the API endpoint in all our configuration files.” This seemingly simple task can become a significant challenge when dealing with dozens or hundreds of files across multiple directories. Manual editing is tedious and error-prone, while specialized tools might not be available in all environments.

Enter sed, the powerful stream editor available on virtually every Unix-like system. While many developers know sed can replace text in a single file, its real power emerges when applied to multiple files systematically.

After years of DevOps and system administration work, I’ve refined a set of reliable patterns for multi-file text replacement using sed. In this comprehensive guide, I’ll share these techniques, from basic replacements to complex pattern matching across large codebases.

Understanding the Fundamentals

Before diving into multi-file operations, let’s ensure we understand how sed handles single-file replacements. As Robert Love explains in “Linux System Programming”, sed processes text line by line, applying specified operations to each line in sequence.

The basic syntax for text replacement is:

sed 's/old_text/new_text/' file.txt

However, this only prints the modified content to standard output without changing the file. For in-place editing, we need the -i flag:

sed -i 's/old_text/new_text/' file.txt

According to Brian Ward’s “How Linux Works”, the -i option is one of sed’s most powerful features for system administration tasks, as it enables direct modification of configuration files.

Method 1: Using Shell Globbing for Multiple Files

The simplest approach to multi-file operations uses shell globbing patterns to select files.

Basic Globbing Example

# Replace "old_api.example.com" with "new_api.example.com" in all .json files
sed -i 's/old_api\.example\.com/new_api.example.com/g' *.json

This command:

1. Selects all files with .json extension in the current directory
2. Replaces all occurrences of old_api.example.com with new_api.example.com
3. The g flag ensures all occurrences in each line are replaced, not just the first

Using Globbing with Directories

For nested directories, you can use extended globbing features if your shell supports them:

# In Bash with globstar option enabled
shopt -s globstar
sed -i 's/old_api\.example\.com/new_api.example.com/g' **/*.json

This matches .json files in the current directory and all subdirectories.

Backup Before Replacing

“https://www.oreilly.com/library/view/unix-power-tools/0596003307/">Unix Power Tools” emphasizes the importance of backups before bulk operations. The -i flag can create backups by appending a suffix:

# Create .bak backups before making changes
sed -i.bak 's/old_api\.example\.com/new_api.example.com/g' *.json

This generates a backup file (e.g., config.json.bak) for each modified file.

Method 2: Using find with sed for Precise Control

Shell globbing works for simple cases, but for more complex file selection, combining find with sed provides greater flexibility and power.

Basic find and sed Example

# Replace text in all .js files modified in the last 7 days
find . -name "*.js" -mtime -7 -exec sed -i 's/const API_URL = ".*"/const API_URL = "https:\/\/new-api.example.com"/g' {} \;

This pattern:

1. Uses find to locate .js files modified within the last 7 days
2. Executes sed on each matching file to update the API URL

Handling Spaces in Filenames

File paths with spaces can cause issues with simple approaches. Here’s a robust solution:

# Safely handle filenames with spaces
find . -name "*.config" -print0 | xargs -0 sed -i 's/debug=false/debug=true/g'

The -print0 and -0 flags ensure filenames are separated by null characters, safely handling spaces and special characters.

Filtering with grep Before Replacing

Sometimes you only want to modify files containing specific content. Combine find, grep, and sed:

# Only replace in files that contain the old text
find . -name "*.yaml" -type f -exec grep -l "oldValue" {} \; | xargs sed -i 's/oldValue/newValue/g'

This approach:

1. Uses find to locate all .yaml files
2. Filters with grep -l to get only files containing “oldValue”
3. Passes those filenames to sed for replacement

According to “Shell Scripting” guides, this pre-filtering significantly improves performance for large file sets by avoiding unnecessary processing.

Method 3: Using sed with Process Substitution

For more complex scenarios, you can combine sed with process substitution:

# Replace in all files listed in a manifest
sed -i 's/VERSION=.*/VERSION="2.0.1"/' $(cat files_to_update.txt)

This reads filenames from files_to_update.txt and applies the replacement to each.

You can also dynamically generate the file list:

# Replace in all non-binary files containing a specific string
sed -i 's/api\.v1\./api.v2./g' $(grep -l -r --include="*.php" "api.v1." .)

This searches recursively for .php files containing “api.v1.” and replaces the text in all matches.

Advanced Sed Techniques for Multi-File Operations

With the basics covered, let’s explore more powerful techniques for complex scenarios.

Regular Expression Replacements

For pattern-based replacements, use sed’s regular expression capabilities:

# Replace all phone numbers with a specific format
find . -name "*.html" -exec sed -i -E 's/\(([0-9]{3})\) ([0-9]{3})-([0-9]{4})/\1-\2-\3/g' {} \;

This converts phone numbers from (123) 456-7890 format to 123-456-7890 format.

The -E flag enables extended regular expressions, making the pattern more readable.

Replacing Across Line Boundaries

Sometimes the text you need to replace spans multiple lines. Here’s how to handle that:

# Replace a multi-line XML tag
find . -name "*.xml" -exec sed -i '/<settings>/,/<\/settings>/{
s/<debug>false<\/debug>/<debug>true<\/debug>/g
}' {} \;

This only applies the replacement within the <settings> tags.

Conditional Replacements

For more targeted changes, use conditional logic:

# Replace only in production configuration files
find . -name "*.conf" -exec sed -i '/\[production\]/,/\[/{
s/debug=true/debug=false/g
}' {} \;

This only changes debug=true to debug=false within the [production] section of configuration files.

Real-World Examples

Let’s apply these techniques to common scenarios developers and system administrators face.

Example 1: Updating Dependencies in a Project

#!/bin/bash
# Update a dependency version across a project

OLD_VERSION="1.2.3"
NEW_VERSION="2.0.0"
DEPENDENCY_NAME="example-lib"

echo "Updating $DEPENDENCY_NAME from v$OLD_VERSION to v$NEW_VERSION..."

# Create a timestamped backup directory
BACKUP_DIR="backup_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Update package.json files
find . -name "package.json" -exec grep -l "\"$DEPENDENCY_NAME\"" {} \; | while read file; do
    # Create backup
    cp "$file" "$BACKUP_DIR/$(basename "$file").$(date +%s)"
    
    # Update the dependency version
    sed -i "s/\"$DEPENDENCY_NAME\": \".*$OLD_VERSION.*\"/\"$DEPENDENCY_NAME\": \"^$NEW_VERSION\"/g" "$file"
    
    echo "Updated $file"
done

# Update import statements in code
find . -name "*.js" -o -name "*.ts" | xargs grep -l "$DEPENDENCY_NAME" | while read file; do
    # Create backup
    cp "$file" "$BACKUP_DIR/$(basename "$file").$(date +%s)"
    
    # Update require statements
    sed -i "s/require(['\"]$DEPENDENCY_NAME\/v$OLD_VERSION['\"])/require('$DEPENDENCY_NAME\/v$NEW_VERSION')/g" "$file"
    
    # Update import statements
    sed -i "s/from ['\"]$DEPENDENCY_NAME\/v$OLD_VERSION['\"])/from '$DEPENDENCY_NAME\/v$NEW_VERSION'/g" "$file"
    
    echo "Updated $file"
done

echo "Update complete. Backups saved to $BACKUP_DIR/"

This script:

1. Creates a timestamped backup directory
2. Updates version references in package.json files
3. Updates import/require statements in JavaScript and TypeScript files
4. Provides feedback on each updated file

Example 2: Changing Configuration Across Environments

#!/bin/bash
# Update database connection strings across multiple environments

OLD_DB_HOST="db-old.internal"
NEW_DB_HOST="db-new.internal"

# Files to check (using find for flexibility)
CONFIG_FILES=$(find ./config -type f -name "*.properties" -o -name "*.xml" -o -name "*.yaml")

# Check if any files were found
if [ -z "$CONFIG_FILES" ]; then
    echo "No configuration files found!"
    exit 1
fi

echo "The following files will be modified:"
echo "$CONFIG_FILES"
echo

# Prompt for confirmation
read -p "Do you want to proceed? (y/n): " confirm
if [[ "$confirm" != "y" && "$confirm" != "Y" ]]; then
    echo "Operation cancelled."
    exit 0
fi

# Create backup directory
BACKUP_DIR="./config_backup_$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"
echo "Creating backups in $BACKUP_DIR"

# Process each file
echo "$CONFIG_FILES" | while read -r file; do
    # Skip if file doesn't exist (might have been moved/deleted)
    [ ! -f "$file" ] && continue
    
    # Create backup
    cp "$file" "$BACKUP_DIR/$(basename "$file").bak"
    
    # Check file type and use appropriate replacement strategy
    case "$file" in
        *.properties)
            # Handle Java properties files
            sed -i "s/$OLD_DB_HOST/$NEW_DB_HOST/g" "$file"
            ;;
        *.xml)
            # Handle XML files, being careful with escaping
            sed -i "s|<url>jdbc:.*$OLD_DB_HOST|<url>jdbc:mysql://$NEW_DB_HOST|g" "$file"
            sed -i "s|<host>$OLD_DB_HOST</host>|<host>$NEW_DB_HOST</host>|g" "$file"
            ;;
        *.yaml|*.yml)
            # Handle YAML files
            sed -i "s/host: $OLD_DB_HOST/host: $NEW_DB_HOST/g" "$file"
            ;;
    esac
    
    echo "Updated $file"
done

echo "Operation completed successfully."

This script:

1. Finds configuration files of different types
2. Asks for confirmation before proceeding
3. Creates a dated backup of all files
4. Uses different sed patterns based on file type
5. Provides feedback for each modified file

Example 3: Code Refactoring Across a Codebase

#!/bin/bash
# Refactor deprecated API calls across a codebase

OLD_API_PATTERN="getUserData\([^)]*\)"
NEW_API_PATTERN="fetchUserInfo\(\1\)"

# Find all JavaScript and TypeScript files
CODE_FILES=$(find ./src -type f -name "*.js" -o -name "*.ts" | grep -v "node_modules" | grep -v "dist")

echo "Scanning $(echo "$CODE_FILES" | wc -l) files for deprecated API usage..."

# First, create a report of all occurrences
echo "Files containing deprecated API calls:"
for file in $CODE_FILES; do
    count=$(grep -c "$OLD_API_PATTERN" "$file" || true)
    if [ "$count" -gt 0 ]; then
        echo "  $file: $count occurrences"
    fi
done

# Prompt for confirmation
read -p "Do you want to update these files? (y/n): " confirm
if [[ "$confirm" != "y" && "$confirm" != "Y" ]]; then
    echo "Operation cancelled."
    exit 0
fi

# Create backup directory
BACKUP_DIR="./refactor_backup_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Process each file
for file in $CODE_FILES; do
    if grep -q "$OLD_API_PATTERN" "$file"; then
        # Create backup
        mkdir -p "$BACKUP_DIR/$(dirname "$file")"
        cp "$file" "$BACKUP_DIR/$file"
        
        # Perform replacement with extended regex
        sed -i -E "s/$OLD_API_PATTERN/$NEW_API_PATTERN/g" "$file"
        
        echo "Updated $file"
    fi
done

echo "Refactoring complete. Backups saved to $BACKUP_DIR/"

This script:

1. Scans a codebase for deprecated API usage
2. Creates a report of all occurrences
3. Asks for confirmation before proceeding
4. Creates a backup of all modified files
5. Uses extended regex for complex pattern matching
6. Provides feedback on each updated file

Best Practices for Multi-File Text Replacement

Based on my experience and principles from “How Linux Works” and “Shell Scripting” guides, here are essential best practices for safe and effective multi-file operations:

1. Always Create Backups

Never perform multi-file replacements without a backup strategy:

# Create backups with the -i option
sed -i.bak 's/old/new/g' *.conf

# Or create a backup directory first
mkdir -p backups
find . -name "*.conf" -exec cp {} backups/ \; -exec sed -i 's/old/new/g' {} \;

2. Test on a Subset First

Before processing all files, test your pattern on a representative sample:

# Test on one file and display the result without changing it
sed 's/old/new/g' example.conf

# Test on a small subset
find . -name "*.conf" | head -n 2 | xargs sed -i.test 's/old/new/g'

3. Use Version Control

If your files are in a version control system, commit before making bulk changes:

git add .
git commit -m "Commit before bulk replacement"
# Perform replacements
git diff # Review the changes

4. Verify After Replacement

Always verify the results after performing replacements:

# Count occurrences before and after to verify all were replaced
before=$(grep -r "old_text" --include="*.conf" . | wc -l)
sed -i 's/old_text/new_text/g' *.conf
after=$(grep -r "old_text" --include="*.conf" . | wc -l)
echo "Replaced $((before - after)) occurrences"

5. Use Appropriate Delimiters

When your search or replacement text contains slashes, use alternative delimiters:

# Using | as delimiter for paths
sed -i 's|/var/www/old/|/var/www/new/|g' *.conf

# Using @ for URLs
sed -i 's@http://old.com@https://new.com@g' *.conf

6. Consider Line Endings

Be aware of different line endings when working across platforms:

# Convert Windows line endings to Unix before processing
find . -name "*.txt" -exec dos2unix {} \; -exec sed -i 's/old/new/g' {} \;

Troubleshooting Common Issues

Even with careful planning, you may encounter issues. Here’s how to address common problems:

Issue 1: Changes Not Being Applied

Symptoms: sed runs without errors, but files aren’t changed.

Potential causes and solutions:

1. File permissions: Ensure you have write permissions.

   # Check file permissions
   ls -la *.conf
   # Update if needed
   chmod u+w *.conf
   

2. Pattern not matching: Your regex might not match any content.

   # Verify your pattern matches something
   grep "old_pattern" *.conf
   # If not matching, adjust your pattern
   

3. Escaping issues: Special characters need proper escaping.

   # For patterns with special characters, use -E and proper escaping
   sed -i -E 's/value=\$\{old\}/value=${new}/g' *.conf
   

Issue 2: Corrupted Output Files

Symptoms: Files are modified but contain corrupted text or binary data.

Potential causes and solutions:

1. Binary files: sed doesn’t handle binary files well.

   # Exclude binary files
   find . -type f -name "*.conf" -exec grep -Iq . {} \; -print | xargs sed -i 's/old/new/g'
   

2. Encoding issues: File encoding might be affected.

   # Preserve encoding with iconv
   for file in *.conf; do
       iconv -f UTF-8 -t UTF-8 "$file" | sed 's/old/new/g' > "$file.new"
       mv "$file.new" "$file"
   done
   

Issue 3: Performance Problems with Large Files

Symptoms: Operations on large files are extremely slow or cause high system load.

Potential causes and solutions:

1. File size: Very large files can slow down processing.

   # Process large files in chunks
   split -b 100m large_file.txt chunk_
   for chunk in chunk_*; do
       sed -i 's/old/new/g' "$chunk"
   done
   cat chunk_* > large_file_new.txt
   

2. Inefficient patterns: Complex patterns can cause performance issues.

   # Pre-filter files to reduce processing
   grep -l "old" *.txt | xargs sed -i 's/old/new/g'
   

Advanced Topic: Combining sed with awk and perl

For even more powerful text processing, combine sed with other tools:

Using sed with awk

# Use awk for more complex conditional replacements
find . -name "*.csv" -exec awk '{
    if ($1 == "id" && $2 == "old") {
        gsub("old", "new", $0)
    }
    print
}' {} > {}.new \; -exec mv {}.new {} \;

Using sed with perl

# Use perl for multi-line pattern replacement
find . -name "*.html" -exec perl -i -0pe 's/<div class="old">\s*<p>(.*?)<\/p>\s*<\/div>/<div class="new">\n  <p>\1<\/p>\n<\/div>/gs' {} \;

According to “Unix Power Tools”, these combinations leverage the strengths of each tool for more sophisticated text processing.

Conclusion: Mastering Multi-File Text Replacement

Throughout this guide, we’ve explored a comprehensive set of techniques for replacing text across multiple files using sed. From basic shell globbing to complex find commands and advanced pattern matching, you now have a toolkit for handling virtually any text replacement scenario.

Remember these key principles:

1. Always create backups before performing batch operations
2. Test your patterns on a small sample before applying widely
3. Use appropriate file selection methods based on your needs
4. Consider the specific requirements of different file types
5. Verify your changes after performing replacements

With these techniques and best practices, you can confidently perform even complex text replacements across large codebases, saving countless hours of manual editing and reducing the risk of errors.

What multi-file text replacement challenges have you encountered? I’d love to hear about your experiences and additional techniques in the comments below!