Skip main navigation
/user/kayd @ :~$ cat mastering-sed-update-remove-configuration-files.md

Mastering sed for Configuration Files: The Definitive Guide to Updating YAML, JSON, TOML, and Properties Files

Karandeep Singh
Karandeep Singh
• 15 minutes

Summary

Learn how to leverage sed’s powerful text processing capabilities to efficiently update, remove, and transform data in various configuration file formats like YAML, JSON, TOML, and properties files.

“I broke production by updating the wrong config value.” These words haunted me early in my DevOps career after a botched manual edit to a configuration file. That painful lesson taught me the value of reliable, automated configuration management—and the power of sed for precise file manipulation.

The Stream EDitor (sed) is one of the most powerful yet misunderstood text processing tools in the Unix toolkit. As Robert Love notes in Linux System Programming: Talking Directly to the Kernel and C Library”, sed enables direct manipulation of text streams with surgical precision. This makes it ideal for updating configuration files in automated deployments and maintenance tasks.

In this comprehensive guide, I’ll share the sed techniques I’ve refined over years of DevOps work to safely and efficiently update various configuration file formats. You’ll learn how to remove, replace, and transform data in YAML, JSON, TOML, and properties files—skills that will save you countless hours of manual editing and prevent costly configuration errors.

Understanding sed’s Power for Configuration Management

Before diving into specific file formats, let’s understand why sed is particularly valuable for configuration management. Brian Ward’s “How Linux Works” explains that sed’s power comes from its ability to:

  1. Process text line-by-line with pattern matching
  2. Make precise, targeted changes without loading entire files into memory
  3. Work within shell scripts for automated operations
  4. Modify files in-place with proper backups
  5. Chain multiple operations for complex transformations

The core sed command structure we’ll use throughout this article is:

sed -i[suffix] 'operation/pattern/replacement/flags' filename

Where:

  • -i[suffix] enables in-place editing with optional backup
  • operation is the sed command (s for substitute, d for delete, etc.)
  • pattern is what we’re looking for (often a regular expression)
  • replacement is what we’re changing it to (for substitute operations)
  • flags modify how the operation works (g for global, I for case-insensitive, etc.)
  • filename is the target file to modify

Let’s start with some foundational examples before tackling specific file formats.

Essential sed Commands for Configuration Files

When working with configuration files, these are the most useful sed operations I’ve found:

1. Deleting Lines

To remove configuration entries entirely:

# Delete lines containing "deprecated-feature"
sed -i '/deprecated-feature/d' config.yaml

# Delete lines matching a pattern and the following line
sed -i '/temp_setting/{N;d;}' config.properties

# Delete lines between two markers (inclusive)
sed -i '/BEGIN_LEGACY_CONFIG/,/END_LEGACY_CONFIG/d' config.json

2. Replacing Values

To update configuration values:

# Replace a specific port number
sed -i 's/port: 8080/port: 9090/' config.yaml

# Replace all occurrences of a specific IP address
sed -i 's/192.168.1.1/10.0.0.1/g' config.properties

# Replace only the 2nd occurrence on each line
sed -i 's/DEBUG/INFO/2' logging.properties

3. Conditional Replacement

For more targeted changes:

# Replace value only in [production] section in TOML
sed -i '/^\[production\]/,/^\[/s/debug = true/debug = false/' config.toml

# Change only uncommented settings
sed -i 's/^[^#].*max_connections = 100/max_connections = 500/' database.conf

4. Adding or Appending Content

To inject new configuration:

# Add a line after matching a pattern
sed -i '/\[logging\]/a log_level = "INFO"' config.toml

# Insert before a specific line
sed -i '/^end_settings/i new_feature = true' config.properties

Now that we’ve covered the fundamentals, let’s dive into format-specific techniques.

Mastering YAML Manipulation with sed

YAML files present unique challenges due to their indentation-based structure. As “Unix Power Tools” points out, preserving whitespace and indentation is crucial when modifying YAML.

Removing Entries from YAML

Let’s start with a sample YAML file (config.yaml):

server:
  host: api.example.com
  port: 8080
  timeout: 30
  features:
    - authentication
    - rate-limiting
    - caching
  debug: true
  deprecated:
    old_feature: true
    legacy_auth: basic

database:
  host: db.internal
  port: 5432
  max_connections: 100

To remove a simple key-value pair:

# Remove the debug setting
sed -i '/^\s*debug: true/d' config.yaml

Deleting a multi-line YAML block requires more care:

# Remove the entire deprecated block with all its children
sed -i '/^\s*deprecated:/,/^\s*[a-z]/s/^\s*deprecated:.*$//;/^\s*old_feature\|^\s*legacy_auth/d' config.yaml

This is a complex command, so let’s break it down:

  • /^\s*deprecated:/,/^\s*[a-z]/ selects the range from the “deprecated:” line to the next line with the same indentation level
  • s/^\s*deprecated:.*$// removes the “deprecated:” line
  • /^\s*old_feature\|^\s*legacy_auth/d removes the child properties

For array items in YAML, we can use:

# Remove the "caching" feature
sed -i '/^\s*- caching/d' config.yaml

Updating YAML Values

Updating values in YAML requires careful pattern matching:

# Update the server port
sed -i 's/^\(\s*port: \)8080/\1 9090/' config.yaml

# Change the host without affecting other hosts
sed -i '/server:/,/^\s*[a-z]/{s/^\(\s*host: \)api\.example\.com/\1 new-api.example.com/}' config.yaml

Note the use of capture groups \( \) to preserve indentation. The second example uses a range selector to only change the host within the server section.

Adding YAML Configuration Lines

For adding new settings:

# Add a new feature to the list
sed -i '/^\s*features:/a\ \ \ \ - monitoring' config.yaml

# Add a new complete section at the end of the file
sed -i '$a\\nlogging:\n  level: INFO\n  file: "/var/log/app.log"' config.yaml

The second example demonstrates adding a multi-line section, preserving proper YAML indentation.

According to Brian Ward in “How Linux Works”, YAML processing with sed requires special attention to whitespace and structure. I’ve found that capturing and reusing the existing indentation pattern via regex groups is the key to maintaining valid YAML.

Tackling JSON Files with sed

JSON’s syntax with braces, quotes, and commas requires careful handling with sed. While specialized tools like jq are often preferred for JSON, there are many situations where sed is your only option, such as in minimal containers or when modifying files in-place.

Let’s work with this example JSON file (config.json):

{
  "server": {
    "host": "api.example.com",
    "port": 8080,
    "timeout": 30,
    "features": [
      "authentication",
      "rate-limiting",
      "caching"
    ],
    "debug": true,
    "deprecated": {
      "old_feature": true,
      "legacy_auth": "basic"
    }
  },
  "database": {
    "host": "db.internal",
    "port": 5432,
    "max_connections": 100
  }
}

Removing JSON Properties

To remove a simple key-value pair:

# Remove the debug setting
sed -i '/"debug": true,\?/d' config.json

The \? makes the trailing comma optional, handling both middle and end properties.

For more complex nested structures:

# Remove the entire deprecated object
sed -i '/"deprecated": {/,/}/d' config.json

# Cleanup the extra comma if needed
sed -i 's/,\s*}/}/' config.json

Removing items from JSON arrays:

# Remove "caching" from features array
sed -i '/^\s*"caching",\?/d' config.json

Updating JSON Values

For updating simple values:

# Update server port
sed -i 's/"port": 8080/"port": 9090/' config.json

# Update server host with quoted value
sed -i 's/"host": "api\.example\.com"/"host": "new-api.example.com"/' config.json

For conditional updates within specific JSON objects:

# Only update max_connections in the database object
sed -i '/"database": {/,/}/s/"max_connections": 100/"max_connections": 200/' config.json

Adding JSON Configuration

Adding new properties to JSON is trickier because of the comma placement rules:

# Add a new property to the server object (before the closing brace)
sed -i '/"server": {/,/}/{s/}/,\n    "environment": "production"\n}/}' config.json

# Add an item to the features array
sed -i '/"features": \[/a\      "monitoring",' config.json

As Robert Love notes in “Linux System Programming”, JSON’s rigid syntax makes it one of the more challenging formats to manipulate with sed. I’ve found that breaking complex modifications into multiple sed operations often leads to more reliable results.

Precise TOML Editing with sed

TOML (Tom’s Obvious Minimal Language) is becoming increasingly popular for configuration files. Its sections in brackets and clear key-value format make it amenable to sed operations.

Let’s use this sample TOML file (config.toml):

# Server configuration
[server]
host = "api.example.com"
port = 8080
timeout = 30
debug = true

# Feature flags
[server.features]
authentication = true
rate-limiting = true
caching = true

# Database settings
[database]
host = "db.internal"
port = 5432
max_connections = 100

# Deprecated features
[deprecated]
old_feature = true
legacy_auth = "basic"

Removing TOML Configuration

To remove single settings:

# Remove the debug setting
sed -i '/^debug = true/d' config.toml

# Remove a setting with quoted value
sed -i '/^host = "api\.example\.com"/d' config.toml

To remove entire sections:

# Remove the deprecated section and all its settings
sed -i '/^\[deprecated\]/,/^\[/d' config.toml

# Fix the file if the last section was removed
sed -i '${/^\s*$/d}' config.toml

The second command removes any trailing empty lines that might be left after section removal.

Updating TOML Values

For simple key-value updates:

# Update the server port
sed -i 's/^port = 8080/port = 9090/' config.toml

# Update a string value (need to handle quotes)
sed -i 's/^host = "api\.example\.com"/host = "new-api.example.com"/' config.toml

For updating values within specific sections:

# Update timeout only in server section
sed -i '/^\[server\]/,/^\[/{s/^timeout = 30/timeout = 60/}' config.toml

# Disable a feature
sed -i '/^\[server\.features\]/,/^\[/{s/^authentication = true/authentication = false/}' config.toml

Adding TOML Configuration

To add new settings:

# Add a new setting to the server section
sed -i '/^\[server\]/a environment = "production"' config.toml

# Add a completely new section
sed -i '$a\[logging]\nlevel = "INFO"\nfile = "/var/log/app.log"' config.toml

The “Shell Scripting” guide emphasizes that TOML’s section-based structure makes it particularly well-suited for sed processing. I’ve found that focusing on section boundaries with pattern ranges is the key to reliable TOML manipulation.

Manipulating Properties Files with sed

Java properties files and similar key-value configurations are perhaps the most straightforward to work with using sed. Their simple format of key=value or key: value pairs, often with comments, is exactly what sed was designed to handle.

Let’s use this sample properties file (application.properties):

# Server configuration
server.host=api.example.com
server.port=8080
server.timeout=30
server.debug=true

# Feature flags
feature.authentication=true
feature.rate-limiting=true
feature.caching=true

# Database settings
database.host=db.internal
database.port=5432
database.max_connections=100

# Deprecated features
deprecated.old_feature=true
deprecated.legacy_auth=basic

Removing Properties

To remove individual properties:

# Remove the debug setting
sed -i '/^server\.debug=true/d' application.properties

# Remove all deprecated properties
sed -i '/^deprecated\./d' application.properties

# Remove commented lines about features
sed -i '/^# Feature flags/d' application.properties

Updating Properties Values

Updating values is straightforward:

# Update server port
sed -i 's/^server\.port=8080/server.port=9090/' application.properties

# Update all true values to false in feature properties
sed -i 's/^feature\.[^=]*=true/&=false/' application.properties

# Replace all instances of an IP address
sed -i 's/192\.168\.1\.1/10.0.0.1/g' application.properties

The second example uses the & backreference, which represents the entire matched pattern.

Adding Properties Configuration

Adding new properties is simple:

# Add a new server property
sed -i '/^server\./a server.environment=production' application.properties

# Add a comment and a new section
sed -i '$a\# Logging configuration\nlogging.level=INFO\nlogging.file=/var/log/app.log' application.properties

Handling Properties with Spaces

Some properties files use spaces around the equals sign or colon. Here’s how to handle those:

# Update with spaces around equals
sed -i 's/^server\.timeout = 30/server.timeout = 60/' application.properties

# Update with colon separator
sed -i 's/^database\.host: db\.internal/database.host: new-db.internal/' application.properties

According to “Unix Power Tools”, properties files are the perfect use case for sed, as they follow the line-based text processing model that sed was designed for.

Advanced sed Techniques for Complex Configuration Updates

Now that we’ve covered format-specific examples, let’s explore some advanced sed techniques that can help with complex configuration management tasks.

Using sed with Environment Variables

Dynamically replacing values from environment variables:

# Replace database password with value from environment variable
sed -i "s/^database\.password=.*/database.password=$DB_PASSWORD/" application.properties

# Using variables for multiple replacements
sed -i "s/api\.example\.com/$API_HOST/g; s/8080/$API_PORT/g" config.yaml

Note the use of double quotes to allow variable expansion.

Making Conditional Changes with Hold Space

The hold space is a powerful sed feature for complex multi-line operations:

# Only change debug to false if in production section in TOML
sed -i '/^\[production\]/{
:a
n
/debug = true/s/true/false/
/^\[/!ba
}' config.toml

This script:

  1. Finds the [production] section
  2. Creates a label a
  3. Reads the next line
  4. If it contains debug = true, changes it to false
  5. If the line doesn’t start a new section, loops back
  6. Continues processing the rest of the file normally

Creating Backup Files Before Changes

Always create backups before critical changes:

# Create a timestamped backup
sed -i.$(date +%Y%m%d_%H%M%S) 's/max_connections = 100/max_connections = 500/' database.conf

# Keep simple backup with extension
sed -i.bak 's/debug: true/debug: false/' config.yaml

The first example creates a timestamped backup, while the second creates a simple .bak backup.

Multi-stage Updates with sed Scripts

For complex updates, sed scripts provide better readability and maintainability:

# Create a sed script file
cat > update_config.sed << 'EOF'
# Remove deprecated section
/^\[deprecated\]/,/^\[/d

# Update port in server section
/^\[server\]/,/^\[/{
    s/port = 8080/port = 9090/
    s/debug = true/debug = false/
}

# Add new logging section if not exists
$a\
\
# Logging configuration\
[logging]\
level = "INFO"\
file = "/var/log/app.log"
EOF

# Apply the script
sed -i.bak -f update_config.sed config.toml

This approach allows for better documentation and more complex logic in a single operation.

Handling Escaping and Delimiter Challenges

One of the most common issues with sed is dealing with delimiters and escaping. Here are some techniques to handle those challenges:

Alternative Delimiters

When your pattern contains slashes, use alternative delimiters:

# Using pipe as delimiter for file paths
sed -i 's|/var/log/old_app.log|/var/log/new_app.log|g' config.json

# Using colon for URLs
sed -i 's:http\://old-api\.example\.com:https://new-api.example.com:g' application.properties

Escaping Special Characters

Handling strings with special characters:

# Escaping regex metacharacters for literal matching
sed -i 's/password = "p@ssw0rd\*"/password = "new-p@ssw0rd\*"/' config.toml

# Using extended regex mode for cleaner patterns
sed -E -i 's/(api_key: )"[^"]*"/\1"NEW_API_KEY"/' config.yaml

The -E flag enables extended regex mode, which reduces the need for some escaping.

Format Validation After sed Operations

After making changes, it’s crucial to validate that the file remains syntactically correct. Here’s how to do that for each format:

# Validate YAML
function yaml_validate() {
    python3 -c 'import yaml, sys; yaml.safe_load(sys.stdin)' < "$1"
}

# Validate JSON
function json_validate() {
    python3 -c 'import json, sys; json.load(sys.stdin)' < "$1"
}

# Validate TOML
function toml_validate() {
    python3 -c 'import tomli, sys; tomli.loads(sys.stdin.read())' < "$1"
}

# Example usage in a script
sed -i 's/debug: true/debug: false/' config.yaml
if ! yaml_validate config.yaml; then
    echo "YAML validation failed, reverting changes"
    mv config.yaml.bak config.yaml
    exit 1
fi

As Brian Ward recommends in “How Linux Works”, always validate changes to critical configuration files before deploying them.

Real-world Examples: sed in DevOps Pipelines

Let’s put everything together with real-world examples from my DevOps experience:

Example 1: Updating Kubernetes ConfigMaps

#!/bin/bash
# Update ConfigMap values for different environments

ENV="${1:-dev}"
CONFIG_FILE="kubernetes/configmap.yaml"

# Create backup
cp "$CONFIG_FILE" "$CONFIG_FILE.bak"

case "$ENV" in
  dev)
    # Update development settings
    sed -i "s/log_level: .*/log_level: DEBUG/" "$CONFIG_FILE"
    sed -i "s/replica_count: [0-9]*/replica_count: 1/" "$CONFIG_FILE"
    sed -i "/^  features:/,/^[a-z]/{s/enabled: false/enabled: true/}" "$CONFIG_FILE"
    ;;
  staging)
    # Update staging settings
    sed -i "s/log_level: .*/log_level: INFO/" "$CONFIG_FILE"
    sed -i "s/replica_count: [0-9]*/replica_count: 2/" "$CONFIG_FILE"
    ;;
  prod)
    # Update production settings
    sed -i "s/log_level: .*/log_level: WARN/" "$CONFIG_FILE"
    sed -i "s/replica_count: [0-9]*/replica_count: 5/" "$CONFIG_FILE"
    sed -i "s/debug_mode: true/debug_mode: false/" "$CONFIG_FILE"
    ;;
  *)
    echo "Unknown environment: $ENV"
    exit 1
    ;;
esac

# Validate the YAML
if ! python3 -c 'import yaml, sys; yaml.safe_load(open(sys.argv[1]))' "$CONFIG_FILE"; then
    echo "YAML validation failed, reverting changes"
    mv "$CONFIG_FILE.bak" "$CONFIG_FILE"
    exit 1
fi

echo "Successfully updated ConfigMap for $ENV environment"

Example 2: Version Bumping in package.json

#!/bin/bash
# Bump version in package.json based on semantic versioning

BUMP_TYPE="${1:-patch}"
PACKAGE_FILE="package.json"

# Extract current version
CURRENT_VERSION=$(sed -n 's/.*"version": "\([^"]*\)".*/\1/p' "$PACKAGE_FILE")
MAJOR=$(echo "$CURRENT_VERSION" | cut -d. -f1)
MINOR=$(echo "$CURRENT_VERSION" | cut -d. -f2)
PATCH=$(echo "$CURRENT_VERSION" | cut -d. -f3)

# Calculate new version
case "$BUMP_TYPE" in
  major)
    NEW_VERSION="$((MAJOR + 1)).0.0"
    ;;
  minor)
    NEW_VERSION="$MAJOR.$((MINOR + 1)).0"
    ;;
  patch)
    NEW_VERSION="$MAJOR.$MINOR.$((PATCH + 1))"
    ;;
  *)
    echo "Unknown bump type: $BUMP_TYPE (use major, minor, or patch)"
    exit 1
    ;;
esac

# Update the version
sed -i "s/\"version\": \"$CURRENT_VERSION\"/\"version\": \"$NEW_VERSION\"/" "$PACKAGE_FILE"

# Validate JSON
if ! python3 -c 'import json, sys; json.load(open(sys.argv[1]))' "$PACKAGE_FILE"; then
    echo "JSON validation failed, something went wrong"
    exit 1
fi

echo "Version bumped from $CURRENT_VERSION to $NEW_VERSION"

Example 3: Multi-environment Properties Configuration

#!/bin/bash
# Configure application.properties for different environments

ENV="${1:-dev}"
PROPS_FILE="src/main/resources/application.properties"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Create backup
cp "$PROPS_FILE" "${PROPS_FILE}.${TIMESTAMP}.bak"

# Common updates for all environments
sed -i "s/^app\.version=.*/app.version=$(date +%Y.%m.%d)/" "$PROPS_FILE"
sed -i "/^# Generated config/c# Generated config on $(date)" "$PROPS_FILE"

# Environment-specific configurations
case "$ENV" in
  dev)
    sed -i "s/^spring\.profiles\.active=.*/spring.profiles.active=dev/" "$PROPS_FILE"
    sed -i "s/^server\.port=.*/server.port=8080/" "$PROPS_FILE"
    sed -i "s/^logging\.level\.root=.*/logging.level.root=DEBUG/" "$PROPS_FILE"
    sed -i "s/^spring\.datasource\.url=.*/spring.datasource.url=jdbc:mysql:\/\/localhost:3306\/devdb/" "$PROPS_FILE"
    ;;
  staging)
    sed -i "s/^spring\.profiles\.active=.*/spring.profiles.active=staging/" "$PROPS_FILE"
    sed -i "s/^server\.port=.*/server.port=8080/" "$PROPS_FILE"
    sed -i "s/^logging\.level\.root=.*/logging.level.root=INFO/" "$PROPS_FILE"
    sed -i "s/^spring\.datasource\.url=.*/spring.datasource.url=jdbc:mysql:\/\/staging-db:3306\/stagingdb/" "$PROPS_FILE"
    ;;
  prod)
    sed -i "s/^spring\.profiles\.active=.*/spring.profiles.active=prod/" "$PROPS_FILE"
    sed -i "s/^server\.port=.*/server.port=80/" "$PROPS_FILE"
    sed -i "s/^logging\.level\.root=.*/logging.level.root=WARN/" "$PROPS_FILE"
    sed -i "s/^spring\.datasource\.url=.*/spring.datasource.url=jdbc:mysql:\/\/prod-db-cluster:3306\/proddb/" "$PROPS_FILE"
    
    # Add production-specific properties if they don't exist
    grep -q "^management\.endpoints\.web\.exposure\.include=" "$PROPS_FILE" || \
      echo "management.endpoints.web.exposure.include=health,info" >> "$PROPS_FILE"
    ;;
  *)
    echo "Unknown environment: $ENV"
    exit 1
    ;;
esac

echo "Successfully configured application.properties for $ENV environment"

Best Practices and Limitations

After years of using sed for configuration management, I’ve learned several important best practices:

Best Practices for sed with Configuration Files

  1. Always create backups before making changes

    sed -i.bak 's/old/new/g' config.file
    
  2. Use line anchors for precise matching

    # Use ^ for start of line to prevent partial matches
    sed -i 's/^debug: true/debug: false/' config.yaml
    
  3. Validate files after modification

    sed -i 's/old/new/g' config.json && json_validate config.json
    
  4. Use comments in complex sed scripts

    # Well-documented sed script
    sed -i -e '# Update API endpoint 
    s|api.old.com|api.new.com|g' config.file
    
  5. Prefer multiple simple sed commands over complex ones

    # Instead of one complex command, use multiple focused ones
    sed -i 's/key1=old1/key1=new1/' config.file
    sed -i 's/key2=old2/key2=new2/' config.file
    
  6. Use version control for configuration files

    # Commit changes after sed modifications
    sed -i 's/old/new/g' config.file && git commit -m "Update config" config.file
    

Limitations and When Not to Use sed

As powerful as sed is, it’s not the right tool for every job. As Robert Love notes in “Linux System Programming”, tools should be used for their intended purpose.

  1. Complex JSON/YAML transformations: For deeply nested structures or complex array manipulations, use dedicated parsers like jq or yq.

  2. When preserving comments is critical: sed can struggle with maintaining comments in specific positions when modifying complex structures.

  3. When you need validation during editing: Format-specific tools can validate while editing.

  4. When the format is exceptionally complex: Some formats have too many edge cases for reliable sed processing.

In these cases, consider alternatives like:

0
  • jq for JSON
  • yq for YAML
  • toml command-line tools for TOML
  • Language-specific configuration libraries

Conclusion: Becoming a sed Master for Configuration Files

Throughout this guide, we’ve explored how to use sed to manipulate different configuration file formats with precision and reliability. From simple property files to complex JSON structures, sed provides the power to automate configuration changes that would otherwise require manual editing.

As Brian Ward emphasizes in “How Linux Works”, mastering text processing tools like sed is a fundamental skill for anyone working with Unix-like systems. The techniques covered in this article will not only make you more efficient at configuration management but also help prevent the kinds of errors that can lead to production outages.

Remember the key principles:

  1. Understand your file format’s structure
  2. Create backups before making changes
  3. Use the right sed techniques for each format
  4. Validate your changes after modification
  5. Start with simple patterns and build up to more complex ones

I encourage you to practice these techniques on sample configuration files before applying them to production systems. With practice, you’ll develop the confidence to automate even the most complex configuration changes safely and efficiently.

What configuration management challenges have you solved with sed? I’d love to hear your experiences and additional techniques in the comments below!

Similar Articles

More from devops

Knowledge Quiz

Test your general knowledge with this quick quiz!

The quiz consists of 5 multiple-choice questions.

Take as much time as you need.

Your score will be shown at the end.