Which command flag is essential when using sed to edit a JSON file in-place with a backup?

Correct answer: -i.bak. The '-i.bak' flag enables in-place editing while creating a backup file with the .bak extension, which is crucial for safely modifying JSON files with sed.

/user/kayd @ devops :~$ cat sed-json-manipulation-without-jq.md

Sed for JSON: Emergency Patterns When jq Is Unavailable Sed for JSON: Emergency Patterns When jq Is Unavailable

Karandeep Singh

Apr 28, 2024 • 19 minutes

Summary

Emergency JSON manipulation patterns using sed when jq is unavailable. For scenarios requiring urgent config changes on a locked-down server with no permission to install tools. Always use jq for JSON - this is the emergency backup plan.

Multiple monitors displaying code in a dark room for JSON data manipulation

Use jq for JSON. Always.

This article shows sed for JSON manipulation as an academic exercise and emergency fallback only.

In production:

Install jq (2MB binary, available everywhere)
Use proper JSON parsers (Python json, Node.js, etc.)
sed breaks on:
- Whitespace changes
- Nested structures
- Unicode escapes
- Edge cases you haven’t thought of

Why this is worth knowing:

You can get stuck on locked-down corporate servers without jq
Production emergencies can require immediate config changes
Sometimes security policies make installing tools take weeks

But ALWAYS push to get jq installed properly. This is the emergency backup plan, not the primary approach.

The Kind of Emergency That Requires sed

Picture a production emergency: a payment processing service is down. The root cause is a Kubernetes ConfigMap with incorrect API endpoints, still pointing to the old payment provider instead of the new one migrated to earlier.

The fix is simple - update one JSON value in the ConfigMap:

{
  "paymentGateway": {
    "provider": "old-provider",
    "apiUrl": "https://old-api.example.com"
  }
}

It should be:

{
  "paymentGateway": {
    "provider": "new-provider",
    "apiUrl": "https://new-api.example.com"
  }
}

The problem: you’re SSH’d into a locked-down production bastion host with no package installation rights. No jq. No Node.js. No yq. Just bash, sed, awk, a stock python3 interpreter, and standard Unix tools — enough to validate JSON, but nothing purpose-built for editing it.

The service is down and every minute counts. You need to update that JSON and kubectl apply the ConfigMap immediately.

This article collects the sed patterns that solve exactly this kind of emergency.

The Emergency Fix: A Pattern for This Scenario

For the situation above, here’s a command sequence that fixes the payment gateway ConfigMap:

# Get the ConfigMap JSON
kubectl get configmap payment-config -o json > /tmp/config.json

# Back up the original
cp /tmp/config.json /tmp/config.json.bak

# Fix the provider value
sed -i 's/\("provider": *\)"old-provider"/\1"new-provider"/' /tmp/config.json

# Fix the API URL (| delimiter avoids escaping the slashes; \. escapes the literal dots)
sed -i 's|\("apiUrl": *\)"https://old-api\.example\.com"|\1"https://new-api.example.com"|' /tmp/config.json

# Validate JSON is still valid
python3 -c "import json; json.load(open('/tmp/config.json'))" 2>&1 | head -1

# Apply the fix
kubectl apply -f /tmp/config.json

This approach lets you apply the patch immediately instead of waiting for approval to install jq. Notice the critical elements:

Backup before modification (cp config.json config.json.bak)
Whitespace flexibility ("provider": * allows any spaces)
Use | as delimiter (avoids escaping / in URLs)
Validate before applying (Python json.load check)

Pattern 1: Updating Simple Key-Value Pairs (The Foundation)

Building on that scenario, here’s the reliable pattern for simple JSON updates:

# Generic pattern for string values
sed -i 's/\("keyname": *\)"old-value"/\1"new-value"/' file.json

# Pattern for numeric values
sed -i 's/\("port": *\)[0-9]*/\18080/' file.json

# Pattern for boolean values
sed -i 's/\("enabled": *\)false/\1true/' file.json

Why This Pattern Works

Capture group preserves key: $"keyname": *$ captures the key and any whitespace
Flexible whitespace matching: * matches zero or more spaces
Backreference rebuilds: \1 inserts the captured key back
Only value changes: Reduces risk of breaking JSON structure

A Practical Example

A common example — changing database endpoints during a failover:

#!/bin/bash
# Failover script - switch to backup database

CONFIG="/etc/app/config.json"
BACKUP="${CONFIG}.bak.$(date +%s)"

# Backup
cp "$CONFIG" "$BACKUP"

# Update DB host
sed -i 's|\("host": *\)"primary-db\.local"|\1"backup-db.local"|' "$CONFIG"

# Update DB port (backup uses different port)
sed -i 's/\("port": *\)5432/\15433/' "$CONFIG"

# Validate
if python3 -c "import json; json.load(open('$CONFIG'))" 2>/dev/null; then
    echo "Database endpoint updated successfully"
    systemctl restart app-service
else
    echo "JSON validation failed, rolling back"
    cp "$BACKUP" "$CONFIG"
    exit 1
fi

A script like this runs reliably during database maintenance windows.

Common Pitfalls

Pitfall #1: Matching wrong values

# BAD: Too generic - matches any "v1"
sed 's/"v1"/"v2"/' config.json

# GOOD: Includes key context
sed 's/\("apiVersion": *\)"v1"/\1"v2"/' config.json

Pitfall #2: Hardcoded whitespace

# BAD: Requires exactly one space
sed 's/"key": "value"/"key": "newvalue"/' config.json

# GOOD: Flexible whitespace
sed 's/"key": *"value"/"key": "newvalue"/' config.json

Pitfall #3: Unescaped special characters

# BAD: Slashes break the pattern
sed 's/"url": "https://old.com"/"url": "https://new.com"/' config.json

# GOOD: Use | as delimiter
sed 's|"url": *"https://old\.com"|"url": "https://new.com"|' config.json

Pattern 2: Navigating Nested Objects

JSON’s power comes from nested structures, which present a greater challenge for line-based tools like sed. Let’s tackle our nested database configuration:

"database": {
  "host": "localhost",
  "port": 5432,
  "credentials": {
    "user": "admin",
    "password": "secret123"
  }
}

Approach 1: Single-Line Replacement for Shallow Nesting

For relatively shallow nesting (1-2 levels), you can use:

# Update database host
sed -i 's/\("database":[^{]*{[^}]*"host": \)"localhost"/\1"db.example.com"/' config.json

This pattern:

Matches "database": followed by any characters up to the opening brace
Continues matching until it finds "host":
Captures all of this in a group
Replaces only the host value

Approach 2: Line-Based Targeting for Deeper Nesting

For deeper nesting, a multi-line approach is more reliable:

# Update database user credentials
sed -i '/"credentials": {/,/}/s/\("user": \)"admin"/\1"dbadmin"/' config.json

This pattern:

Restricts operations to lines between "credentials": { and the next }
Within that range, performs a substitution to update the user

Approach 3: Recursive Descent for Deep Nesting

For the deepest nested structures, recursive descent with multiple conditions provides the most reliability:

# Update database password with deep nesting targeting
sed -i '/"database": {/,/}/{/"credentials": {/,/}/{s/\("password": \)"secret123"/\1"new-secure-pwd"/}}' config.json

This pattern:

First restricts to the database object section
Within that scope, further restricts to the credentials object
Within that narrowed context, performs the password substitution

Key Points for Pattern 2

When working with nested JSON objects, it helps to break complex patterns into logical units. For nested JSON, I’ve found:

Use address ranges - Limit operations to specific sections with /pattern1/,/pattern2/
Narrow scope progressively - Apply multiple filters to target deep structures
Be wary of similar keys - Choose unique context patterns to avoid false matches
Consider field position - Sometimes targeting by position is more reliable than by name

Pattern 3: Handling JSON Arrays

JSON arrays present unique challenges because their elements lack keys. Let’s work with this features array:

"features": ["authentication", "logging", "metrics"]

Replacing a Specific Array Element

To replace a specific array element, you need to target its exact position:

# Replace "logging" with "advanced-logging"
sed -i 's/\("features": \[\("[^"]*", \)*\)"logging"/\1"advanced-logging"/' config.json

This is quite complex, so let’s break it down:

"features": \[ matches the array opening
$"[^"]*", $* matches any number of preceding elements with their commas
"logging" targets the specific element
\1 preserves everything before the target element

Adding an Element to an Array

Adding an element to the end of an array:

# Add "notifications" to the features array
sed -i 's/\("features": \[.*\)\]/\1, "notifications"]/' config.json

This pattern:

Captures everything from "features": [ to just before the closing bracket
Replaces the closing bracket with a comma, the new element, and a new closing bracket

Removing an Element from an Array

Removing an element is perhaps the trickiest operation:

# Remove "metrics" from features array
sed -i 's/\("features": \[.*\), "metrics"\]/\1]/' config.json

# Alternative for middle elements with comma handling
sed -i 's/\("features": \[.*\)"metrics", /\1/; s/\("features": \[.*\), "metrics"/\1/' config.json

The second example shows how to handle both cases (element with trailing comma and element with preceding comma).

Key Points for Pattern 3

Array manipulation requires careful consideration of edge cases. Text processing should account for all possible input variations:

Consider element position - First, middle, and last elements need different handling
Watch for commas - Ensure proper comma placement when adding/removing elements
Use multiple passes if needed - Complex array operations may require multiple sed commands
Validate afterwards - Array operations are particularly prone to breaking JSON validity

Pattern 4: Deleting Properties and Blocks

Sometimes you need to remove entire properties or blocks from JSON. This pattern handles both simple properties and complex nested structures.

Removing a Simple Property

To delete a simple key-value pair:

# Remove the logLevel property
sed -i '/"logLevel": "[^"]*"/d' config.json

This directly deletes any line containing the logLevel property.

Removing a Property and Fixing Commas

The previous approach doesn’t handle commas, which can break JSON syntax. Here’s a more reliable approach:

# Remove logLevel property, then clean up a leftover trailing comma.
# The comma cleanup runs in -z (slurp) mode so \n matches the line break.
sed -i '/"logLevel": "[^"]*",/d' config.json
sed -i -z 's/,\s*\n\s*}/\n}/' config.json

# Remove logLevel property where it has a preceding comma.
# In -z mode a single sub can swallow the comma and the property together,
# even when they sit on different lines.
sed -i -z 's/,\s*\n\s*"logLevel": "[^"]*"//' config.json

A critical detail here: in normal (line-by-line) mode, \n never matches the break between lines, so a sub like s/,\s*\n\s*}/\n}/ is a silent no-op on pretty-printed JSON. The -z flag makes sed treat the whole file as one NUL-separated “line,” so newlines become matchable and the comma cleanup actually fires.

Removing a Nested Block

For removing entire nested structures, you need to identify the block boundaries:

# Remove the entire credentials object
sed -i '/"credentials": {/,/}/d' config.json

This removes all lines from the credentials object opening to its closing brace.

Safely Removing Blocks with Comma Handling

To handle commas properly when removing blocks:

# Remove credentials block and handle commas
sed -i -e '/"credentials": {/,/}/{ /,\s*$/{ s/,\s*$//; b; }; /^\s*}/{n;s/^\s*,\s*//;b}; d;}' config.json

This complex pattern:

Identifies the credentials block boundaries
If the line before the block ends with a comma, removes it
If the line after the block starts with a comma, removes it
Deletes all lines in the block

Key Points for Pattern 4

Removing elements requires particular attention to maintaining valid JSON syntax:

Handle commas carefully - Both trailing and leading commas need cleaning up
Consider whitespace - Account for various indentation patterns
Multi-step approach - Complex removals may require multiple operations
Validate structure - Removal operations are high-risk for breaking JSON syntax

Pattern 5: Adding New Properties and Objects

Finally, let’s look at adding entirely new properties or objects to JSON.

Adding a Simple Property

To add a property to the root object:

# Add a new timeout property at the root level
sed -i 's/\([^{]*{[^}]*\)}/\1,\n  "timeout": 30\n}/' config.json

This pattern:

Captures everything from the opening { to the last property before the closing }
Adds a comma, newline, and the new property before the closing brace

Important: this s/$[^{]*{[^}]*$}/.../ form only works on single-line / compact JSON, where the {, the properties, and the closing } all sit on one line. On the pretty-printed, multi-line examples shown elsewhere in this article it’s a silent no-op, because [^}]* stops at the first newline before it ever reaches the closing brace. To use it on multi-line files, either compact the JSON first (e.g. tr -d '\n') or run the substitution in -z (slurp) mode. The same caveat applies to the nested-object additions below.

Adding a Nested Property

To add a property to a nested object:

# Add sslEnabled property to database object
sed -i '/"database": {/,/}/{s/\([^{]*{[^}]*\)}/\1,\n    "sslEnabled": true\n  }/}' config.json

This restricts the operation to the database section and then adds the new property.

Adding a Complete Nested Object

To add an entirely new nested object:

# Add a new monitoring object
sed -i 's/\([^{]*{[^}]*\)}/\1,\n  "monitoring": {\n    "enabled": true,\n    "interval": 60\n  }\n}/' config.json

This follows the same pattern but adds a multi-line JSON object with proper indentation.

Targeting Array Positions

For more precise positioning, you can target known properties:

# Add property after apiVersion
sed -i '/"apiVersion"/a\  "environment": "production",' config.json

This adds the new property on a new line after the line containing “apiVersion”.

Key Points for Pattern 5

Adding new properties requires careful attention to structure:

Maintain proper indentation - Keep consistent formatting for readability
Handle commas properly - Ensure commas separate properties correctly
Consider pretty-printing afterward - Complex additions may benefit from reformatting
Incremental approach - Add complex structures in multiple steps

Validation Techniques for sed-Modified JSON

After modifying JSON with sed, validation is crucial. Here are techniques I’ve developed to ensure JSON remains valid:

Basic Syntax Validation

The simplest approach uses Python (available on most systems):

# Validate JSON after modification
function validate_json() {
    python3 -c "import json; json.load(open('$1'))" 2>/dev/null
    return $?
}

# Example usage
sed -i 's/"debug": true/"debug": false/' config.json
if ! validate_json config.json; then
    echo "JSON validation failed!"
    # Restore from backup if using sed -i.bak
    [ -f config.json.bak ] && mv config.json.bak config.json
    exit 1
fi

In-Place Validation with Automatic Rollback

This pattern combines modification and validation in one step:

#!/bin/bash
# Safe JSON editing with automatic rollback

JSON_FILE="config.json"
BACKUP="${JSON_FILE}.bak"

# Create backup
cp "$JSON_FILE" "$BACKUP"

# Perform modification
sed -i 's/"apiVersion": "v1"/"apiVersion": "v2"/' "$JSON_FILE"

# Validate
if ! python3 -c "import json; json.load(open('$JSON_FILE'))" 2>/dev/null; then
    echo "JSON validation failed, rolling back changes"
    mv "$BACKUP" "$JSON_FILE"
    exit 1
else
    echo "JSON successfully modified and validated"
    # Remove backup if no longer needed
    rm "$BACKUP"
fi

Pretty-Printing After Modification

For better readability after complex modifications:

#!/bin/bash
# Modify and pretty-print JSON

JSON_FILE="config.json"

# Create backup
cp "$JSON_FILE" "${JSON_FILE}.bak"

# Perform modification
sed -i 's/"port": 8080/"port": 9090/' "$JSON_FILE"

# Validate and pretty-print
if python3 -c "
import json, sys
with open('$JSON_FILE') as f:
    data = json.load(f)
with open('$JSON_FILE', 'w') as f:
    json.dump(data, f, indent=2)
sys.exit(0)
" 2>/dev/null; then
    echo "JSON successfully modified, validated, and formatted"
else
    echo "JSON validation failed, rolling back changes"
    mv "${JSON_FILE}.bak" "$JSON_FILE"
    exit 1
fi

Structured Validation with Specific Checks

For more advanced validation:

#!/bin/bash
# Validate specific JSON properties after modification

JSON_FILE="config.json"

# Create backup
cp "$JSON_FILE" "${JSON_FILE}.bak"

# Perform modification
sed -i 's/"apiVersion": "v1"/"apiVersion": "v2"/' "$JSON_FILE"

# Validate structure and specific values
if python3 -c "
import json, sys
try:
    with open('$JSON_FILE') as f:
        data = json.load(f)
    
    # Verify specific properties
    assert data['apiVersion'] == 'v2', 'API version not updated correctly'
    assert 'port' in data, 'Required property \"port\" is missing'
    
    print('JSON validation successful')
    sys.exit(0)
except Exception as e:
    print(f'Validation error: {e}')
    sys.exit(1)
" 2>/dev/null; then
    echo "JSON successfully modified and validated"
else
    echo "JSON validation failed, rolling back changes"
    mv "${JSON_FILE}.bak" "$JSON_FILE"
    exit 1
fi

Real-World Examples: sed in CI/CD Pipelines

Let’s put these patterns into practice with real-world examples from CI/CD environments where specialized JSON tools might not be available.

Example 1: Dynamic Environment Configuration

#!/bin/bash
# Update application.json for different environments

ENV="${1:-dev}"
CONFIG_FILE="application.json"

# Create backup
cp "$CONFIG_FILE" "${CONFIG_FILE}.bak"

case "$ENV" in
  dev)
    # Development environment settings
    sed -i 's/\("apiUrl": \)"[^"]*"/\1"https:\/\/dev-api.example.com"/' "$CONFIG_FILE"
    sed -i 's/\("logLevel": \)"[^"]*"/\1"debug"/' "$CONFIG_FILE"
    sed -i 's/\("debug": \)[^,]*/\1true/' "$CONFIG_FILE"
    sed -i '/"features": \[/,/]/s/\("features": \[\)[^]]*\]/\1"login", "profile", "dev-console"]/' "$CONFIG_FILE"
    ;;
    
  staging)
    # Staging environment settings
    sed -i 's/\("apiUrl": \)"[^"]*"/\1"https:\/\/staging-api.example.com"/' "$CONFIG_FILE"
    sed -i 's/\("logLevel": \)"[^"]*"/\1"info"/' "$CONFIG_FILE"
    sed -i 's/\("debug": \)[^,]*/\1false/' "$CONFIG_FILE"
    sed -i '/"features": \[/,/]/s/\("features": \[\)[^]]*\]/\1"login", "profile"]/' "$CONFIG_FILE"
    ;;
    
  prod)
    # Production environment settings
    sed -i 's/\("apiUrl": \)"[^"]*"/\1"https:\/\/api.example.com"/' "$CONFIG_FILE"
    sed -i 's/\("logLevel": \)"[^"]*"/\1"warn"/' "$CONFIG_FILE"
    sed -i 's/\("debug": \)[^,]*/\1false/' "$CONFIG_FILE"
    sed -i '/"features": \[/,/]/s/\("features": \[\)[^]]*\]/\1"login", "profile"]/' "$CONFIG_FILE"
    
    # Remove development-only settings
    sed -i '/"devTools": {/,/}/d' "$CONFIG_FILE"
    sed -i -z 's/,\s*\n\s*}/\n}/' "$CONFIG_FILE" # Fix trailing commas (-z so \n matches the line break)
    ;;
    
  *)
    echo "Unknown environment: $ENV"
    exit 1
    ;;
esac

# Validate the JSON
if ! python3 -c "import json; json.load(open('$CONFIG_FILE'))" 2>/dev/null; then
    echo "JSON validation failed, reverting changes"
    mv "${CONFIG_FILE}.bak" "$CONFIG_FILE"
    exit 1
fi

echo "Successfully configured application for $ENV environment"

Example 2: Dynamic Kubernetes Resource Configuration

#!/bin/bash
# Update Kubernetes resource limits in deployment.json

RESOURCE_PRESET="${1:-medium}"
DEPLOYMENT_FILE="deployment.json"

# Create backup
cp "$DEPLOYMENT_FILE" "${DEPLOYMENT_FILE}.bak"

# Extract the current resource section for easier targeting
RESOURCE_SECTION=$(grep -A 10 '"resources":' "$DEPLOYMENT_FILE")

case "$RESOURCE_PRESET" in
  small)
    # Small resource allocation
    sed -i '/"resources": {/,/}/{
      s/\("cpu": \)"[^"]*"/\1"100m"/
      s/\("memory": \)"[^"]*"/\1"128Mi"/
    }' "$DEPLOYMENT_FILE"
    ;;
    
  medium)
    # Medium resource allocation
    sed -i '/"resources": {/,/}/{
      s/\("cpu": \)"[^"]*"/\1"500m"/
      s/\("memory": \)"[^"]*"/\1"512Mi"/
    }' "$DEPLOYMENT_FILE"
    ;;
    
  large)
    # Large resource allocation
    sed -i '/"resources": {/,/}/{
      s/\("cpu": \)"[^"]*"/\1"1000m"/
      s/\("memory": \)"[^"]*"/\1"1Gi"/
    }' "$DEPLOYMENT_FILE"
    ;;
    
  *)
    echo "Unknown resource preset: $RESOURCE_PRESET"
    exit 1
    ;;
esac

# Also update replica count based on preset
case "$RESOURCE_PRESET" in
  small)
    sed -i 's/\("replicas": \)[0-9]*/\11/' "$DEPLOYMENT_FILE"
    ;;
  medium)
    sed -i 's/\("replicas": \)[0-9]*/\13/' "$DEPLOYMENT_FILE"
    ;;
  large)
    sed -i 's/\("replicas": \)[0-9]*/\15/' "$DEPLOYMENT_FILE"
    ;;
esac

# Validate the JSON
if ! python3 -c "import json; json.load(open('$DEPLOYMENT_FILE'))" 2>/dev/null; then
    echo "JSON validation failed, reverting changes"
    mv "${DEPLOYMENT_FILE}.bak" "$DEPLOYMENT_FILE"
    exit 1
fi

echo "Successfully updated deployment resources to $RESOURCE_PRESET preset"

Example 3: Feature Flag Management

#!/bin/bash
# Manage feature flags in config.json

ACTION="$1"
FEATURE="$2"
CONFIG_FILE="config.json"

# Validate input
if [ -z "$ACTION" ] || [ -z "$FEATURE" ]; then
    echo "Usage: $0 [enable|disable|add|remove] feature_name"
    exit 1
fi

# Create backup
cp "$CONFIG_FILE" "${CONFIG_FILE}.bak"

# Check if features section exists
if ! grep -q '"features":' "$CONFIG_FILE"; then
    echo "Error: Features section not found in $CONFIG_FILE"
    exit 1
fi

case "$ACTION" in
  enable)
    # Enable an existing feature
    sed -i '/"features": {/,/}/{s/\("'"$FEATURE"'": \)false/\1true/}' "$CONFIG_FILE"
    ;;
    
  disable)
    # Disable an existing feature
    sed -i '/"features": {/,/}/{s/\("'"$FEATURE"'": \)true/\1false/}' "$CONFIG_FILE"
    ;;
    
  add)
    # Add a new feature (enabled by default).
    # An inline `i\` insert here trips over GNU sed ("unmatched `{'") because the
    # inserted text isn't terminated before the block's closing brace. awk inserts
    # the new key right after the "features": { line instead, which is both
    # readable and portable.
    awk -v feat="$FEATURE" '
      /"features": \{/ { print; print "    \"" feat "\": true,"; next }
      { print }
    ' "$CONFIG_FILE" > "${CONFIG_FILE}.tmp" && mv "${CONFIG_FILE}.tmp" "$CONFIG_FILE"
    ;;
    
  remove)
    # Remove a feature entirely
    sed -i '/"features": {/,/}/{/\s*"'"$FEATURE"'": [^,]*,\?/d}' "$CONFIG_FILE"
    # Fix trailing commas if needed
    sed -i '/"features": {/,/}/s/,\s*}/\n  }/' "$CONFIG_FILE"
    ;;
    
  *)
    echo "Unknown action: $ACTION (use enable, disable, add, or remove)"
    exit 1
    ;;
esac

# Validate the JSON
if ! python3 -c "import json; json.load(open('$CONFIG_FILE'))" 2>/dev/null; then
    echo "JSON validation failed, reverting changes"
    mv "${CONFIG_FILE}.bak" "$CONFIG_FILE"
    exit 1
fi

echo "Successfully $ACTION""d feature '$FEATURE'"

Best Practices for JSON Manipulation with sed

After years of working with sed for JSON manipulation, I’ve established these best practices:

1. Always Create Backups

# Always use -i with a suffix for in-place editing
sed -i.bak 's/"debug": true/"debug": false/' config.json

Data integrity requires defense-in-depth strategies.

2. Validate After Every Change

# Simple validation function
validate_json() {
    python3 -c "import json; json.load(open('$1'))" 2>/dev/null
    return $?
}

# Use with error handling
sed -i.bak 's/"debug": true/"debug": false/' config.json
if ! validate_json config.json; then
    echo "Invalid JSON after modification"
    mv config.json.bak config.json
    exit 1
fi

3. Use Capture Groups for Context

# Bad: Fragile pattern without context
sed -i 's/"v1"/"v2"/' config.json  # Might replace wrong values!

# Good: Captures key context for precision
sed -i 's/\("apiVersion": \)"v1"/\1"v2"/' config.json

4. Break Complex Operations into Steps

# Update multiple values in stages with validation between steps
sed -i.bak1 's/"port": 8080/"port": 9090/' config.json
validate_json config.json || { mv config.json.bak1 config.json; exit 1; }

sed -i.bak2 's/"debug": true/"debug": false/' config.json
validate_json config.json || { mv config.json.bak2 config.json; exit 1; }

5. Consider Formatting Variations

# More flexible pattern that handles whitespace variations
sed -i 's/\("port"[[:space:]]*:[[:space:]]*\)8080/\19090/' config.json

Reliable pattern matching accounts for all valid input variations.

6. Document Complex Patterns

# Well-documented complex sed command
sed -i '
# Target the database credentials block
/"credentials": {/,/}/{
  # Replace the password with a new value
  s/\("password": \)"[^"]*"/\1"new-secure-password"/
}' config.json

7. Test on Representative Samples

Always test your patterns on sample data that represents all formatting variations you might encounter in production.

8. Consider Function Libraries

Build reusable functions for common JSON operations:

# Function library for JSON manipulation with sed
update_json_string_property() {
    local file="$1"
    local property="$2"
    local value="$3"
    
    sed -i.bak "s/\\(\"$property\": \\)\"[^\"]*\"/\\1\"$value\"/" "$file"
    if ! validate_json "$file"; then
        mv "$file.bak" "$file"
        return 1
    fi
    return 0
}

# Usage
update_json_string_property "config.json" "apiVersion" "v2"

Limitations and Alternatives

While these patterns are effective for many scenarios, there are limitations to using sed for JSON manipulation:

When to Use Other Tools

Complex queries - When you need to extract data based on complex criteria
Deeply nested structures - When targeting elements requires traversing many layers
Large-scale transformations - When performing many operations that might compound errors
Array manipulations - When operations go beyond simple replacements

Lightweight Alternatives to jq

If you can’t use jq but need more reliable JSON handling, consider:

Python one-liners - For systems with Python but no dedicated JSON tools

python3 -c "
import json, sys;
data = json.load(open('config.json'));
data['apiVersion'] = 'v2';
json.dump(data, open('config.json', 'w'), indent=2)
"

Node.js one-liners - For systems with Node.js installed

node -e "
const fs = require('fs');
const data = JSON.parse(fs.readFileSync('config.json'));
data.apiVersion = 'v2';
fs.writeFileSync('config.json', JSON.stringify(data, null, 2));
"

grep/cut/awk combinations - For extremely minimal environments
Deepen your understanding in Sed vs Awk vs Grep: When to Use Which (with Decision Matrix)
1
```
# Extract a value using grep/cut (very primitive approach)
grep '"apiVersion"' config.json | cut -d'"' -f4
```

When sed for JSON is Actually Appropriate

After many production uses, here’s when sed for JSON is acceptable:

Valid Use Cases

Emergency production fixes on locked-down servers
Minimal container images where adding jq bloats the image (Alpine Linux CI containers)
Simple config updates in bootstrap scripts before package managers work
One-time migrations where installing tools takes longer than writing sed

Never Use sed for JSON When

Data is user-generated or untrusted - you’ll miss edge cases
Structure is deeply nested (more than 2 levels deep)
You’re processing arrays with dynamic content
Whitespace matters for your application
You can install jq - seriously, just install jq

The Right Way: Always Push for jq

After any emergency sed-for-JSON workaround, file a ticket to fix the underlying problem:

Locked-down bastion host: Request a security exception to install jq.
CI container: Add jq to the base image (small footprint, saves countless sed hacks).
Bootstrap script: Modify provisioning to install jq earlier.

In all of these cases, having jq available eliminates entire categories of bugs and dramatically reduces script complexity.

Lessons from Production

What worked:

Always backup before modification (cp file.json file.json.bak)
Always validate after changes (python3 -c "import json; json.load(...)")
Use flexible whitespace patterns ( * instead of single space)
Capture groups preserve context ($"key": *$)

What failed:

Complex nested updates (multiple attempts before getting right)
Array manipulations (broke JSON often enough to be unreliable)
Updates without validation (broke production more than once)
Patterns without backups (couldn’t rollback failed changes)

Patterns from production use:

Simple key-value updates: highly reliable
Nested updates (2 levels): mostly reliable, but occasionally fragile
Array manipulations: unreliable, frequently reverted to manual fixes

The Bottom Line

sed for JSON is a hack. An emergency hack. A “the building is on fire” hack.

Its value is allowing an immediate fix when you can’t install jq, instead of waiting for security approval first.

But it’s not a strategy. It’s a fallback.

If you’re reading this to learn sed for JSON patterns: stop. Install jq instead. It’s 2MB and solves this problem correctly.

If you’re reading this because you’re in an emergency and can’t install jq: these patterns will get you through. Then fix the underlying problem so you never need sed for JSON again.

Install jq: The Real Solution

# Debian/Ubuntu
apt-get install jq

# RHEL/CentOS
yum install jq

# Alpine Linux
apk add jq

# macOS
brew install jq

# From source (when you have no package manager)
wget https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64
chmod +x jq-linux-amd64
sudo mv jq-linux-amd64 /usr/local/bin/jq

The same emergency fix with jq:

# The same fix, which would have taken seconds with jq
jq '.paymentGateway.provider = "new-provider" | .paymentGateway.apiUrl = "https://new-api.example.com"' config.json > config.json.new
mv config.json.new config.json

Clean. Safe. Readable. No regex escaping. No validation concerns. No edge cases.

That’s why you use jq.

References

Question

Have you been stuck manipulating JSON without proper tools? What forced you into using sed or awk for JSON?

More from devops

Upgrading Jenkins: Migration Strategies for Major Version Jumps

How to plan and execute Jenkins upgrades safely, including in-place, blue-green, and phased paths …

Docker Compose, Bake & ECR: Build and Ship Apps

Build a multi-container app with Docker Compose, then build images with Docker Bake and push them to …

Kubernetes on AWS: EKS Setup with eksctl

Set up a Kubernetes cluster on AWS EKS with eksctl: prerequisites, one-command cluster creation, …

Sed for Log Analysis: Errors, Time Filters, Patterns

Practical sed patterns for log analysis: extract errors, filter time ranges, anonymize PII, parse …

Sed Gotchas: GNU vs BSD and Safe In-Place Editing

The sed gotchas that bite in production: GNU vs BSD differences, in-place editing safety, escape …

Sed in CI/CD: Safe Patterns for GitHub Actions and Jenkins

Use sed safely in CI/CD pipelines: idempotent edits, exit-code checks, dry-run patterns, and the …

Knowledge Quiz

Test your general knowledge with this quick quiz!

A set of multiple-choice questions to test your knowledge.

Take as much time as you need.

Your score will be shown at the end.

Question 1 of 5

Score: 0

Quiz Complete!

Your score: 0 out of 5

Loading next question...

Use jq for JSON. Always.

The Kind of Emergency That Requires sed

The Emergency Fix: A Pattern for This Scenario

Pattern 1: Updating Simple Key-Value Pairs (The Foundation)

Why This Pattern Works

A Practical Example

Common Pitfalls

Pattern 2: Navigating Nested Objects

Approach 1: Single-Line Replacement for Shallow Nesting

Approach 2: Line-Based Targeting for Deeper Nesting

Approach 3: Recursive Descent for Deep Nesting

Key Points for Pattern 2

Pattern 3: Handling JSON Arrays

Replacing a Specific Array Element

Adding an Element to an Array

Removing an Element from an Array

Key Points for Pattern 3

Pattern 4: Deleting Properties and Blocks

Removing a Simple Property

Removing a Property and Fixing Commas

Removing a Nested Block

Safely Removing Blocks with Comma Handling

Key Points for Pattern 4

Pattern 5: Adding New Properties and Objects

Adding a Simple Property

Adding a Nested Property

Adding a Complete Nested Object

Targeting Array Positions

Key Points for Pattern 5

Validation Techniques for sed-Modified JSON

Basic Syntax Validation

In-Place Validation with Automatic Rollback

Pretty-Printing After Modification

Structured Validation with Specific Checks

Real-World Examples: sed in CI/CD Pipelines

Example 1: Dynamic Environment Configuration

Example 2: Dynamic Kubernetes Resource Configuration

Example 3: Feature Flag Management

Best Practices for JSON Manipulation with sed

1. Always Create Backups

2. Validate After Every Change

3. Use Capture Groups for Context

4. Break Complex Operations into Steps

5. Consider Formatting Variations

6. Document Complex Patterns

7. Test on Representative Samples

8. Consider Function Libraries

Limitations and Alternatives

When to Use Other Tools

Lightweight Alternatives to jq

When sed for JSON is Actually Appropriate

Valid Use Cases

Never Use sed for JSON When

The Right Way: Always Push for jq

Lessons from Production

The Bottom Line

Install jq: The Real Solution

References

Similar Articles

Related Content

More from devops

You Might Also Like

Knowledge Quiz

Question 1 of 5

Quiz Complete!