A CI pipeline runs your tests, builds your code, and tells you if something broke. Most teams use hosted services for this. But if you understand what happens inside a CI pipeline, you can debug failures faster, write better build configurations, and know when a tool is doing something wrong.
This article builds a CI pipeline from scratch. Each step starts with shell commands that do the work. Then we write Go code that does the same thing programmatically. Along the way, we hit real bugs and fix them.
By the end, you will have a working CI build runner written in Go. It polls a git repository, runs build stages, parses test output, collects artifacts, and runs stages in parallel.
You need basic shell knowledge and some Go experience. All Go code uses only the standard library.
Step 1: Running Tests from Shell
The simplest CI pipeline runs tests and checks the exit code. If the exit code is zero, the tests passed. If it is not zero, something failed.
Shell: Run tests and check the result
Start with a Go project. Run the tests and look at the exit code.
go test ./...
echo $?
If all tests pass, echo $? prints 0. If any test fails, it prints 1 or 2.
But there is a problem. If you run multiple commands in a script, the script continues even after a failure. Add set -e to stop on the first error.
#!/bin/bash
set -e
go test ./...
echo "Tests passed"
go build -o ./app ./cmd/server
echo "Build succeeded"
With set -e, the script stops at go test if any test fails. The echo "Tests passed" line never runs.
There is another trap. Pipes do not propagate errors by default. This command succeeds even if go test fails:
go test ./... | tee test-output.txt
echo $?
The exit code comes from tee, not from go test. Fix this with set -o pipefail:
#!/bin/bash
set -e
set -o pipefail
go test ./... | tee test-output.txt
echo "Tests passed"
Now the script exits if go test fails, even through the pipe.
Save the output to a file for later review:
go test ./... > test-output.txt 2>&1
The 2>&1 sends stderr to the same file as stdout. Without it, you lose error messages.
Go: Build a command runner
Now build the same thing in Go. A function that runs a command and returns the output and exit code.
Here is the first attempt:
package main
import (
"fmt"
"os/exec"
)
func runCommand(name string, args ...string) (string, int, error) {
cmd := exec.Command(name, args...)
output, err := cmd.Output()
if err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
return string(output), exitErr.ExitCode(), nil
}
return "", -1, err
}
return string(output), 0, nil
}
func main() {
output, code, err := runCommand("go", "test", "./...")
if err != nil {
fmt.Printf("Error: %v\n", err)
return
}
fmt.Printf("Exit code: %d\n", code)
fmt.Printf("Output:\n%s\n", output)
}
Run this against a project with a failing test. The exit code is 1, which is correct. But the output is empty.
The bug: Missing stderr
cmd.Output() only captures stdout. When a test fails, Go prints the failure message to stdout, but compilation errors and some diagnostics go to stderr. The captured output misses important information.
Here is what you see:
Exit code: 1
Output:
The output is blank. The failure details went to stderr, which was discarded.
The fix: Combine stdout and stderr
Replace cmd.Output() with cmd.CombinedOutput():
package main
import (
"fmt"
"os/exec"
)
func runCommand(name string, args ...string) (string, int, error) {
cmd := exec.Command(name, args...)
output, err := cmd.CombinedOutput()
if err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
return string(output), exitErr.ExitCode(), nil
}
return "", -1, err
}
return string(output), 0, nil
}
func main() {
output, code, err := runCommand("go", "test", "./...")
if err != nil {
fmt.Printf("Error: %v\n", err)
return
}
fmt.Printf("Exit code: %d\n", code)
fmt.Printf("Output:\n%s\n", output)
}
Now the output contains both stdout and stderr. You see the full failure message:
Exit code: 1
Output:
--- FAIL: TestAdd (0.00s)
math_test.go:10: expected 4, got 5
FAIL example.com/myproject/math 0.003s
This is the foundation. Every CI system does this: run a command, capture output, check exit code.
Step 2: Build Steps and Artifacts
A real CI pipeline has multiple steps. Compile the code. Run the tests. Package the result. Each step depends on the previous one succeeding.
Shell: Build, package, and verify
A typical build script:
#!/bin/bash
set -e
set -o pipefail
echo "=== Running tests ==="
go test ./...
echo "=== Building binary ==="
mkdir -p dist
go build -o ./dist/app ./cmd/server
echo "=== Collecting artifacts ==="
cp config.yaml dist/
cp -r templates/ dist/templates/
echo "=== Packaging release ==="
tar -czf release.tar.gz -C dist .
echo "=== Generating checksum ==="
sha256sum release.tar.gz > checksums.txt
echo "=== Done ==="
ls -la release.tar.gz checksums.txt
The sha256sum command creates a checksum file. Anyone who downloads the release can verify the file was not corrupted or tampered with:
sha256sum -c checksums.txt
This prints release.tar.gz: OK if the file matches.
Go: Build a step executor
Now build this in Go. Define a step as a name and a command. Run each step in order.
Here is the first attempt:
package main
import (
"fmt"
"os/exec"
"time"
)
type Step struct {
Name string
Command string
Args []string
}
func runStep(s Step) (string, int, error) {
cmd := exec.Command(s.Command, s.Args...)
output, err := cmd.CombinedOutput()
if err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
return string(output), exitErr.ExitCode(), nil
}
return "", -1, err
}
return string(output), 0, nil
}
func runPipeline(steps []Step) {
for _, step := range steps {
fmt.Printf("--- %s ---\n", step.Name)
start := time.Now()
output, code, err := runStep(step)
elapsed := time.Since(start)
if err != nil {
fmt.Printf(" Error: %v\n", err)
}
fmt.Printf(" Exit code: %d\n", code)
fmt.Printf(" Duration: %s\n", elapsed.Round(time.Millisecond))
if output != "" {
fmt.Printf(" Output:\n%s\n", output)
}
}
}
func main() {
steps := []Step{
{Name: "Run tests", Command: "go", Args: []string{"test", "./..."}},
{Name: "Build binary", Command: "go", Args: []string{"build", "-o", "./dist/app", "./cmd/server"}},
{Name: "Package release", Command: "tar", Args: []string{"-czf", "release.tar.gz", "-C", "dist", "."}},
}
runPipeline(steps)
}
The bug: No error propagation
Run this with a failing test step. The test step fails with exit code 1. But the build step runs anyway. And the package step runs after that. The pipeline reports three results, but the last two are meaningless because they ran on broken code.
--- Run tests ---
Exit code: 1
Duration: 1.234s
--- Build binary ---
Exit code: 0
Duration: 2.456s
--- Package release ---
Exit code: 0
Duration: 0.089s
The pipeline looks like it mostly succeeded. It did not. The tests failed. Everything after that should have stopped.
The fix: Stop on first failure
Check the exit code after each step. If it is not zero, stop the pipeline.
package main
import (
"fmt"
"os/exec"
"time"
)
type Step struct {
Name string
Command string
Args []string
}
type StepResult struct {
Name string
Output string
ExitCode int
Duration time.Duration
Err error
}
func runStep(s Step) StepResult {
start := time.Now()
cmd := exec.Command(s.Command, s.Args...)
output, err := cmd.CombinedOutput()
elapsed := time.Since(start)
if err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
return StepResult{
Name: s.Name,
Output: string(output),
ExitCode: exitErr.ExitCode(),
Duration: elapsed,
}
}
return StepResult{
Name: s.Name,
ExitCode: -1,
Duration: elapsed,
Err: err,
}
}
return StepResult{
Name: s.Name,
Output: string(output),
ExitCode: 0,
Duration: elapsed,
}
}
func runPipeline(steps []Step) []StepResult {
var results []StepResult
for _, step := range steps {
fmt.Printf("--- %s ---\n", step.Name)
result := runStep(step)
results = append(results, result)
fmt.Printf(" Exit code: %d\n", result.ExitCode)
fmt.Printf(" Duration: %s\n", result.Duration.Round(time.Millisecond))
if result.Output != "" {
fmt.Printf(" Output:\n%s\n", result.Output)
}
if result.ExitCode != 0 || result.Err != nil {
fmt.Printf(" FAILED. Stopping pipeline.\n")
break
}
}
return results
}
func main() {
steps := []Step{
{Name: "Run tests", Command: "go", Args: []string{"test", "./..."}},
{Name: "Build binary", Command: "go", Args: []string{"build", "-o", "./dist/app", "./cmd/server"}},
{Name: "Package release", Command: "tar", Args: []string{"-czf", "release.tar.gz", "-C", "dist", "."}},
}
results := runPipeline(steps)
fmt.Printf("\n=== Summary ===\n")
failed := false
for _, r := range results {
status := "PASS"
if r.ExitCode != 0 || r.Err != nil {
status = "FAIL"
failed = true
}
fmt.Printf(" [%s] %s (%s)\n", status, r.Name, r.Duration.Round(time.Millisecond))
}
if failed {
fmt.Println("Pipeline: FAILED")
} else {
fmt.Println("Pipeline: PASSED")
}
}
Now when the test step fails, the pipeline stops immediately:
--- Run tests ---
Exit code: 1
Duration: 1.234s
Output:
--- FAIL: TestAdd (0.00s)
math_test.go:10: expected 4, got 5
FAIL
FAILED. Stopping pipeline.
=== Summary ===
[FAIL] Run tests (1.234s)
Pipeline: FAILED
No wasted time building code that already failed tests.
Step 3: Parsing Test Output
Exit codes tell you pass or fail. But a CI pipeline should report which tests failed, how many passed, and what the coverage is. Go has built-in JSON test output that makes this possible.
Shell: JSON test output
Run tests with the -json flag:
go test -json ./...
Each line is a JSON object:
{"Time":"2024-01-15T10:00:00Z","Action":"run","Package":"example.com/app/math","Test":"TestAdd"}
{"Time":"2024-01-15T10:00:00Z","Action":"output","Package":"example.com/app/math","Test":"TestAdd","Output":"=== RUN TestAdd\n"}
{"Time":"2024-01-15T10:00:00Z","Action":"output","Package":"example.com/app/math","Test":"TestAdd","Output":"--- PASS: TestAdd (0.00s)\n"}
{"Time":"2024-01-15T10:00:00Z","Action":"pass","Package":"example.com/app/math","Test":"TestAdd","Elapsed":0}
The Action field has values like run, output, pass, fail, and skip. To find just the failures:
go test -json ./... | grep '"Action":"fail"'
For coverage:
go test -cover ./...
This prints a line like coverage: 82.5% of statements for each package.
To combine JSON output with coverage:
go test -json -cover ./...
Coverage appears in output action lines.
Go: Parse JSON test output
Build a parser that reads JSON test output and collects results. Here is the first attempt:
package main
import (
"bufio"
"encoding/json"
"fmt"
"os/exec"
"strings"
)
type TestEvent struct {
Action string `json:"Action"`
Package string `json:"Package"`
Test string `json:"Test"`
Output string `json:"Output"`
Elapsed float64 `json:"Elapsed"`
}
type TestResult struct {
Package string
Test string
Status string
Output string
Elapsed float64
}
func parseTestOutput(data string) []TestResult {
var results []TestResult
scanner := bufio.NewScanner(strings.NewReader(data))
for scanner.Scan() {
var event TestEvent
if err := json.Unmarshal(scanner.Bytes(), &event); err != nil {
continue
}
if event.Test == "" {
continue
}
if event.Action == "pass" || event.Action == "fail" || event.Action == "skip" {
results = append(results, TestResult{
Package: event.Package,
Test: event.Test,
Status: event.Action,
Elapsed: event.Elapsed,
})
}
}
return results
}
func main() {
cmd := exec.Command("go", "test", "-json", "./...")
output, _ := cmd.CombinedOutput()
results := parseTestOutput(string(output))
passed := 0
failed := 0
skipped := 0
for _, r := range results {
switch r.Status {
case "pass":
passed++
case "fail":
failed++
case "skip":
skipped++
}
}
fmt.Printf("Passed: %d, Failed: %d, Skipped: %d\n", passed, failed, skipped)
if failed > 0 {
fmt.Println("\nFailed tests:")
for _, r := range results {
if r.Status == "fail" {
fmt.Printf(" %s/%s\n", r.Package, r.Test)
}
}
}
}
This version works for simple cases. But it has a subtle bug.
The bug: Counting output lines as results
The JSON output has multiple actions for each test: run, output (possibly many times), then a final pass or fail. The code above correctly filters for pass, fail, and skip actions only. That part is right.
But here is the problem: subtests. A test like TestMath/Add produces a pass action. Then the parent test TestMath also produces a pass action. If you count both, your numbers are inflated.
Run it against a project with subtests:
Passed: 8, Failed: 0, Skipped: 0
But the project only has 4 test functions. The subtests and parent tests are all counted separately.
The fix: Track tests by key, count only leaf tests
Use a map keyed by package and test name. Only count tests that do not have subtests (leaf tests). If a test name contains /, it is a subtest. Count it. If a test name has no /, check if any subtest exists for it. If so, skip the parent.
package main
import (
"bufio"
"encoding/json"
"fmt"
"os/exec"
"strings"
)
type TestEvent struct {
Action string `json:"Action"`
Package string `json:"Package"`
Test string `json:"Test"`
Output string `json:"Output"`
Elapsed float64 `json:"Elapsed"`
}
type TestResult struct {
Package string
Test string
Status string
Output string
Elapsed float64
}
func parseTestOutput(data string) []TestResult {
type testKey struct {
pkg string
test string
}
finalStatus := make(map[testKey]TestResult)
hasSubtest := make(map[testKey]bool)
scanner := bufio.NewScanner(strings.NewReader(data))
for scanner.Scan() {
var event TestEvent
if err := json.Unmarshal(scanner.Bytes(), &event); err != nil {
continue
}
if event.Test == "" {
continue
}
key := testKey{pkg: event.Package, test: event.Test}
if event.Action == "output" {
existing := finalStatus[key]
existing.Output += event.Output
existing.Package = event.Package
existing.Test = event.Test
finalStatus[key] = existing
}
if event.Action == "pass" || event.Action == "fail" || event.Action == "skip" {
r := finalStatus[key]
r.Package = event.Package
r.Test = event.Test
r.Status = event.Action
r.Elapsed = event.Elapsed
finalStatus[key] = r
}
if strings.Contains(event.Test, "/") {
parts := strings.SplitN(event.Test, "/", 2)
parentKey := testKey{pkg: event.Package, test: parts[0]}
hasSubtest[parentKey] = true
}
}
var results []TestResult
for key, r := range finalStatus {
if r.Status == "" {
continue
}
if hasSubtest[key] {
continue
}
results = append(results, r)
}
return results
}
func main() {
cmd := exec.Command("go", "test", "-json", "./...")
output, _ := cmd.CombinedOutput()
results := parseTestOutput(string(output))
passed := 0
failed := 0
skipped := 0
for _, r := range results {
switch r.Status {
case "pass":
passed++
case "fail":
failed++
case "skip":
skipped++
}
}
fmt.Printf("Passed: %d, Failed: %d, Skipped: %d\n", passed, failed, skipped)
if failed > 0 {
fmt.Println("\nFailed tests:")
for _, r := range results {
if r.Status == "fail" {
fmt.Printf(" %s/%s\n", r.Package, r.Test)
if r.Output != "" {
fmt.Printf(" Output:\n")
for _, line := range strings.Split(r.Output, "\n") {
if strings.TrimSpace(line) != "" {
fmt.Printf(" %s\n", line)
}
}
}
}
}
}
}
Now the count matches the actual number of unique tests:
Passed: 4, Failed: 0, Skipped: 0
Each subtest is counted individually. Parent tests with subtests are excluded.
Step 4: Git Integration — Detect Changes
A CI pipeline needs to know when new code is pushed. It watches a git repository and triggers a build when it sees a new commit.
Shell: Read git information
Get the latest commit hash:
git log -1 --format=%H
Get the short hash and commit message:
git log -1 --format="%h %s"
This prints something like a1b2c3d Fix login timeout bug.
See which files changed in the last commit:
git diff --name-only HEAD~1
Get the author:
git log -1 --format="%an <%ae>"
To watch for new commits on a remote, you need to fetch first:
git fetch origin
git log origin/main -1 --format=%H
Compare the local and remote hashes to detect new commits:
LOCAL=$(git log main -1 --format=%H)
REMOTE=$(git log origin/main -1 --format=%H)
if [ "$LOCAL" != "$REMOTE" ]; then
echo "New commits detected"
git log "$LOCAL".."$REMOTE" --oneline
fi
Go: Poll a git repo for new commits
Build a watcher that polls a git repository and detects new commits. Here is the first attempt:
package main
import (
"fmt"
"os/exec"
"strings"
"time"
)
func getLatestCommit(repoPath string) (string, error) {
cmd := exec.Command("git", "log", "-1", "--format=%H")
cmd.Dir = repoPath
output, err := cmd.Output()
if err != nil {
return "", err
}
return strings.TrimSpace(string(output)), nil
}
func getCommitInfo(repoPath string, hash string) (string, string, error) {
cmd := exec.Command("git", "log", "-1", "--format=%s", hash)
cmd.Dir = repoPath
msgOut, err := cmd.Output()
if err != nil {
return "", "", err
}
cmd = exec.Command("git", "log", "-1", "--format=%an", hash)
cmd.Dir = repoPath
authorOut, err := cmd.Output()
if err != nil {
return "", "", err
}
return strings.TrimSpace(string(msgOut)), strings.TrimSpace(string(authorOut)), nil
}
func watchRepo(repoPath string, interval time.Duration, buildFunc func(string)) {
lastCommit, err := getLatestCommit(repoPath)
if err != nil {
fmt.Printf("Error getting initial commit: %v\n", err)
return
}
fmt.Printf("Starting watch. Current commit: %s\n", lastCommit[:8])
for {
time.Sleep(interval)
current, err := getLatestCommit(repoPath)
if err != nil {
fmt.Printf("Error checking commits: %v\n", err)
continue
}
if current != lastCommit {
msg, author, _ := getCommitInfo(repoPath, current)
fmt.Printf("New commit: %s by %s: %s\n", current[:8], author, msg)
buildFunc(current)
lastCommit = current
}
}
}
func main() {
repoPath := "/path/to/your/repo"
watchRepo(repoPath, 10*time.Second, func(commit string) {
fmt.Printf("Building commit %s...\n", commit[:8])
})
}
The bug: Forgetting to fetch
This code calls git log to check the latest commit. But git log only knows about local commits. If someone pushes to the remote, this watcher never sees it. It only detects commits made locally.
Run it and push a commit from another machine. Nothing happens. The watcher keeps polling and finds the same commit every time.
The fix: Fetch before checking
Add a git fetch call before reading the latest commit:
package main
import (
"fmt"
"os/exec"
"strings"
"time"
)
func gitFetch(repoPath string) error {
cmd := exec.Command("git", "fetch", "origin")
cmd.Dir = repoPath
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("fetch failed: %s: %w", string(output), err)
}
return nil
}
func getLatestCommit(repoPath string, branch string) (string, error) {
ref := fmt.Sprintf("origin/%s", branch)
cmd := exec.Command("git", "log", "-1", "--format=%H", ref)
cmd.Dir = repoPath
output, err := cmd.Output()
if err != nil {
return "", err
}
return strings.TrimSpace(string(output)), nil
}
func getCommitInfo(repoPath string, hash string) (string, string, error) {
cmd := exec.Command("git", "log", "-1", "--format=%s", hash)
cmd.Dir = repoPath
msgOut, err := cmd.Output()
if err != nil {
return "", "", err
}
cmd = exec.Command("git", "log", "-1", "--format=%an", hash)
cmd.Dir = repoPath
authorOut, err := cmd.Output()
if err != nil {
return "", "", err
}
return strings.TrimSpace(string(msgOut)), strings.TrimSpace(string(authorOut)), nil
}
func getChangedFiles(repoPath string, oldCommit string, newCommit string) ([]string, error) {
cmd := exec.Command("git", "diff", "--name-only", oldCommit, newCommit)
cmd.Dir = repoPath
output, err := cmd.Output()
if err != nil {
return nil, err
}
lines := strings.Split(strings.TrimSpace(string(output)), "\n")
var files []string
for _, line := range lines {
if line != "" {
files = append(files, line)
}
}
return files, nil
}
func watchRepo(repoPath string, branch string, interval time.Duration, buildFunc func(string)) {
if err := gitFetch(repoPath); err != nil {
fmt.Printf("Initial fetch error: %v\n", err)
return
}
lastCommit, err := getLatestCommit(repoPath, branch)
if err != nil {
fmt.Printf("Error getting initial commit: %v\n", err)
return
}
fmt.Printf("Starting watch on %s. Current commit: %s\n", branch, lastCommit[:8])
for {
time.Sleep(interval)
if err := gitFetch(repoPath); err != nil {
fmt.Printf("Fetch error: %v\n", err)
continue
}
current, err := getLatestCommit(repoPath, branch)
if err != nil {
fmt.Printf("Error checking commits: %v\n", err)
continue
}
if current != lastCommit {
msg, author, _ := getCommitInfo(repoPath, current)
files, _ := getChangedFiles(repoPath, lastCommit, current)
fmt.Printf("New commit: %s by %s: %s\n", current[:8], author, msg)
fmt.Printf("Changed files: %d\n", len(files))
for _, f := range files {
fmt.Printf(" %s\n", f)
}
buildFunc(current)
lastCommit = current
}
}
}
func main() {
repoPath := "/path/to/your/repo"
watchRepo(repoPath, "main", 10*time.Second, func(commit string) {
fmt.Printf("Triggering build for %s...\n", commit[:8])
})
}
The key changes: call git fetch origin before every check, and read from origin/main instead of just main. Now the watcher sees remote commits as soon as they are pushed.
Step 5: Parallel Stage Execution
Some CI stages are independent. Running linters does not depend on building the binary. Running unit tests does not depend on integration tests. Running these in parallel saves time.
Shell: Run commands in parallel
Bash can run commands in the background with &:
#!/bin/bash
set -e
echo "Starting parallel stages"
go test ./... &
PID1=$!
go vet ./... &
PID2=$!
staticcheck ./... &
PID3=$!
wait $PID1
STATUS1=$?
wait $PID2
STATUS2=$?
wait $PID3
STATUS3=$?
echo "Test: exit $STATUS1"
echo "Vet: exit $STATUS2"
echo "Staticcheck: exit $STATUS3"
if [ $STATUS1 -ne 0 ] || [ $STATUS2 -ne 0 ] || [ $STATUS3 -ne 0 ]; then
echo "FAILED"
exit 1
fi
echo "PASSED"
This runs all three commands at the same time. The wait command blocks until a specific background process finishes and sets $? to its exit code.
A simpler but less precise approach:
cmd1 & cmd2 & cmd3 & wait
This waits for all three to finish, but you cannot check individual exit codes.
To capture output from parallel commands, redirect each to a separate file:
go test ./... > test-output.txt 2>&1 &
go vet ./... > vet-output.txt 2>&1 &
wait
Go: Run stages concurrently with goroutines
Build a parallel stage runner in Go. Here is the first attempt:
package main
import (
"fmt"
"os/exec"
"sync"
"time"
)
type Stage struct {
Name string
Command string
Args []string
}
type StageResult struct {
Name string
Output string
ExitCode int
Duration time.Duration
Err error
}
func runStagesParallel(stages []Stage) []StageResult {
results := make([]StageResult, len(stages))
var wg sync.WaitGroup
var outputFile *string
_ = outputFile // suppress unused warning
for i, stage := range stages {
wg.Add(1)
go func(idx int, s Stage) {
defer wg.Done()
start := time.Now()
cmd := exec.Command(s.Command, s.Args...)
output, err := cmd.CombinedOutput()
elapsed := time.Since(start)
result := StageResult{
Name: s.Name,
Output: string(output),
Duration: elapsed,
}
if err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
result.ExitCode = exitErr.ExitCode()
} else {
result.ExitCode = -1
result.Err = err
}
}
results[idx] = result
}(i, stage)
}
wg.Wait()
return results
}
func main() {
stages := []Stage{
{Name: "Unit tests", Command: "go", Args: []string{"test", "./..."}},
{Name: "Vet", Command: "go", Args: []string{"vet", "./..."}},
}
fmt.Println("Running stages in parallel...")
results := runStagesParallel(stages)
for _, r := range results {
status := "PASS"
if r.ExitCode != 0 || r.Err != nil {
status = "FAIL"
}
fmt.Printf("[%s] %s (%s)\n", status, r.Name, r.Duration.Round(time.Millisecond))
}
}
This looks correct. Each goroutine writes to its own index in the results slice, so there is no race on the results. But there is a different problem.
The bug: Shared output file
Imagine two stages both write their output to a shared log file. Or two stages both build to the same output directory. The goroutines run at the same time, and their writes interleave. You get corrupted output.
Here is the problematic version:
package main
import (
"fmt"
"os"
"os/exec"
"sync"
)
func main() {
var wg sync.WaitGroup
logFile, _ := os.Create("pipeline.log")
defer logFile.Close()
commands := []struct {
name string
cmd string
args []string
}{
{"tests", "go", []string{"test", "./..."}},
{"vet", "go", []string{"vet", "./..."}},
}
for _, c := range commands {
wg.Add(1)
go func(name, cmd string, args []string) {
defer wg.Done()
execCmd := exec.Command(cmd, args...)
execCmd.Stdout = logFile
execCmd.Stderr = logFile
execCmd.Run()
}(c.name, c.cmd, c.args)
}
wg.Wait()
fmt.Println("Done. Check pipeline.log")
}
Both goroutines write to logFile at the same time. The file ends up with interleaved lines from both commands. Lines from the test output appear in the middle of vet output. The log is unreadable.
The fix: Separate buffers, merge after completion
Give each stage its own output buffer. Merge the results after all stages complete.
package main
import (
"bytes"
"fmt"
"os/exec"
"sync"
"time"
)
type Stage struct {
Name string
Command string
Args []string
}
type StageResult struct {
Name string
Output string
ExitCode int
Duration time.Duration
Err error
}
func runStagesParallel(stages []Stage) []StageResult {
results := make([]StageResult, len(stages))
var wg sync.WaitGroup
for i, stage := range stages {
wg.Add(1)
go func(idx int, s Stage) {
defer wg.Done()
start := time.Now()
var buf bytes.Buffer
cmd := exec.Command(s.Command, s.Args...)
cmd.Stdout = &buf
cmd.Stderr = &buf
err := cmd.Run()
elapsed := time.Since(start)
result := StageResult{
Name: s.Name,
Output: buf.String(),
Duration: elapsed,
}
if err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
result.ExitCode = exitErr.ExitCode()
} else {
result.ExitCode = -1
result.Err = err
}
}
results[idx] = result
}(i, stage)
}
wg.Wait()
return results
}
func main() {
stages := []Stage{
{Name: "Unit tests", Command: "go", Args: []string{"test", "./..."}},
{Name: "Vet", Command: "go", Args: []string{"vet", "./..."}},
}
fmt.Println("Running stages in parallel...")
start := time.Now()
results := runStagesParallel(stages)
totalDuration := time.Since(start)
allPassed := true
for _, r := range results {
status := "PASS"
if r.ExitCode != 0 || r.Err != nil {
status = "FAIL"
allPassed = false
}
fmt.Printf("\n[%s] %s (%s)\n", status, r.Name, r.Duration.Round(time.Millisecond))
if r.Output != "" {
fmt.Printf("%s\n", r.Output)
}
}
fmt.Printf("\nTotal time: %s\n", totalDuration.Round(time.Millisecond))
if allPassed {
fmt.Println("All stages passed.")
} else {
fmt.Println("Some stages failed.")
}
}
Each bytes.Buffer is owned by one goroutine. No sharing, no race condition. After wg.Wait() returns, all goroutines are done, and you can safely read every result.
Output is clean and separated:
Running stages in parallel...
[PASS] Unit tests (1.234s)
ok example.com/app/math 0.003s
[PASS] Vet (0.456s)
Total time: 1.235s
All stages passed.
The total time is close to the slowest stage, not the sum of all stages. That is the benefit of parallel execution.
Step 6: Complete CI Build Runner
Now combine everything into a single program. This build runner polls a git repository, detects new commits, runs a pipeline of stages (some sequential, some parallel), parses test output, collects artifacts, and prints a summary with durations.
Configuration
Define the pipeline as a list of stages. Each stage has a name, command, and a flag for parallel execution. Stages in the same parallel group run at the same time.
package main
import (
"bufio"
"bytes"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"time"
)
// --- Types ---
type StageConfig struct {
Name string
Command string
Args []string
Parallel string // stages with same non-empty parallel group run together
}
type StageResult struct {
Name string
Output string
ExitCode int
Duration time.Duration
Err error
}
type TestEvent struct {
Action string `json:"Action"`
Package string `json:"Package"`
Test string `json:"Test"`
Output string `json:"Output"`
Elapsed float64 `json:"Elapsed"`
}
type TestSummary struct {
Passed int
Failed int
Skipped int
Failures []string
}
type BuildReport struct {
Commit string
Author string
Message string
Stages []StageResult
Tests *TestSummary
TotalTime time.Duration
Success bool
}
// --- ANSI Colors ---
const (
colorReset = "\033[0m"
colorRed = "\033[31m"
colorGreen = "\033[32m"
colorYellow = "\033[33m"
colorCyan = "\033[36m"
colorBold = "\033[1m"
)
func colorize(color string, text string) string {
return color + text + colorReset
}
// --- Git Functions ---
func gitFetch(repoPath string) error {
cmd := exec.Command("git", "fetch", "origin")
cmd.Dir = repoPath
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("fetch failed: %s: %w", string(output), err)
}
return nil
}
func getLatestCommit(repoPath string, branch string) (string, error) {
ref := fmt.Sprintf("origin/%s", branch)
cmd := exec.Command("git", "log", "-1", "--format=%H", ref)
cmd.Dir = repoPath
output, err := cmd.Output()
if err != nil {
return "", err
}
return strings.TrimSpace(string(output)), nil
}
func getCommitInfo(repoPath string, hash string) (string, string, error) {
cmd := exec.Command("git", "log", "-1", "--format=%s", hash)
cmd.Dir = repoPath
msgOut, err := cmd.Output()
if err != nil {
return "", "", err
}
cmd = exec.Command("git", "log", "-1", "--format=%an", hash)
cmd.Dir = repoPath
authorOut, err := cmd.Output()
if err != nil {
return "", "", err
}
return strings.TrimSpace(string(msgOut)), strings.TrimSpace(string(authorOut)), nil
}
// --- Stage Execution ---
func runOneStage(s StageConfig, workDir string) StageResult {
start := time.Now()
var buf bytes.Buffer
cmd := exec.Command(s.Command, s.Args...)
cmd.Dir = workDir
cmd.Stdout = &buf
cmd.Stderr = &buf
err := cmd.Run()
elapsed := time.Since(start)
result := StageResult{
Name: s.Name,
Output: buf.String(),
Duration: elapsed,
}
if err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
result.ExitCode = exitErr.ExitCode()
} else {
result.ExitCode = -1
result.Err = err
}
}
return result
}
func runParallelGroup(stages []StageConfig, workDir string) []StageResult {
results := make([]StageResult, len(stages))
var wg sync.WaitGroup
for i, stage := range stages {
wg.Add(1)
go func(idx int, s StageConfig) {
defer wg.Done()
results[idx] = runOneStage(s, workDir)
}(i, stage)
}
wg.Wait()
return results
}
func runPipeline(stages []StageConfig, workDir string) []StageResult {
var allResults []StageResult
i := 0
for i < len(stages) {
if stages[i].Parallel != "" {
// collect all stages in the same parallel group
group := stages[i].Parallel
var parallelStages []StageConfig
for i < len(stages) && stages[i].Parallel == group {
parallelStages = append(parallelStages, stages[i])
i++
}
names := make([]string, len(parallelStages))
for j, s := range parallelStages {
names[j] = s.Name
}
fmt.Printf("%s Running parallel: %s\n",
colorize(colorCyan, ">>>"),
strings.Join(names, ", "))
results := runParallelGroup(parallelStages, workDir)
allResults = append(allResults, results...)
// print results for this group
groupFailed := false
for _, r := range results {
printStageResult(r)
if r.ExitCode != 0 || r.Err != nil {
groupFailed = true
}
}
if groupFailed {
fmt.Printf("%s Parallel group failed. Stopping.\n",
colorize(colorRed, "!!!"))
break
}
} else {
stage := stages[i]
i++
fmt.Printf("%s Running: %s\n",
colorize(colorCyan, ">>>"), stage.Name)
result := runOneStage(stage, workDir)
allResults = append(allResults, result)
printStageResult(result)
if result.ExitCode != 0 || result.Err != nil {
fmt.Printf("%s Stage failed. Stopping.\n",
colorize(colorRed, "!!!"))
break
}
}
}
return allResults
}
func printStageResult(r StageResult) {
if r.ExitCode == 0 && r.Err == nil {
fmt.Printf(" %s %s (%s)\n",
colorize(colorGreen, "PASS"),
r.Name,
r.Duration.Round(time.Millisecond))
} else {
fmt.Printf(" %s %s (%s)\n",
colorize(colorRed, "FAIL"),
r.Name,
r.Duration.Round(time.Millisecond))
if r.Output != "" {
lines := strings.Split(r.Output, "\n")
limit := 20
if len(lines) < limit {
limit = len(lines)
}
for _, line := range lines[:limit] {
fmt.Printf(" %s\n", line)
}
if len(lines) > 20 {
fmt.Printf(" ... (%d more lines)\n", len(lines)-20)
}
}
}
}
// --- Test Output Parsing ---
func parseTestOutput(data string) TestSummary {
type testKey struct {
pkg string
test string
}
type testInfo struct {
status string
output string
}
finalStatus := make(map[testKey]testInfo)
hasSubtest := make(map[testKey]bool)
scanner := bufio.NewScanner(strings.NewReader(data))
for scanner.Scan() {
var event TestEvent
if err := json.Unmarshal(scanner.Bytes(), &event); err != nil {
continue
}
if event.Test == "" {
continue
}
key := testKey{pkg: event.Package, test: event.Test}
if event.Action == "output" {
info := finalStatus[key]
info.output += event.Output
finalStatus[key] = info
}
if event.Action == "pass" || event.Action == "fail" || event.Action == "skip" {
info := finalStatus[key]
info.status = event.Action
finalStatus[key] = info
}
if strings.Contains(event.Test, "/") {
parts := strings.SplitN(event.Test, "/", 2)
parentKey := testKey{pkg: event.Package, test: parts[0]}
hasSubtest[parentKey] = true
}
}
var summary TestSummary
for key, info := range finalStatus {
if info.status == "" {
continue
}
if hasSubtest[key] {
continue
}
switch info.status {
case "pass":
summary.Passed++
case "fail":
summary.Failed++
summary.Failures = append(summary.Failures,
fmt.Sprintf("%s/%s", key.pkg, key.test))
case "skip":
summary.Skipped++
}
}
return summary
}
// --- Build Report ---
func printReport(report BuildReport) {
fmt.Printf("\n%s\n", colorize(colorBold, "========== Build Report =========="))
fmt.Printf("Commit: %s\n", report.Commit)
fmt.Printf("Author: %s\n", report.Author)
fmt.Printf("Message: %s\n", report.Message)
fmt.Printf("Time: %s\n", report.TotalTime.Round(time.Millisecond))
fmt.Println()
fmt.Printf("%s\n", colorize(colorBold, "Stages:"))
for _, r := range report.Stages {
status := colorize(colorGreen, "PASS")
if r.ExitCode != 0 || r.Err != nil {
status = colorize(colorRed, "FAIL")
}
fmt.Printf(" [%s] %-20s %s\n", status, r.Name, r.Duration.Round(time.Millisecond))
}
if report.Tests != nil {
fmt.Println()
fmt.Printf("%s\n", colorize(colorBold, "Tests:"))
fmt.Printf(" Passed: %s\n", colorize(colorGreen, fmt.Sprintf("%d", report.Tests.Passed)))
fmt.Printf(" Failed: %s\n", colorize(colorRed, fmt.Sprintf("%d", report.Tests.Failed)))
fmt.Printf(" Skipped: %s\n", colorize(colorYellow, fmt.Sprintf("%d", report.Tests.Skipped)))
if len(report.Tests.Failures) > 0 {
fmt.Println()
fmt.Printf(" %s\n", colorize(colorRed, "Failed tests:"))
for _, name := range report.Tests.Failures {
fmt.Printf(" - %s\n", name)
}
}
}
fmt.Println()
if report.Success {
fmt.Printf("Result: %s\n", colorize(colorGreen, "PIPELINE PASSED"))
} else {
fmt.Printf("Result: %s\n", colorize(colorRed, "PIPELINE FAILED"))
}
fmt.Println(colorize(colorBold, "=================================="))
}
// --- Main ---
func buildForCommit(repoPath string, commit string, stages []StageConfig) BuildReport {
msg, author, _ := getCommitInfo(repoPath, commit)
short := commit
if len(commit) > 8 {
short = commit[:8]
}
fmt.Printf("\n%s Build started for %s\n", colorize(colorBold, "==>"), short)
fmt.Printf(" Author: %s\n", author)
fmt.Printf(" Message: %s\n", msg)
fmt.Println()
start := time.Now()
results := runPipeline(stages, repoPath)
totalTime := time.Since(start)
success := true
var testSummary *TestSummary
for _, r := range results {
if r.ExitCode != 0 || r.Err != nil {
success = false
}
if r.Name == "Run tests" && strings.Contains(strings.Join(stages[0].Args, " "), "-json") {
// find the test stage and parse its output
}
}
// find test stage output and parse it
for _, r := range results {
// look for JSON test output in any stage
if strings.Contains(r.Output, `"Action"`) && strings.Contains(r.Output, `"Package"`) {
summary := parseTestOutput(r.Output)
testSummary = &summary
break
}
}
report := BuildReport{
Commit: short,
Author: author,
Message: msg,
Stages: results,
Tests: testSummary,
TotalTime: totalTime,
Success: success,
}
printReport(report)
return report
}
func main() {
if len(os.Args) < 2 {
fmt.Println("Usage: ci-runner <repo-path> [branch]")
fmt.Println(" Watches a git repo and runs CI stages on new commits.")
os.Exit(1)
}
repoPath, err := filepath.Abs(os.Args[1])
if err != nil {
fmt.Printf("Invalid path: %v\n", err)
os.Exit(1)
}
branch := "main"
if len(os.Args) > 2 {
branch = os.Args[2]
}
stages := []StageConfig{
{
Name: "Run tests",
Command: "go",
Args: []string{"test", "-json", "./..."},
},
{
Name: "Vet",
Command: "go",
Args: []string{"vet", "./..."},
Parallel: "checks",
},
{
Name: "Format check",
Command: "gofmt",
Args: []string{"-l", "."},
Parallel: "checks",
},
{
Name: "Build binary",
Command: "go",
Args: []string{"build", "-o", "./dist/app", "./cmd/server"},
},
}
fmt.Printf("%s CI Runner starting\n", colorize(colorBold, "==>"))
fmt.Printf(" Repo: %s\n", repoPath)
fmt.Printf(" Branch: %s\n", branch)
fmt.Printf(" Stages: %d\n", len(stages))
fmt.Println()
// initial fetch
if err := gitFetch(repoPath); err != nil {
fmt.Printf("Fetch error: %v\n", err)
os.Exit(1)
}
lastCommit, err := getLatestCommit(repoPath, branch)
if err != nil {
fmt.Printf("Cannot read latest commit: %v\n", err)
os.Exit(1)
}
fmt.Printf("Current commit: %s\n", lastCommit[:8])
fmt.Printf("Polling every 10 seconds...\n\n")
// run an initial build
buildForCommit(repoPath, lastCommit, stages)
// poll for changes
for {
time.Sleep(10 * time.Second)
if err := gitFetch(repoPath); err != nil {
fmt.Printf("Fetch error: %v\n", err)
continue
}
current, err := getLatestCommit(repoPath, branch)
if err != nil {
fmt.Printf("Error: %v\n", err)
continue
}
if current != lastCommit {
buildForCommit(repoPath, current, stages)
lastCommit = current
}
}
}
How it works
The runner starts by fetching the latest commit from the remote. It runs an initial build, then enters a polling loop.
Each build runs the configured stages in order. When consecutive stages share the same Parallel group, they run at the same time using goroutines. If any stage fails, the pipeline stops.
The test stage output is parsed to extract pass/fail/skip counts. The final report shows each stage with its duration and status, followed by test results and an overall verdict.
Running the build runner
Build and run it:
go build -o ci-runner .
./ci-runner /path/to/your/go/project main
You see output like this:
==> CI Runner starting
Repo: /home/user/myproject
Branch: main
Stages: 4
Current commit: a1b2c3d4
Polling every 10 seconds...
==> Build started for a1b2c3d4
Author: Jane Doe
Message: Add user validation
>>> Running: Run tests
PASS Run tests (1.456s)
>>> Running parallel: Vet, Format check
PASS Vet (0.678s)
PASS Format check (0.045s)
>>> Running: Build binary
PASS Build binary (2.345s)
========== Build Report ==========
Commit: a1b2c3d4
Author: Jane Doe
Message: Add user validation
Time: 4.524s
Stages:
[PASS] Run tests 1.456s
[PASS] Vet 678ms
[PASS] Format check 45ms
[PASS] Build binary 2.345s
Tests:
Passed: 12
Failed: 0
Skipped: 1
Result: PIPELINE PASSED
==================================
When a test fails, the report changes:
========== Build Report ==========
Commit: b2c3d4e5
Author: John Smith
Message: Refactor auth module
Stages:
[FAIL] Run tests 0.892s
Tests:
Passed: 10
Failed: 2
Skipped: 1
Failed tests:
- example.com/app/auth/TestValidateToken
- example.com/app/auth/TestRefreshToken
Result: PIPELINE FAILED
==================================
The pipeline stopped after the test stage. Vet, format check, and build never ran.
What to add next
This runner handles the core CI workflow. Here are things you could add:
- Webhook server: Replace polling with an HTTP endpoint that receives push events. Faster response, less wasted network calls.
- Build artifact storage: Copy the built binary and checksum to a timestamped directory. Keep the last N builds.
- Notifications: Send results to a Slack webhook or email. Include the failed test names and commit message.
- Timeout per stage: Kill a stage if it runs longer than a configured duration. Use
context.WithTimeoutwithexec.CommandContext. - Log persistence: Write each build’s output to a file named by commit hash. Makes debugging old failures possible.
None of these require external libraries. The standard library has everything you need.
Key Takeaways
A CI pipeline is a sequence of commands that run automatically when code changes. The core operations are simple: run a command, check the exit code, capture the output.
Shell scripts handle these operations directly. set -e stops on failure. set -o pipefail catches errors in pipes. Background processes with & and wait handle parallelism.
Go provides more structure. os/exec runs commands. sync.WaitGroup manages goroutines. encoding/json parses test output. bytes.Buffer captures output without race conditions.
The bugs we hit are the same ones that cause problems in any CI system: missing stderr, no error propagation between steps, forgetting to fetch before checking for changes, and race conditions with shared output. Understanding these patterns helps you debug CI failures regardless of which tool you use.
References and Further Reading
- Donovan, A. & Kernighan, B. (2015). The Go Programming Language. Addison-Wesley.
- Cox, R. (2019). Go Blog: Using Go Modules. The Go Blog.
- Go Team. (2024). os/exec package documentation. Go Standard Library.
- Go Team. (2024). go test command documentation. Go Standard Library.
- Humble, J. & Farley, D. (2010). Continuous Delivery. Addison-Wesley.
Keep Reading
- Git Hooks and Automation: From Shell Hooks to a Go Webhook Server — trigger the build runner you built here automatically when code is pushed.
- Deployment Automation: From SSH Scripts to a Go Deploy Tool — deploy the artifacts your CI pipeline produces with health checks and rollbacks.
- Task Automation: From Cron and Make to a Go Task Runner — schedule builds with cron and manage task dependencies.
What part of your CI pipeline took the longest to debug, and what was the root cause?