/user/kayd @ devops :~$ cat advanced-aws-social-network-architecture.md

Building a URL Shortener: From Linux Networking to Go Building a URL Shortener: From Linux Networking to Go

Karandeep Singh

Aug 28, 2023 • 32 minutes

Summary

Build a URL shortener step by step with Linux networking and Go. From TCP sockets and HTTP redirects to a complete web service with storage, analytics, and rate limiting.

A URL shortener takes a long URL and gives back a short one. When someone visits the short URL, the server redirects them to the original. That is the whole idea. But building one teaches you TCP connections, HTTP headers, redirects, persistent storage, and analytics.

We will build this step by step. Each step starts with Linux commands so you can see the concept in your terminal. Then we write the Go equivalent. Along the way, we will make mistakes on purpose, see what breaks, and fix it.

You need a Linux terminal and Go installed. Nothing else.

Step 1: TCP Connections and HTTP Basics

Before writing any Go code, we need to understand what happens when a browser talks to a server. It all starts with TCP.

Linux: Listen on a Port

Open two terminal windows. In the first one, start a listener on port 8080.

nc -l 8080

This tells netcat to listen for a TCP connection on port 8080. It will wait until someone connects.

In the second terminal, send an HTTP request.

curl -v http://localhost:8080/test

Now look at the first terminal. You will see something like this.

GET /test HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.81.0
Accept: */*

That is a raw HTTP request. It is plain text sent over a TCP connection. The first line is the request line: the method (GET), the path (/test), and the protocol version. The lines after that are headers. The empty line at the end marks the end of the headers.

Type a response in the netcat terminal and press Enter.

HTTP/1.1 200 OK
Content-Type: text/plain

Hello from netcat

Press Ctrl+C to close the connection. In the curl terminal, you will see your response appear. HTTP is just text over TCP. Nothing magical.

Go: Build a TCP Server

Now let us do the same thing in Go. We will listen on a TCP port and read the raw bytes that arrive.

package main

import (
	"fmt"
	"net"
	"os"
)

func main() {
	listener, err := net.Listen("tcp", ":8080")
	if err != nil {
		fmt.Println("Error starting server:", err)
		os.Exit(1)
	}
	defer listener.Close()
	fmt.Println("Listening on :8080")

	for {
		conn, err := listener.Accept()
		if err != nil {
			fmt.Println("Error accepting connection:", err)
			continue
		}
		go handleConnection(conn)
	}
}

func handleConnection(conn net.Conn) {
	defer conn.Close()

	buf := make([]byte, 1024)
	n, err := conn.Read(buf)
	if err != nil {
		fmt.Println("Error reading:", err)
		return
	}

	fmt.Println("--- Received ---")
	fmt.Println(string(buf[:n]))

	response := "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\nHello from Go TCP server\n"
	conn.Write([]byte(response))
}

Save this as main.go and run it.

go run main.go

In another terminal, test it.

curl -v http://localhost:8080/test

You will see the raw HTTP request printed in the server terminal, and curl will show the response. This is the same thing netcat did, but now your Go program controls everything.

The Bug: Missing POST Body

Try sending a POST request with a body.

curl -X POST -d '{"url":"https://example.com"}' http://localhost:8080/shorten

The server prints the request, but where is the body? Sometimes you see it. Sometimes you do not. The problem is our buffer. We read only 1024 bytes in a single Read call. The TCP stack might deliver the headers and body in separate packets. Our single Read call might only get the headers.

Look at the output carefully. You will see a Content-Length header in the request.

POST /shorten HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.81.0
Accept: */*
Content-Length: 28
Content-Type: application/x-www-form-urlencoded

The body is 28 bytes. But our code did not wait for it.

The Fix: Read the Content-Length

We need to parse the headers, find Content-Length, and then read exactly that many bytes for the body.

package main

import (
	"bufio"
	"fmt"
	"net"
	"os"
	"strconv"
	"strings"
)

func main() {
	listener, err := net.Listen("tcp", ":8080")
	if err != nil {
		fmt.Println("Error starting server:", err)
		os.Exit(1)
	}
	defer listener.Close()
	fmt.Println("Listening on :8080")

	for {
		conn, err := listener.Accept()
		if err != nil {
			fmt.Println("Error accepting connection:", err)
			continue
		}
		go handleConnection(conn)
	}
}

func handleConnection(conn net.Conn) {
	defer conn.Close()

	reader := bufio.NewReader(conn)

	// Read the request line
	requestLine, err := reader.ReadString('\n')
	if err != nil {
		fmt.Println("Error reading request line:", err)
		return
	}
	fmt.Print("Request: ", requestLine)

	// Read headers
	contentLength := 0
	for {
		line, err := reader.ReadString('\n')
		if err != nil {
			fmt.Println("Error reading header:", err)
			return
		}
		line = strings.TrimSpace(line)
		if line == "" {
			break // Empty line means end of headers
		}
		fmt.Println("Header:", line)

		if strings.HasPrefix(strings.ToLower(line), "content-length:") {
			parts := strings.SplitN(line, ":", 2)
			contentLength, _ = strconv.Atoi(strings.TrimSpace(parts[1]))
		}
	}

	// Read the body if there is one
	if contentLength > 0 {
		body := make([]byte, contentLength)
		bytesRead := 0
		for bytesRead < contentLength {
			n, err := reader.Read(body[bytesRead:])
			if err != nil {
				fmt.Println("Error reading body:", err)
				return
			}
			bytesRead += n
		}
		fmt.Println("Body:", string(body))
	}

	response := "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\nReceived your request\n"
	conn.Write([]byte(response))
}

Now run this version and send the POST request again.

curl -X POST -d '{"url":"https://example.com"}' http://localhost:8080/shorten

This time the server prints the body correctly. The key change: we use a buffered reader, parse headers line by line, find the Content-Length value, and then read exactly that many bytes. No more missing data.

This is how HTTP works underneath. Every web framework does this same parsing. Now that you understand it, we can move to the standard library and let it handle the details.

Step 2: HTTP Server with Routing

Now we need a real HTTP server with proper routing. Our URL shortener needs two endpoints: one to create short URLs and one to redirect.

Linux: Test HTTP Methods

Before writing the server, let us understand the HTTP methods we will use.

Create a short URL.

curl -X POST -d '{"url":"https://example.com"}' http://localhost:8080/shorten

Check a redirect without following it.

curl -I http://localhost:8080/abc123

The -I flag sends a HEAD request. You will see the response headers without the body. This is useful for checking if a redirect is set up correctly.

Follow the redirect.

curl -L http://localhost:8080/abc123

The -L flag tells curl to follow Location headers automatically. This is what browsers do.

Go: Two Routes with net/http

Let us build a proper HTTP server using the standard library.

package main

import (
	"encoding/json"
	"fmt"
	"net/http"
	"strings"
	"sync"
)

type URLStore struct {
	mu   sync.RWMutex
	urls map[string]string // short code -> original URL
}

var store = &URLStore{
	urls: make(map[string]string),
}

type ShortenRequest struct {
	URL string `json:"url"`
}

type ShortenResponse struct {
	ShortURL string `json:"short_url"`
	Code     string `json:"code"`
}

func main() {
	http.HandleFunc("/", handler)

	fmt.Println("Server running on :8080")
	err := http.ListenAndServe(":8080", nil)
	if err != nil {
		fmt.Println("Error:", err)
	}
}

func handler(w http.ResponseWriter, r *http.Request) {
	if r.URL.Path == "/shorten" && r.Method == http.MethodPost {
		handleShorten(w, r)
		return
	}

	code := strings.TrimPrefix(r.URL.Path, "/")
	if code != "" {
		handleRedirect(w, r, code)
		return
	}

	w.WriteHeader(http.StatusOK)
	fmt.Fprintf(w, "URL Shortener is running\n")
}

func handleShorten(w http.ResponseWriter, r *http.Request) {
	var req ShortenRequest
	err := json.NewDecoder(r.Body).Decode(&req)
	if err != nil || req.URL == "" {
		http.Error(w, "Invalid request. Send JSON with a 'url' field.", http.StatusBadRequest)
		return
	}

	code := "abc123" // Hardcoded for now. We fix this in Step 3.

	store.mu.Lock()
	store.urls[code] = req.URL
	store.mu.Unlock()

	resp := ShortenResponse{
		ShortURL: fmt.Sprintf("http://localhost:8080/%s", code),
		Code:     code,
	}

	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(resp)
}

func handleRedirect(w http.ResponseWriter, r *http.Request, code string) {
	store.mu.RLock()
	originalURL, exists := store.urls[code]
	store.mu.RUnlock()

	if !exists {
		http.Error(w, "Short URL not found", http.StatusNotFound)
		return
	}

	http.Redirect(w, r, originalURL, http.StatusFound)
}

Run it and test.

go run main.go

Create a short URL.

curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' \
  http://localhost:8080/shorten

You get back something like this.

{"short_url":"http://localhost:8080/abc123","code":"abc123"}

Now test the redirect.

curl -I http://localhost:8080/abc123

HTTP/1.1 302 Found
Location: https://example.com

The server sends back a 302 status with a Location header pointing to the original URL.

The Bug: No 404 for Unknown Paths

Try requesting a path that is not a short code and not /shorten.

curl -v http://localhost:8080/favicon.ico

HTTP/1.1 404 Not Found

That looks right. But try this.

curl -v http://localhost:8080/shorten/extra

The handler receives this request because the pattern / matches everything. The path /shorten/extra does not match /shorten exactly, so it falls through to the redirect logic. The code extracted is shorten/extra. That code does not exist, so you get a 404.

This seems fine on the surface, but it means any path that is not /shorten exactly will be treated as a short code lookup. What if someone requests /shorten?url=test with a GET method? It falls through to the redirect handler and tries to look up shorten as a code.

curl -v "http://localhost:8080/shorten?url=test"

HTTP/1.1 404 Not Found
Short URL not found

That is confusing. The user tried to shorten a URL using the wrong method, but got “Short URL not found” instead of a helpful error.

The Fix: Explicit Route Checking

func handler(w http.ResponseWriter, r *http.Request) {
	path := r.URL.Path

	if path == "/shorten" {
		if r.Method != http.MethodPost {
			http.Error(w, "Method not allowed. Use POST.", http.StatusMethodNotAllowed)
			return
		}
		handleShorten(w, r)
		return
	}

	if path == "/" {
		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "URL Shortener is running\n")
		return
	}

	// Only treat single-segment paths as short codes
	code := strings.TrimPrefix(path, "/")
	if strings.Contains(code, "/") {
		http.NotFound(w, r)
		return
	}

	handleRedirect(w, r, code)
}

Now test the edge cases.

curl -v "http://localhost:8080/shorten?url=test"

HTTP/1.1 405 Method Not Allowed
Method not allowed. Use POST.

curl -v http://localhost:8080/shorten/extra

HTTP/1.1 404 Not Found
404 page not found

Each path gets a clear and correct response. The server only treats single-segment paths as short code lookups. Multi-segment paths get a 404. GET requests to /shorten get a 405 Method Not Allowed.

Step 3: Generating Short Codes

Right now every URL gets the code abc123. That is not useful. We need unique short codes.

Linux: Two Ways to Generate Codes

Hash-based. Take the URL, hash it, use the first few characters.

echo -n "https://example.com" | sha256sum | cut -c1-8

f0e6a6a3

The same URL always produces the same hash. That means the same URL always gets the same short code. This is useful if you want to avoid duplicates.

Random. Pull bytes from the system random source.

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 8

kQ7mBx2p

Every run gives a different result. This is useful if you want different short codes even for the same URL.

Try running the hash command twice with the same URL.

echo -n "https://example.com" | sha256sum | cut -c1-8
echo -n "https://example.com" | sha256sum | cut -c1-8

You get f0e6a6a3 both times. Now run the random command twice.

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 8 && echo
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 8 && echo

Two different results. Each approach has tradeoffs. We will start with hash-based.

Go: Hash-Based Code Generation

package main

import (
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"fmt"
	"net/http"
	"strings"
	"sync"
)

type URLStore struct {
	mu   sync.RWMutex
	urls map[string]string // short code -> original URL
}

var store = &URLStore{
	urls: make(map[string]string),
}

type ShortenRequest struct {
	URL string `json:"url"`
}

type ShortenResponse struct {
	ShortURL string `json:"short_url"`
	Code     string `json:"code"`
}

func generateCode(url string) string {
	hash := sha256.Sum256([]byte(url))
	return hex.EncodeToString(hash[:])
}

func main() {
	http.HandleFunc("/", handler)
	fmt.Println("Server running on :8080")
	http.ListenAndServe(":8080", nil)
}

func handler(w http.ResponseWriter, r *http.Request) {
	path := r.URL.Path

	if path == "/shorten" {
		if r.Method != http.MethodPost {
			http.Error(w, "Method not allowed. Use POST.", http.StatusMethodNotAllowed)
			return
		}
		handleShorten(w, r)
		return
	}

	if path == "/" {
		fmt.Fprintf(w, "URL Shortener is running\n")
		return
	}

	code := strings.TrimPrefix(path, "/")
	if strings.Contains(code, "/") {
		http.NotFound(w, r)
		return
	}

	handleRedirect(w, r, code)
}

func handleShorten(w http.ResponseWriter, r *http.Request) {
	var req ShortenRequest
	err := json.NewDecoder(r.Body).Decode(&req)
	if err != nil || req.URL == "" {
		http.Error(w, "Invalid request. Send JSON with a 'url' field.", http.StatusBadRequest)
		return
	}

	code := generateCode(req.URL)

	store.mu.Lock()
	store.urls[code] = req.URL
	store.mu.Unlock()

	resp := ShortenResponse{
		ShortURL: fmt.Sprintf("http://localhost:8080/%s", code),
		Code:     code,
	}

	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(resp)
}

func handleRedirect(w http.ResponseWriter, r *http.Request, code string) {
	store.mu.RLock()
	originalURL, exists := store.urls[code]
	store.mu.RUnlock()

	if !exists {
		http.Error(w, "Short URL not found", http.StatusNotFound)
		return
	}

	http.Redirect(w, r, originalURL, http.StatusFound)
}

Test it.

curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' \
  http://localhost:8080/shorten

{"short_url":"http://localhost:8080/f0e6a6a3a0d6...","code":"f0e6a6a3a0d6..."}

The Bug: Codes Are Too Long

The code is the full SHA256 hash. That is 64 hexadecimal characters. The whole point of a URL shortener is to make URLs short. A 64-character code defeats the purpose.

http://localhost:8080/f0e6a6a3a0d6b6c1e2d3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5

That is longer than most original URLs.

The Fix: Truncate and Handle Collisions

Take only the first 7 characters of the hash. But now two different URLs might produce the same 7-character prefix. We need to check for collisions.

func generateCode(url string) string {
	hash := sha256.Sum256([]byte(url))
	fullHash := hex.EncodeToString(hash[:])
	return fullHash[:7]
}

func handleShorten(w http.ResponseWriter, r *http.Request) {
	var req ShortenRequest
	err := json.NewDecoder(r.Body).Decode(&req)
	if err != nil || req.URL == "" {
		http.Error(w, "Invalid request. Send JSON with a 'url' field.", http.StatusBadRequest)
		return
	}

	code := generateCode(req.URL)

	store.mu.Lock()
	// Check for collision: same code but different URL
	if existingURL, exists := store.urls[code]; exists && existingURL != req.URL {
		// Collision. Try longer prefixes until we find a unique one.
		hash := sha256.Sum256([]byte(req.URL))
		fullHash := hex.EncodeToString(hash[:])
		for length := 8; length <= len(fullHash); length++ {
			code = fullHash[:length]
			if existing, exists := store.urls[code]; !exists || existing == req.URL {
				break
			}
		}
	}
	store.urls[code] = req.URL
	store.mu.Unlock()

	resp := ShortenResponse{
		ShortURL: fmt.Sprintf("http://localhost:8080/%s", code),
		Code:     code,
	}

	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(resp)
}

Now the same URL always returns the same short code. Different URLs get different codes. If two URLs happen to share the same 7-character prefix, the code grows longer until it is unique.

Test it.

curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' \
  http://localhost:8080/shorten

{"short_url":"http://localhost:8080/f0e6a6a","code":"f0e6a6a"}

Seven characters. Short and clean.

Submit the same URL again.

curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' \
  http://localhost:8080/shorten

Same code: f0e6a6a. No duplicates.

Step 4: Storing URLs Persistently

Right now all URLs are stored in memory. Restart the server and they are gone. We need persistent storage.

Linux: File-Based Storage

The simplest storage is a text file. One line per URL mapping.

echo "f0e6a6a https://example.com" >> urls.txt
echo "b3c8d9e https://golang.org" >> urls.txt

Look up a code.

grep "^f0e6a6a " urls.txt | cut -d' ' -f2-

https://example.com

The ^ anchors the match to the start of the line. The space after the code ensures we match the whole code and not a prefix. The cut command extracts everything after the first space.

Check how fast this is with many entries. Generate 10,000 lines.

for i in $(seq 1 10000); do
  echo "code${i} https://example.com/${i}" >> urls_big.txt
done

Now search for the last entry.

time grep "^code10000 " urls_big.txt | cut -d' ' -f2-

It is fast for 10,000 entries. But grep scans the entire file from top to bottom. With millions of entries, every lookup reads the whole file. That gets slow.

Go: File-Based Store with In-Memory Lookup

We will write URLs to a file for persistence but keep them in a map in memory for fast lookups. On startup, we load the file into the map.

package main

import (
	"bufio"
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"fmt"
	"net/http"
	"os"
	"strings"
	"sync"
)

const dataFile = "urls.txt"

type URLStore struct {
	mu   sync.RWMutex
	urls map[string]string
	file *os.File
}

type ShortenRequest struct {
	URL string `json:"url"`
}

type ShortenResponse struct {
	ShortURL string `json:"short_url"`
	Code     string `json:"code"`
}

var store *URLStore

func newURLStore(filename string) (*URLStore, error) {
	s := &URLStore{
		urls: make(map[string]string),
	}

	// Load existing URLs from file
	if _, err := os.Stat(filename); err == nil {
		file, err := os.Open(filename)
		if err != nil {
			return nil, fmt.Errorf("cannot open data file: %w", err)
		}

		scanner := bufio.NewScanner(file)
		count := 0
		for scanner.Scan() {
			line := scanner.Text()
			parts := strings.SplitN(line, " ", 2)
			if len(parts) == 2 {
				s.urls[parts[0]] = parts[1]
				count++
			}
		}
		file.Close()

		if err := scanner.Err(); err != nil {
			return nil, fmt.Errorf("error reading data file: %w", err)
		}
		fmt.Printf("Loaded %d URLs from %s\n", count, filename)
	}

	// Open file for appending new URLs
	file, err := os.OpenFile(filename, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
	if err != nil {
		return nil, fmt.Errorf("cannot open data file for writing: %w", err)
	}
	s.file = file

	return s, nil
}

func (s *URLStore) Save(code, url string) error {
	s.mu.Lock()
	defer s.mu.Unlock()

	s.urls[code] = url
	_, err := fmt.Fprintf(s.file, "%s %s\n", code, url)
	if err != nil {
		return fmt.Errorf("failed to write to file: %w", err)
	}
	return s.file.Sync()
}

func (s *URLStore) Get(code string) (string, bool) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	url, exists := s.urls[code]
	return url, exists
}

func (s *URLStore) Exists(code string) (string, bool) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	url, exists := s.urls[code]
	return url, exists
}

func generateCode(url string) string {
	hash := sha256.Sum256([]byte(url))
	fullHash := hex.EncodeToString(hash[:])
	return fullHash[:7]
}

func main() {
	var err error
	store, err = newURLStore(dataFile)
	if err != nil {
		fmt.Println("Error initializing store:", err)
		os.Exit(1)
	}
	defer store.file.Close()

	http.HandleFunc("/", handler)
	fmt.Println("Server running on :8080")
	err = http.ListenAndServe(":8080", nil)
	if err != nil {
		fmt.Println("Error:", err)
	}
}

func handler(w http.ResponseWriter, r *http.Request) {
	path := r.URL.Path

	if path == "/shorten" {
		if r.Method != http.MethodPost {
			http.Error(w, "Method not allowed. Use POST.", http.StatusMethodNotAllowed)
			return
		}
		handleShorten(w, r)
		return
	}

	if path == "/" {
		fmt.Fprintf(w, "URL Shortener is running\n")
		return
	}

	code := strings.TrimPrefix(path, "/")
	if strings.Contains(code, "/") {
		http.NotFound(w, r)
		return
	}

	handleRedirect(w, r, code)
}

func handleShorten(w http.ResponseWriter, r *http.Request) {
	var req ShortenRequest
	err := json.NewDecoder(r.Body).Decode(&req)
	if err != nil || req.URL == "" {
		http.Error(w, "Invalid request. Send JSON with a 'url' field.", http.StatusBadRequest)
		return
	}

	code := generateCode(req.URL)

	// Check for collision
	if existingURL, exists := store.Exists(code); exists && existingURL != req.URL {
		hash := sha256.Sum256([]byte(req.URL))
		fullHash := hex.EncodeToString(hash[:])
		for length := 8; length <= len(fullHash); length++ {
			code = fullHash[:length]
			if existing, exists := store.Exists(code); !exists || existing == req.URL {
				break
			}
		}
	}

	// Check if this URL was already shortened
	if existingURL, exists := store.Exists(code); exists && existingURL == req.URL {
		resp := ShortenResponse{
			ShortURL: fmt.Sprintf("http://localhost:8080/%s", code),
			Code:     code,
		}
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(resp)
		return
	}

	err = store.Save(code, req.URL)
	if err != nil {
		http.Error(w, "Failed to save URL", http.StatusInternalServerError)
		return
	}

	resp := ShortenResponse{
		ShortURL: fmt.Sprintf("http://localhost:8080/%s", code),
		Code:     code,
	}

	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(resp)
}

func handleRedirect(w http.ResponseWriter, r *http.Request, code string) {
	originalURL, exists := store.Get(code)
	if !exists {
		http.Error(w, "Short URL not found", http.StatusNotFound)
		return
	}

	http.Redirect(w, r, originalURL, http.StatusFound)
}

Run it and create some URLs.

go run main.go

curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' \
  http://localhost:8080/shorten

curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://golang.org"}' \
  http://localhost:8080/shorten

Check the data file.

cat urls.txt

f0e6a6a https://example.com
7e0e78c https://golang.org

Now stop the server with Ctrl+C and start it again.

go run main.go

Loaded 2 URLs from urls.txt
Server running on :8080

Test that the old codes still work.

curl -I http://localhost:8080/f0e6a6a

HTTP/1.1 302 Found
Location: https://example.com

The URLs survived the restart. The file gives us persistence. The map gives us fast lookups.

The Bug: Reading the Whole File on Every Lookup

Look at how we would have done it without the in-memory map. A naive approach might read the file on every request.

// BAD: Do not do this
func getURLFromFile(code string) (string, bool) {
	file, err := os.Open(dataFile)
	if err != nil {
		return "", false
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	for scanner.Scan() {
		parts := strings.SplitN(scanner.Text(), " ", 2)
		if len(parts) == 2 && parts[0] == code {
			return parts[1], true
		}
	}
	return "", false
}

This scans the entire file on every request. With 100,000 URLs, every redirect request reads 100,000 lines. If you get 100 requests per second, that is 10 million line reads per second. The server would crawl.

The Fix

We already fixed this in our implementation above. Load the file into a map on startup. Serve lookups from the map. Write new entries to the file for persistence. The map gives us O(1) lookup time. The file gives us durability.

This is the same pattern that databases like Redis use: serve reads from memory, persist writes to disk.

Step 5: HTTP Redirects and Click Analytics

The core feature of a URL shortener is the redirect. But we also want to know how many times each short URL is used.

Linux: Understanding Redirects

Let us see what a redirect looks like at the HTTP level.

curl -o /dev/null -s -w "Status: %{http_code}\nRedirect: %{redirect_url}\n" http://localhost:8080/f0e6a6a

Status: 302
Redirect: https://example.com

The -o /dev/null discards the body. The -s silences the progress bar. The -w flag formats the output to show just the status code and redirect URL.

There are two types of redirects in HTTP.

301 Moved Permanently. The browser caches this. Next time you visit the short URL, the browser goes straight to the destination without asking your server.

302 Found. The browser does not cache this. Every visit goes through your server first.

You can see the difference with curl.

curl -v -L http://localhost:8080/f0e6a6a

The -v flag shows headers. The -L flag follows redirects. You will see the 302 response with the Location header, then the final 200 response from the destination.

Go: Redirects with Analytics

We need to track every redirect. Add a click counter and timestamp tracking.

package main

import (
	"bufio"
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"fmt"
	"net/http"
	"os"
	"strings"
	"sync"
	"time"
)

const dataFile = "urls.txt"

type ClickRecord struct {
	Timestamp time.Time `json:"timestamp"`
	UserAgent string    `json:"user_agent"`
	IP        string    `json:"ip"`
}

type URLEntry struct {
	OriginalURL string
	Clicks      []ClickRecord
	CreatedAt   time.Time
}

type URLStore struct {
	mu      sync.RWMutex
	entries map[string]*URLEntry
	file    *os.File
}

type ShortenRequest struct {
	URL string `json:"url"`
}

type ShortenResponse struct {
	ShortURL string `json:"short_url"`
	Code     string `json:"code"`
}

type StatsResponse struct {
	Code        string    `json:"code"`
	OriginalURL string    `json:"original_url"`
	ClickCount  int       `json:"click_count"`
	CreatedAt   time.Time `json:"created_at"`
	LastClicked string    `json:"last_clicked"`
}

var store *URLStore

func newURLStore(filename string) (*URLStore, error) {
	s := &URLStore{
		entries: make(map[string]*URLEntry),
	}

	if _, err := os.Stat(filename); err == nil {
		file, err := os.Open(filename)
		if err != nil {
			return nil, fmt.Errorf("cannot open data file: %w", err)
		}

		scanner := bufio.NewScanner(file)
		count := 0
		for scanner.Scan() {
			line := scanner.Text()
			parts := strings.SplitN(line, " ", 2)
			if len(parts) == 2 {
				s.entries[parts[0]] = &URLEntry{
					OriginalURL: parts[1],
					Clicks:      []ClickRecord{},
					CreatedAt:   time.Now(),
				}
				count++
			}
		}
		file.Close()

		if err := scanner.Err(); err != nil {
			return nil, fmt.Errorf("error reading data file: %w", err)
		}
		fmt.Printf("Loaded %d URLs from %s\n", count, filename)
	}

	file, err := os.OpenFile(filename, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
	if err != nil {
		return nil, fmt.Errorf("cannot open data file for writing: %w", err)
	}
	s.file = file

	return s, nil
}

func (s *URLStore) Save(code, url string) error {
	s.mu.Lock()
	defer s.mu.Unlock()

	s.entries[code] = &URLEntry{
		OriginalURL: url,
		Clicks:      []ClickRecord{},
		CreatedAt:   time.Now(),
	}
	_, err := fmt.Fprintf(s.file, "%s %s\n", code, url)
	if err != nil {
		return fmt.Errorf("failed to write to file: %w", err)
	}
	return s.file.Sync()
}

func (s *URLStore) Get(code string) (*URLEntry, bool) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	entry, exists := s.entries[code]
	return entry, exists
}

func (s *URLStore) RecordClick(code string, r *http.Request) {
	s.mu.Lock()
	defer s.mu.Unlock()

	entry, exists := s.entries[code]
	if !exists {
		return
	}

	click := ClickRecord{
		Timestamp: time.Now(),
		UserAgent: r.UserAgent(),
		IP:        r.RemoteAddr,
	}
	entry.Clicks = append(entry.Clicks, click)
}

func generateCode(url string) string {
	hash := sha256.Sum256([]byte(url))
	fullHash := hex.EncodeToString(hash[:])
	return fullHash[:7]
}

func main() {
	var err error
	store, err = newURLStore(dataFile)
	if err != nil {
		fmt.Println("Error initializing store:", err)
		os.Exit(1)
	}
	defer store.file.Close()

	http.HandleFunc("/", handler)
	fmt.Println("Server running on :8080")
	err = http.ListenAndServe(":8080", nil)
	if err != nil {
		fmt.Println("Error:", err)
	}
}

func handler(w http.ResponseWriter, r *http.Request) {
	path := r.URL.Path

	if path == "/shorten" {
		if r.Method != http.MethodPost {
			http.Error(w, "Method not allowed. Use POST.", http.StatusMethodNotAllowed)
			return
		}
		handleShorten(w, r)
		return
	}

	if path == "/" {
		fmt.Fprintf(w, "URL Shortener is running\n")
		return
	}

	code := strings.TrimPrefix(path, "/")
	if strings.Contains(code, "/") {
		// Check for /stats/{code} pattern
		if strings.HasPrefix(code, "stats/") {
			statsCode := strings.TrimPrefix(code, "stats/")
			if !strings.Contains(statsCode, "/") {
				handleStats(w, r, statsCode)
				return
			}
		}
		http.NotFound(w, r)
		return
	}

	handleRedirect(w, r, code)
}

func handleShorten(w http.ResponseWriter, r *http.Request) {
	var req ShortenRequest
	err := json.NewDecoder(r.Body).Decode(&req)
	if err != nil || req.URL == "" {
		http.Error(w, "Invalid request. Send JSON with a 'url' field.", http.StatusBadRequest)
		return
	}

	code := generateCode(req.URL)

	// Check if already exists with same URL
	if entry, exists := store.Get(code); exists && entry.OriginalURL == req.URL {
		resp := ShortenResponse{
			ShortURL: fmt.Sprintf("http://localhost:8080/%s", code),
			Code:     code,
		}
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(resp)
		return
	}

	// Check for collision
	if entry, exists := store.Get(code); exists && entry.OriginalURL != req.URL {
		hash := sha256.Sum256([]byte(req.URL))
		fullHash := hex.EncodeToString(hash[:])
		for length := 8; length <= len(fullHash); length++ {
			code = fullHash[:length]
			if e, exists := store.Get(code); !exists || e.OriginalURL == req.URL {
				break
			}
		}
	}

	err = store.Save(code, req.URL)
	if err != nil {
		http.Error(w, "Failed to save URL", http.StatusInternalServerError)
		return
	}

	resp := ShortenResponse{
		ShortURL: fmt.Sprintf("http://localhost:8080/%s", code),
		Code:     code,
	}

	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(resp)
}

func handleRedirect(w http.ResponseWriter, r *http.Request, code string) {
	entry, exists := store.Get(code)
	if !exists {
		http.Error(w, "Short URL not found", http.StatusNotFound)
		return
	}

	store.RecordClick(code, r)

	http.Redirect(w, r, entry.OriginalURL, http.StatusMovedPermanently)
}

func handleStats(w http.ResponseWriter, r *http.Request, code string) {
	entry, exists := store.Get(code)
	if !exists {
		http.Error(w, "Short URL not found", http.StatusNotFound)
		return
	}

	lastClicked := "never"
	if len(entry.Clicks) > 0 {
		lastClicked = entry.Clicks[len(entry.Clicks)-1].Timestamp.Format(time.RFC3339)
	}

	stats := StatsResponse{
		Code:        code,
		OriginalURL: entry.OriginalURL,
		ClickCount:  len(entry.Clicks),
		CreatedAt:   entry.CreatedAt,
		LastClicked: lastClicked,
	}

	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(stats)
}

Test the analytics.

go run main.go

Create a URL and click it a few times.

curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' \
  http://localhost:8080/shorten

curl -s -o /dev/null http://localhost:8080/f0e6a6a
curl -s -o /dev/null http://localhost:8080/f0e6a6a
curl -s -o /dev/null http://localhost:8080/f0e6a6a

Now check the stats.

curl -s http://localhost:8080/stats/f0e6a6a | python3 -m json.tool

{
    "code": "f0e6a6a",
    "original_url": "https://example.com",
    "click_count": 3,
    "created_at": "2024-01-15T10:30:00Z",
    "last_clicked": "2024-01-15T10:30:05Z"
}

Three clicks recorded. The stats endpoint gives us click count, creation time, and last click time.

The Bug: 301 Permanent Redirect Kills Analytics

Look at the redirect handler. We used http.StatusMovedPermanently (301). Test this with curl’s verbose mode.

curl -v -L http://localhost:8080/f0e6a6a

< HTTP/1.1 301 Moved Permanently
< Location: https://example.com

That 301 tells the browser: “This URL has permanently moved. Cache this redirect and never ask me again.”

After the first visit, the browser goes straight to https://example.com without ever contacting your server. Your analytics counter stops incrementing because no requests reach your server.

To see this, open the short URL in a browser. Then check stats. Click count will be 1. Click the link again in the same browser. Click count stays at 1. The browser cached the redirect and bypassed your server.

The Fix: Use 302 Temporary Redirect

Change the redirect status to 302. This tells the browser: “Go to this URL, but ask me again next time.”

func handleRedirect(w http.ResponseWriter, r *http.Request, code string) {
	entry, exists := store.Get(code)
	if !exists {
		http.Error(w, "Short URL not found", http.StatusNotFound)
		return
	}

	store.RecordClick(code, r)

	http.Redirect(w, r, entry.OriginalURL, http.StatusFound) // 302, not 301
}

Now every click goes through your server first. Your analytics are accurate.

The tradeoff: 302 redirects are slightly slower because the browser always contacts your server first. For a URL shortener, this is the right choice. You want accurate click counts more than you want to save a few milliseconds.

Step 6: Complete URL Shortener with Dashboard

Now we combine everything into the final service. We add rate limiting, colored terminal logging, and a stats dashboard.

Rate Limiting with Token Buckets

Before building the final version, understand the concept. A token bucket allows a fixed number of actions in a time window. Each IP address gets a bucket. Every request removes a token. Tokens refill over time.

# Simulate: send 15 requests rapidly to see rate limiting
for i in $(seq 1 15); do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST \
    -H "Content-Type: application/json" \
    -d "{\"url\":\"https://example.com/$i\"}" \
    http://localhost:8080/shorten)
  echo "Request $i: HTTP $STATUS"
done

After the 10th request, you should see HTTP 429 (Too Many Requests).

The Complete Service

Here is the full URL shortener with all features combined.

package main

import (
	"bufio"
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"fmt"
	"net/http"
	"os"
	"strings"
	"sync"
	"time"
)

// --- ANSI color codes for terminal output ---

const (
	colorReset  = "\033[0m"
	colorRed    = "\033[31m"
	colorGreen  = "\033[32m"
	colorYellow = "\033[33m"
	colorBlue   = "\033[34m"
	colorCyan   = "\033[36m"
	colorGray   = "\033[90m"
)

// --- Data types ---

const dataFile = "urls.txt"

type ClickRecord struct {
	Timestamp time.Time `json:"timestamp"`
	UserAgent string    `json:"user_agent"`
	IP        string    `json:"ip"`
	Referrer  string    `json:"referrer"`
}

type URLEntry struct {
	OriginalURL string
	Clicks      []ClickRecord
	CreatedAt   time.Time
}

type URLStore struct {
	mu      sync.RWMutex
	entries map[string]*URLEntry
	file    *os.File
}

type ShortenRequest struct {
	URL string `json:"url"`
}

type ShortenResponse struct {
	ShortURL string `json:"short_url"`
	Code     string `json:"code"`
}

type StatsResponse struct {
	Code         string           `json:"code"`
	OriginalURL  string           `json:"original_url"`
	ClickCount   int              `json:"click_count"`
	CreatedAt    time.Time        `json:"created_at"`
	LastClicked  string           `json:"last_clicked"`
	TopReferrers map[string]int   `json:"top_referrers"`
	RecentClicks []ClickSummary   `json:"recent_clicks"`
}

type ClickSummary struct {
	Timestamp string `json:"timestamp"`
	UserAgent string `json:"user_agent"`
}

// --- Rate limiter ---

type RateLimiter struct {
	mu      sync.Mutex
	buckets map[string]*Bucket
}

type Bucket struct {
	tokens    int
	lastReset time.Time
}

const (
	maxTokens   = 10
	resetPeriod = time.Minute
)

func newRateLimiter() *RateLimiter {
	return &RateLimiter{
		buckets: make(map[string]*Bucket),
	}
}

func (rl *RateLimiter) Allow(ip string) bool {
	rl.mu.Lock()
	defer rl.mu.Unlock()

	bucket, exists := rl.buckets[ip]
	if !exists {
		rl.buckets[ip] = &Bucket{
			tokens:    maxTokens - 1,
			lastReset: time.Now(),
		}
		return true
	}

	// Refill tokens if the reset period has passed
	if time.Since(bucket.lastReset) > resetPeriod {
		bucket.tokens = maxTokens
		bucket.lastReset = time.Now()
	}

	if bucket.tokens <= 0 {
		return false
	}

	bucket.tokens--
	return true
}

// --- URL Store ---

var store *URLStore
var limiter *RateLimiter

func newURLStore(filename string) (*URLStore, error) {
	s := &URLStore{
		entries: make(map[string]*URLEntry),
	}

	if _, err := os.Stat(filename); err == nil {
		file, err := os.Open(filename)
		if err != nil {
			return nil, fmt.Errorf("cannot open data file: %w", err)
		}

		scanner := bufio.NewScanner(file)
		count := 0
		for scanner.Scan() {
			line := scanner.Text()
			parts := strings.SplitN(line, " ", 2)
			if len(parts) == 2 {
				s.entries[parts[0]] = &URLEntry{
					OriginalURL: parts[1],
					Clicks:      []ClickRecord{},
					CreatedAt:   time.Now(),
				}
				count++
			}
		}
		file.Close()

		if err := scanner.Err(); err != nil {
			return nil, fmt.Errorf("error reading data file: %w", err)
		}
		logInfo("Loaded %d URLs from %s", count, filename)
	}

	file, err := os.OpenFile(filename, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
	if err != nil {
		return nil, fmt.Errorf("cannot open data file for writing: %w", err)
	}
	s.file = file

	return s, nil
}

func (s *URLStore) Save(code, url string) error {
	s.mu.Lock()
	defer s.mu.Unlock()

	s.entries[code] = &URLEntry{
		OriginalURL: url,
		Clicks:      []ClickRecord{},
		CreatedAt:   time.Now(),
	}
	_, err := fmt.Fprintf(s.file, "%s %s\n", code, url)
	if err != nil {
		return fmt.Errorf("failed to write to file: %w", err)
	}
	return s.file.Sync()
}

func (s *URLStore) Get(code string) (*URLEntry, bool) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	entry, exists := s.entries[code]
	return entry, exists
}

func (s *URLStore) RecordClick(code string, r *http.Request) {
	s.mu.Lock()
	defer s.mu.Unlock()

	entry, exists := s.entries[code]
	if !exists {
		return
	}

	click := ClickRecord{
		Timestamp: time.Now(),
		UserAgent: r.UserAgent(),
		IP:        r.RemoteAddr,
		Referrer:  r.Referer(),
	}
	entry.Clicks = append(entry.Clicks, click)
}

func (s *URLStore) Count() int {
	s.mu.RLock()
	defer s.mu.RUnlock()
	return len(s.entries)
}

// --- Code generation ---

func generateCode(url string) string {
	hash := sha256.Sum256([]byte(url))
	fullHash := hex.EncodeToString(hash[:])
	return fullHash[:7]
}

func resolveCode(url string) string {
	code := generateCode(url)

	if entry, exists := store.Get(code); exists && entry.OriginalURL != url {
		hash := sha256.Sum256([]byte(url))
		fullHash := hex.EncodeToString(hash[:])
		for length := 8; length <= len(fullHash); length++ {
			code = fullHash[:length]
			if e, exists := store.Get(code); !exists || e.OriginalURL == url {
				break
			}
		}
	}

	return code
}

// --- Colored logging ---

func logInfo(format string, args ...interface{}) {
	timestamp := time.Now().Format("15:04:05")
	msg := fmt.Sprintf(format, args...)
	fmt.Printf("%s[%s]%s %s%s%s\n", colorGray, timestamp, colorReset, colorCyan, msg, colorReset)
}

func logRedirect(code, destination, ip string) {
	timestamp := time.Now().Format("15:04:05")
	fmt.Printf("%s[%s]%s %sREDIRECT%s %s/%s%s -> %s%s%s from %s\n",
		colorGray, timestamp, colorReset,
		colorGreen, colorReset,
		colorYellow, code, colorReset,
		colorBlue, destination, colorReset,
		ip,
	)
}

func logCreate(code, url string) {
	timestamp := time.Now().Format("15:04:05")
	fmt.Printf("%s[%s]%s %sCREATE%s   %s/%s%s -> %s%s%s\n",
		colorGray, timestamp, colorReset,
		colorGreen, colorReset,
		colorYellow, code, colorReset,
		colorBlue, url, colorReset,
	)
}

func logRateLimit(ip string) {
	timestamp := time.Now().Format("15:04:05")
	fmt.Printf("%s[%s]%s %sRATE LIMIT%s %s%s%s\n",
		colorGray, timestamp, colorReset,
		colorRed, colorReset,
		colorYellow, ip, colorReset,
	)
}

func logRequest(method, path string, status int) {
	timestamp := time.Now().Format("15:04:05")
	statusColor := colorGreen
	if status >= 400 {
		statusColor = colorRed
	} else if status >= 300 {
		statusColor = colorYellow
	}
	fmt.Printf("%s[%s]%s %s%d%s %s %s\n",
		colorGray, timestamp, colorReset,
		statusColor, status, colorReset,
		method, path,
	)
}

// --- Extracting client IP ---

func getClientIP(r *http.Request) string {
	// Check X-Forwarded-For header first (for proxied requests)
	forwarded := r.Header.Get("X-Forwarded-For")
	if forwarded != "" {
		parts := strings.SplitN(forwarded, ",", 2)
		return strings.TrimSpace(parts[0])
	}

	// Fall back to RemoteAddr
	ip := r.RemoteAddr
	// Remove port number
	if colonIdx := strings.LastIndex(ip, ":"); colonIdx != -1 {
		ip = ip[:colonIdx]
	}
	return ip
}

// --- HTTP handlers ---

func main() {
	var err error
	store, err = newURLStore(dataFile)
	if err != nil {
		fmt.Println("Error initializing store:", err)
		os.Exit(1)
	}
	defer store.file.Close()

	limiter = newRateLimiter()

	http.HandleFunc("/", handler)

	logInfo("URL Shortener running on :8080")
	logInfo("Endpoints: POST /shorten, GET /{code}, GET /stats/{code}")
	logInfo("Stored URLs: %d", store.Count())

	err = http.ListenAndServe(":8080", nil)
	if err != nil {
		fmt.Println("Error:", err)
	}
}

func handler(w http.ResponseWriter, r *http.Request) {
	path := r.URL.Path

	if path == "/shorten" {
		if r.Method != http.MethodPost {
			logRequest(r.Method, path, http.StatusMethodNotAllowed)
			http.Error(w, "Method not allowed. Use POST.", http.StatusMethodNotAllowed)
			return
		}
		handleShorten(w, r)
		return
	}

	if path == "/" {
		logRequest(r.Method, path, http.StatusOK)
		w.Header().Set("Content-Type", "text/plain")
		fmt.Fprintf(w, "URL Shortener is running. POST to /shorten to create a short URL.\n")
		return
	}

	trimmed := strings.TrimPrefix(path, "/")

	// Check for /stats/{code}
	if strings.HasPrefix(trimmed, "stats/") {
		statsCode := strings.TrimPrefix(trimmed, "stats/")
		if statsCode != "" && !strings.Contains(statsCode, "/") {
			handleStats(w, r, statsCode)
			return
		}
	}

	// Only single-segment paths are short codes
	if strings.Contains(trimmed, "/") {
		logRequest(r.Method, path, http.StatusNotFound)
		http.NotFound(w, r)
		return
	}

	handleRedirect(w, r, trimmed)
}

func handleShorten(w http.ResponseWriter, r *http.Request) {
	clientIP := getClientIP(r)

	// Rate limiting
	if !limiter.Allow(clientIP) {
		logRateLimit(clientIP)
		http.Error(w, "Rate limit exceeded. Max 10 URLs per minute.", http.StatusTooManyRequests)
		return
	}

	var req ShortenRequest
	err := json.NewDecoder(r.Body).Decode(&req)
	if err != nil || req.URL == "" {
		logRequest(r.Method, "/shorten", http.StatusBadRequest)
		http.Error(w, "Invalid request. Send JSON with a 'url' field.", http.StatusBadRequest)
		return
	}

	// Basic URL validation
	if !strings.HasPrefix(req.URL, "http://") && !strings.HasPrefix(req.URL, "https://") {
		logRequest(r.Method, "/shorten", http.StatusBadRequest)
		http.Error(w, "URL must start with http:// or https://", http.StatusBadRequest)
		return
	}

	code := resolveCode(req.URL)

	// Check if already exists with same URL
	if entry, exists := store.Get(code); exists && entry.OriginalURL == req.URL {
		resp := ShortenResponse{
			ShortURL: fmt.Sprintf("http://localhost:8080/%s", code),
			Code:     code,
		}
		logRequest(r.Method, "/shorten", http.StatusOK)
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(resp)
		return
	}

	err = store.Save(code, req.URL)
	if err != nil {
		logRequest(r.Method, "/shorten", http.StatusInternalServerError)
		http.Error(w, "Failed to save URL", http.StatusInternalServerError)
		return
	}

	logCreate(code, req.URL)

	resp := ShortenResponse{
		ShortURL: fmt.Sprintf("http://localhost:8080/%s", code),
		Code:     code,
	}

	logRequest(r.Method, "/shorten", http.StatusCreated)
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusCreated)
	json.NewEncoder(w).Encode(resp)
}

func handleRedirect(w http.ResponseWriter, r *http.Request, code string) {
	entry, exists := store.Get(code)
	if !exists {
		logRequest(r.Method, "/"+code, http.StatusNotFound)
		http.Error(w, "Short URL not found", http.StatusNotFound)
		return
	}

	store.RecordClick(code, r)
	logRedirect(code, entry.OriginalURL, getClientIP(r))

	http.Redirect(w, r, entry.OriginalURL, http.StatusFound) // 302 temporary redirect
}

func handleStats(w http.ResponseWriter, r *http.Request, code string) {
	entry, exists := store.Get(code)
	if !exists {
		logRequest(r.Method, "/stats/"+code, http.StatusNotFound)
		http.Error(w, "Short URL not found", http.StatusNotFound)
		return
	}

	store.mu.RLock()
	clicks := entry.Clicks
	store.mu.RUnlock()

	lastClicked := "never"
	if len(clicks) > 0 {
		lastClicked = clicks[len(clicks)-1].Timestamp.Format(time.RFC3339)
	}

	// Count referrers
	referrers := make(map[string]int)
	for _, click := range clicks {
		ref := click.Referrer
		if ref == "" {
			ref = "direct"
		}
		referrers[ref]++
	}

	// Get recent clicks (last 10)
	recent := []ClickSummary{}
	start := 0
	if len(clicks) > 10 {
		start = len(clicks) - 10
	}
	for _, click := range clicks[start:] {
		recent = append(recent, ClickSummary{
			Timestamp: click.Timestamp.Format(time.RFC3339),
			UserAgent: click.UserAgent,
		})
	}

	stats := StatsResponse{
		Code:         code,
		OriginalURL:  entry.OriginalURL,
		ClickCount:   len(clicks),
		CreatedAt:    entry.CreatedAt,
		LastClicked:  lastClicked,
		TopReferrers: referrers,
		RecentClicks: recent,
	}

	logRequest(r.Method, "/stats/"+code, http.StatusOK)
	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(stats)
}

Running the Complete Service

Start the server.

go run main.go

You will see colored output in your terminal.

[10:30:00] URL Shortener running on :8080
[10:30:00] Endpoints: POST /shorten, GET /{code}, GET /stats/{code}
[10:30:00] Stored URLs: 0

Create a short URL.

curl -s -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://go.dev/doc/"}' \
  http://localhost:8080/shorten | python3 -m json.tool

{
    "short_url": "http://localhost:8080/a1b2c3d",
    "code": "a1b2c3d"
}

The server terminal shows.

[10:30:05] CREATE   /a1b2c3d -> https://go.dev/doc/
[10:30:05] 201 POST /shorten

Click the short URL a few times.

curl -s -o /dev/null http://localhost:8080/a1b2c3d
curl -s -o /dev/null http://localhost:8080/a1b2c3d
curl -s -o /dev/null http://localhost:8080/a1b2c3d

Each redirect appears in the terminal with color.

[10:30:10] REDIRECT /a1b2c3d -> https://go.dev/doc/ from 127.0.0.1
[10:30:11] REDIRECT /a1b2c3d -> https://go.dev/doc/ from 127.0.0.1
[10:30:12] REDIRECT /a1b2c3d -> https://go.dev/doc/ from 127.0.0.1

Check the stats.

curl -s http://localhost:8080/stats/a1b2c3d | python3 -m json.tool

{
    "code": "a1b2c3d",
    "original_url": "https://go.dev/doc/",
    "click_count": 3,
    "created_at": "2024-01-15T10:30:05Z",
    "last_clicked": "2024-01-15T10:30:12Z",
    "top_referrers": {
        "direct": 3
    },
    "recent_clicks": [
        {
            "timestamp": "2024-01-15T10:30:10Z",
            "user_agent": "curl/7.81.0"
        },
        {
            "timestamp": "2024-01-15T10:30:11Z",
            "user_agent": "curl/7.81.0"
        },
        {
            "timestamp": "2024-01-15T10:30:12Z",
            "user_agent": "curl/7.81.0"
        }
    ]
}

Testing Rate Limiting

Send many requests quickly.

for i in $(seq 1 15); do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST \
    -H "Content-Type: application/json" \
    -d "{\"url\":\"https://example.com/$i\"}" \
    http://localhost:8080/shorten)
  echo "Request $i: HTTP $STATUS"
done

Request 1: HTTP 201
Request 2: HTTP 201
Request 3: HTTP 201
Request 4: HTTP 201
Request 5: HTTP 201
Request 6: HTTP 201
Request 7: HTTP 201
Request 8: HTTP 201
Request 9: HTTP 201
Request 10: HTTP 201
Request 11: HTTP 429
Request 12: HTTP 429
Request 13: HTTP 429
Request 14: HTTP 429
Request 15: HTTP 429

The first 10 requests succeed. After that, the rate limiter kicks in and returns 429 Too Many Requests. The server terminal shows rate limit events in red.

[10:31:00] RATE LIMIT 127.0.0.1

Wait one minute and try again. The tokens refill and requests succeed.

Testing URL Validation

Try submitting an invalid URL.

curl -s -X POST -H "Content-Type: application/json" \
  -d '{"url":"not-a-url"}' \
  http://localhost:8080/shorten

URL must start with http:// or https://

Try an empty request.

curl -s -X POST -H "Content-Type: application/json" \
  -d '{}' \
  http://localhost:8080/shorten

Invalid request. Send JSON with a 'url' field.

The server validates input before processing.

Testing Persistence

Stop the server with Ctrl+C. Check the data file.

cat urls.txt

You will see all the URLs you created. Start the server again.

go run main.go

[10:35:00] Loaded 10 URLs from urls.txt
[10:35:00] URL Shortener running on :8080
[10:35:00] Endpoints: POST /shorten, GET /{code}, GET /stats/{code}
[10:35:00] Stored URLs: 10

All URLs are loaded from disk. The short codes still work. Click analytics are reset on restart because they are only stored in memory. If you need persistent analytics, you would write click events to a separate file or a database.

How the Pieces Fit Together

Here is the complete request flow for each operation.

Creating a short URL:

Client sends POST /shorten with a JSON body containing the URL
Rate limiter checks if the client IP has tokens remaining
Server validates the URL format
Server generates a 7-character hash code from the URL
Server checks for hash collisions and extends the code if needed
Server writes the mapping to urls.txt and stores it in memory
Server returns the short URL as JSON

Redirecting:

Client sends GET /{code}
Server looks up the code in the in-memory map
Server records a click with timestamp, User-Agent, IP, and referrer
Server returns a 302 redirect with the Location header set to the original URL
Client follows the redirect to the destination

Viewing stats:

Client sends GET /stats/{code}
Server looks up the code and its click history
Server calculates referrer counts and formats recent clicks
Server returns JSON with click count, timestamps, and referrer data

Every step in this guide built toward this final result. The TCP server showed how HTTP works under the hood. The routing showed how to dispatch requests. The code generation showed how to create unique identifiers. The file store showed how to persist data. The redirect analysis showed why 302 matters. And the rate limiter showed how to protect the service from abuse.

You now have a working URL shortener built from first principles. Every line of code uses only the Go standard library. No frameworks, no databases, no external services.

Keep Reading

Go + Nginx: Deploy a Go API Behind a Reverse Proxy — put your URL shortener behind nginx for production use with TLS and load balancing.
Containers From Scratch in Go — package your URL shortener into a container using the same Go patterns.
Go + DynamoDB: Build a Simple CRUD App — replace the file-based storage with DynamoDB for a more scalable backend.

More from cloud

Log Aggregator From Scratch in Go

Build a log aggregator in Go from scratch. Tail files with inotify, survive log rotation, parse …

Terraform From Scratch: Provision AWS Infrastructure Step by Step

Learn Terraform with AWS from scratch. Start with a single S3 bucket, hit real errors, fix them, …

AWS CLI Automation: From Bash Scripts to Go

Learn AWS automation step by step. Start with AWS CLI commands for S3, EC2, and IAM, then build the …

Nginx Log Analysis: From grep to a Go Log Parser

Learn nginx log analysis step by step — start with grep and awk one-liners for quick answers, then …

Step 1: TCP Connections and HTTP Basics

Linux: Listen on a Port

Go: Build a TCP Server

The Bug: Missing POST Body

The Fix: Read the Content-Length

Step 2: HTTP Server with Routing

Linux: Test HTTP Methods

Go: Two Routes with net/http

The Bug: No 404 for Unknown Paths

The Fix: Explicit Route Checking

Step 3: Generating Short Codes

Linux: Two Ways to Generate Codes

Go: Hash-Based Code Generation

The Bug: Codes Are Too Long

The Fix: Truncate and Handle Collisions

Step 4: Storing URLs Persistently

Linux: File-Based Storage

Go: File-Based Store with In-Memory Lookup

The Bug: Reading the Whole File on Every Lookup

The Fix

Step 5: HTTP Redirects and Click Analytics

Linux: Understanding Redirects

Go: Redirects with Analytics

The Bug: 301 Permanent Redirect Kills Analytics

The Fix: Use 302 Temporary Redirect

Step 6: Complete URL Shortener with Dashboard

Rate Limiting with Token Buckets

The Complete Service

Running the Complete Service

Testing Rate Limiting

Testing URL Validation

Testing Persistence

How the Pieces Fit Together

Keep Reading

Similar Articles

Related Content

More from cloud

You Might Also Like