# Building a YouTube-like System with AWS Lambda and S3 - URL: https://karandeepsingh.ca/posts/youtube-system-design-aws/ - Author: Karandeep Singh - Published: 2023-08-28 - Series: Serverless on AWS - Tags: serverless - Summary: Create a scalable video platform using AWS Lambda functions that process uploads directly from S3. This simple approach works beautifully for video platforms and requires minimal management. ![Data processing technology representing a scalable video platform architecture](https://karandeepsingh.ca/images/posts/youtube-system-design.jpg) ## Building a YouTube-like System: The Simple Approach Simple solutions to complex problems are deeply satisfying. Building a video platform similar to YouTube sounds like months of work and complicated infrastructure — until you see how AWS Lambda and S3 work together. This straightforward approach can take a video platform from concept to working prototype remarkably quickly, without the usual orchestration overhead. Let me walk you through how to build a YouTube-like system using AWS services. This approach is refreshingly simple - videos go in, get processed automatically, and come out ready for viewing. No complicated orchestration required! Because the pieces are event-driven and serverless, this pattern scales with demand, so you can start small and grow without changing your architecture. This pattern is sometimes called a *serverless transformation* because it changes how applications are built and scaled. ## System Overview: The Building Blocks Our YouTube-like system uses these key AWS components: - **S3 buckets** for storing videos (both raw uploads and processed versions) - **Lambda functions** that respond to events and coordinate processing - **MediaConvert** for transcoding videos into different formats - **CloudFront** for delivering videos globally - **DynamoDB** for storing video metadata - **OpenSearch** for making videos searchable - **Cognito** for user management I've named these resources to make them easy to understand: - `raw-video-bucket` - Where uploaded videos first land - `processed-video-bucket` - Where finished, viewable videos live - `video-processing-function` - Lambda that handles new uploads - `transcoding-job-queue` - MediaConvert queue for processing - `video-delivery-network` - CloudFront distribution - `video-metadata-table` - DynamoDB table - `search-indexing-function` - Lambda that updates the search index - `video-search-service` - OpenSearch domain - `user-pool` - Cognito user management The whole system flows like this: ``` [User Upload] → [raw-video-bucket] → [Event] → [video-processing-function] → [transcoding-job-queue] ↓ [User Viewing] ← [video-delivery-network] ← [processed-video-bucket] ← [Transcoded Videos] ``` This event-driven pattern is often called *choreography*: each component reacts to events instead of being told what to do. It's an approach that's both simple and powerful. ## The Upload and Processing Flow Let me walk you through what happens when someone uploads a video: 1. **User Uploads a Video** When Maria uploads her vacation video, our application gets a special pre-signed URL from AWS and uploads directly to our `raw-video-bucket`. This is efficient because the video doesn't need to pass through our servers. 2. **S3 Triggers Lambda** As soon as the video finishes uploading, S3 automatically sends an event to our `video-processing-function`. This happens because we set up event notifications on our bucket that say "when a new object arrives, tell Lambda about it." 3. **Lambda Processes the Video** Our Lambda function wakes up and receives information about Maria's video - where it's stored, its size, when it was uploaded. The function then: - Creates a record in our metadata table saying "processing started" - Analyzes the video to determine the best processing settings - Creates a job specification for MediaConvert with instructions like "create three versions: 480p, 720p, and 1080p" - Sends this job to MediaConvert's `transcoding-job-queue` 4. **MediaConvert Does the Heavy Lifting** MediaConvert processes the video according to our specifications. It's specially designed for video work and handles all the complex encoding operations. When it's done, it places the processed videos in our `processed-video-bucket`. 5. **Completion Handling** When MediaConvert finishes, it sends a completion event. Another Lambda function catches this event and: - Updates our metadata table to show the video is ready - Adds information like video duration, thumbnail URLs, and available resolutions - Makes the video searchable in our search service 6. **Video Delivery** Now when someone wants to watch Maria's vacation video, our application: - Checks the metadata table to find available formats - Selects the appropriate resolution based on the viewer's device - Delivers the video through CloudFront for fast, global access The beauty of this design is that everything happens automatically once the video is uploaded. This event-driven approach — explored in depth in Peter Sbarski's book *Serverless Architectures on AWS* — creates systems that respond to changes as they happen. ## Making Videos Searchable A video platform isn't much use if people can't find the content they want! Here's how we make videos searchable: 1. **Metadata Collection** When Maria uploads her vacation video, she also adds a title ("Summer in Greece"), description ("Two amazing weeks island hopping"), and tags ("travel", "beach", "summer"). 2. **Initial Database Entry** Our `video-processing-function` creates an entry in the `video-metadata-table` with: - Video ID (a unique identifier) - User ID (Maria's account) - Title, description, and tags - Upload time - Status ("PROCESSING") 3. **DynamoDB to Lambda to Search** We've configured DynamoDB to send changes to our `search-indexing-function`. When a record changes: - Lambda receives the change notification - If the video is ready (status = "READY"), it formats the data for search - It updates the `video-search-service` with the searchable content 4. **Completion Updates** When MediaConvert finishes processing Maria's video: - The metadata record is updated with status "READY" - Technical details are added (duration, formats, etc.) - This change triggers the search indexing process 5. **Search Experience** Now when someone searches for "Greece vacation": - Our search service finds Maria's video based on title and description - Results include thumbnails and basic details - Users can filter by duration, upload date, and other attributes This approach creates a seamless search experience similar to YouTube, where content becomes findable as soon as it's ready for viewing. ## Making it Fast: Performance Optimization Video processing can be slow, but I've learned several tricks to speed things up: 1. **Right-Size Your Lambda** Our `video-processing-function` uses 3008MB of memory. This may seem like a lot, but with Lambda, more memory also means more CPU power — and increasing memory can actually reduce costs because functions finish faster. 2. **Smart Video Analysis** Not all videos are created equal. Our Lambda quickly analyzes each upload to determine: - Is it high motion (sports) or mostly static (lecture)? - What's the input resolution and quality? - How long is the content? Then it customizes the MediaConvert job accordingly. 3. **Accelerated Transcoding** For most videos, we use MediaConvert's accelerated transcoding, which can be up to 3x faster. It costs a bit more but dramatically improves the user experience - most people don't want to wait hours to share their videos. 4. **Parallel Processing** For longer videos, the system creates multiple smaller transcoding jobs that can run in parallel. This approach, recommended in the [AWS Media blog](https://aws.amazon.com/blogs/media/), can turn a 2-hour processing job into 30 minutes. 5. **Performance Monitoring** Track processing time for different video types and refine settings accordingly. As a rough rule of thumb, end-to-end processing time scales with video length and resolution — a short mobile clip processes in well under a minute, a 10-minute HD video in a few minutes, and a full-hour presentation in roughly fifteen. These optimizations mean Maria's vacation video is ready to share much faster, creating a better user experience. ## Keeping Videos Safe: Security Implementation Security is critical for a video platform. Here's how we protect content: 1. **Secure Buckets** Our S3 buckets are locked down tight: - `raw-video-bucket` only allows authenticated uploads - All data is encrypted at rest using server-side encryption - Public access is completely blocked 2. **Precise Permissions** Our Lambda functions have exactly the permissions they need and nothing more: - `video-processing-function` can read from raw bucket, submit MediaConvert jobs, and write metadata - `search-indexing-function` can only read from DynamoDB and update the search service 3. **User Management** Cognito's `user-pool` handles authentication with: - Password policies requiring strong credentials - Optional multi-factor authentication for sensitive operations - Temporary credentials for uploads that expire after 15 minutes 4. **Content Validation** Before processing, we validate uploads: - Verify file is actually a video - Check for maximum size limits - Scan for potential security issues 5. **Access Controls** Videos can be marked as: - Public (anyone can view) - Private (only the owner) - Shared (specific users or groups) Our system enforces these permissions at the application and CDN level. These security measures follow AWS best practices outlined in their [Well-Architected Framework](https://aws.amazon.com/architecture/well-architected/) and give users confidence their content is protected. ## Watching Everything: Monitoring and Alerting Good monitoring is like having a security camera system for your application. Here's our approach: 1. **Comprehensive Dashboard** We created a CloudWatch dashboard that shows: - Upload counts by hour/day - Processing success rates - Average processing times - Error counts and types This gives us a quick overview of system health. 2. **Smart Alerting** Not all issues are equal, so our alerts are tiered: - Critical: Processing failures above 5% or system outages - Warning: Performance degradation or capacity concerns - Info: Unusual patterns worth investigating 3. **End-to-End Tracking** We track videos through their entire journey: - Upload success/failure - Processing start/completion - Delivery statistics If a video enters our system but doesn't complete processing within expected timeframes, we investigate automatically. 4. **User Experience Metrics** Beyond just technical metrics, we track: - Video loading times for viewers - Buffering frequency - Search response times These affect how users perceive our platform. Good observability is essential for serverless applications — a point serverless practitioners consistently stress. You can't improve what you can't measure. ## Putting It All Together: The Complete System The complete YouTube-like system combines all these components into a seamless whole. Here's how it all fits together: ``` ┌─── [search-indexing-function] ─── [video-search-service] │ [User] ─── [Upload] ─── [raw-video-bucket] ─── [video-processing-function] ─── [transcoding-job-queue] │ │ │ │ └─── [video-metadata-table] ◄────┘ │ │ [User] ◄─── [video-delivery-network] ◄─── [processed-video-bucket] ◄─── [MediaConvert] ``` The beauty of this design is its simplicity. Each component has a clear purpose and connects directly to the next. AWS handles all the infrastructure, scaling, and reliability concerns automatically. When building your own YouTube-like system, I recommend starting with this approach because: 1. **It's straightforward to implement** - You can build a prototype in days 2. **It scales automatically** - From 10 videos to 10 million 3. **It's cost-effective** - You only pay for what you use 4. **It requires minimal maintenance** - No servers to patch or scale The guiding principle: focus on what makes your application special, and let the cloud handle the rest. ## Getting Started: Your Next Steps Ready to build your own YouTube-like platform? Here's how to get started: 1. **Set Up Your AWS Account** If you don't already have one, create an AWS account and familiarize yourself with the AWS console. 2. **Create Your Resources** Begin by creating: - S3 buckets for raw and processed videos - A DynamoDB table for metadata - A basic Lambda function for processing 3. **Configure Event Triggers** Set up S3 event notifications to trigger your Lambda function when videos are uploaded. 4. **Build a Simple Frontend** Create a basic web application that allows users to: - Upload videos - View their videos after processing - Search for content 5. **Iterate and Improve** Start simple and add features as you go: - Multiple resolution support - Enhanced search - User profiles and subscriptions The [AWS Serverless Workshop](https://aws.amazon.com/getting-started/hands-on/build-serverless-web-app-lambda-apigateway-s3-dynamodb-cognito/) provides excellent hands-on tutorials that follow a similar pattern to what we've discussed. Building a YouTube-like system used to require dozens of servers, complex orchestration, and big budgets. Now, with this straightforward approach using AWS Lambda and S3, you can create a scalable, efficient video platform with minimal infrastructure and maintenance. The event-driven pattern makes everything simpler while giving your users a smooth, responsive experience. I'm still amazed at how easy this approach makes what used to be an incredibly complex task. Now go build something awesome! ## Quiz 1. What triggers video processing in this YouTube-like system? a) API Gateway endpoints b) Scheduled tasks c) S3 event notifications d) Manual approval Answer: c. In this YouTube-like system design, video processing is triggered directly by S3 event notifications when a new video is uploaded to the raw video storage bucket. This straightforward approach keeps the system simple and responsive.