Skip to content

micro-todo/infrastructure

BranchesTags

Repository files navigation

MicroTodo Infrastructure

This repository contains the complete infrastructure-as-code (IaC) configuration for the MicroTodo application - a cloud-native microservices-based todo application deployed on AWS EKS (Elastic Kubernetes Service).

📋 Table of Contents

🏗️ Architecture Overview

MicroTodo is built using a microservices architecture with the following components:

┌─────────────────────────────────────────────────────────────┐
│                         Internet                             │
└───────────────────────────┬─────────────────────────────────┘
                            │
                    ┌───────▼────────┐
                    │  AWS ALB/NLB   │
                    │   (Ingress)    │
                    └───────┬────────┘
                            │
                    ┌───────▼────────┐
                    │  API Gateway   │
                    └───────┬────────┘
                            │
        ┌───────────────────┼───────────────────┐
        │                   │                   │
   ┌────▼─────┐      ┌─────▼──────┐     ┌─────▼──────┐
   │  Users   │      │   Tasks    │     │Notifications│
   │ Service  │      │  Service   │     │  Service    │
   └────┬─────┘      └─────┬──────┘     └─────┬──────┘
        │                  │                   │
   ┌────▼─────┐      ┌─────▼──────┐     ┌─────▼──────┐
   │PostgreSQL│      │PostgreSQL  │     │PostgreSQL  │
   │  Users   │      │   Tasks    │     │Notifications│
   └──────────┘      └────────────┘     └────────────┘
                            │
                    ┌───────▼────────┐
                    │   RabbitMQ     │
                    │ (Message Bus)  │
                    └────────────────┘

Key Features

  • Microservices Architecture: Separate services for users, tasks, and notifications
  • API Gateway Pattern: Single entry point for all client requests
  • Event-Driven Communication: RabbitMQ for asynchronous messaging between services
  • Database per Service: Each microservice has its own PostgreSQL database
  • Container Orchestration: Kubernetes (EKS) for container management
  • GitOps Deployment: ArgoCD for continuous deployment
  • Infrastructure as Code: Terraform for AWS resource provisioning
  • Secrets Management: AWS Secrets Manager with External Secrets Operator

🛠️ Technology Stack

Infrastructure

  • Cloud Provider: AWS
  • Kubernetes: Amazon EKS 1.31
  • Infrastructure as Code: Terraform (~> 1.13)
  • Container Registry: Amazon ECR
  • Secret Management: AWS Secrets Manager
  • State Management: S3 + DynamoDB for Terraform state

Kubernetes Ecosystem

  • Ingress Controller: AWS Load Balancer Controller (v1.13.4)
  • GitOps: ArgoCD (v8.5.7)
  • Secrets: External Secrets Operator (v0.20.1)
  • Storage: AWS EBS CSI Driver
  • Message Queue: RabbitMQ

Networking

  • VPC: Custom VPC with public and private subnets across 3 AZs
  • Load Balancing: Application Load Balancer (ALB)
  • DNS: Kubernetes DNS (CoreDNS)
  • Service Mesh: (Optional - can be added)

📦 Infrastructure Components

AWS Resources

  1. VPC & Networking

    • Custom VPC (10.0.0.0/16)
    • 3 Public subnets across availability zones
    • 3 Private subnets across availability zones
    • Internet Gateway for public subnet access
    • NAT Gateway for private subnet internet access
    • Route tables for public and private subnets

  2. EKS Cluster

    • Kubernetes version: 1.31
    • Node group with t3.small instances (2-5 nodes)
    • Managed node groups in private subnets
    • Public and private API endpoints
    • CloudWatch logging enabled

  3. ECR Repositories

    • microtodo/api-gateway
    • microtodo/users-service
    • microtodo/tasks-service
    • microtodo/notifications-service
    • Lifecycle policies (7-day untagged image expiration, keep last 30 tagged images)

  4. IAM Roles & Policies

    • EKS cluster role
    • EKS node group role
    • AWS Load Balancer Controller role
    • External Secrets Operator role
    • EBS CSI Driver role

  5. Secrets Management

    • AWS Secrets Manager for sensitive data
    • PostgreSQL credentials
    • RabbitMQ credentials
    • JWT secrets

Kubernetes Resources

  1. Namespaces

    • microtodo: Main application namespace
    • argocd: GitOps controller
    • external-secrets-system: Secrets management
    • kube-system: System components

  2. Services

    • API Gateway (entry point)
    • Users Service (authentication & user management)
    • Tasks Service (task management)
    • Notifications Service (notifications)

  3. Databases

    • PostgreSQL for Users (with persistent storage)
    • PostgreSQL for Tasks (with persistent storage)
    • PostgreSQL for Notifications (with persistent storage)
    • RabbitMQ (message broker)

  4. Ingress

    • ALB-based ingress for API Gateway
    • HTTP/HTTPS support
    • (Optional) SSL/TLS termination

✅ Prerequisites

Before you begin, ensure you have the following installed:

AWS Account Setup

  1. AWS account with appropriate permissions
  2. AWS CLI configured with credentials: 
    aws configure
  3. Note your AWS Account ID (you'll need this for Terraform)

🚀 Getting Started

Step 1: Bootstrap Terraform State Management

First, create the S3 bucket and DynamoDB table for Terraform state:

cd terraform-bootstrap
terraform init
terraform apply

Important: Note the outputs from this step:

  • terraform_state_bucket: Use this in terraform/terraform.tf backend configuration
  • terraform_lock_table_name: Use this in terraform/terraform.tf backend configuration

Step 2: Configure Terraform Variables

Create a terraform.tfvars file in the terraform/ directory:

aws_account_id = "your-aws-account-id"

# Database credentials
postgres_username = "postgres"
postgres_password = "your-secure-password"

# RabbitMQ credentials
rabbitmq_username = "rabbit"
rabbitmq_password = "your-secure-password"

# JWT secret
jwt_secret = "your-jwt-secret-key"

Security Note: Never commit terraform.tfvars to version control. It's already in .gitignore.

Step 3: Update Backend Configuration

Update the backend configuration in terraform/terraform.tf with the bucket name from Step 1:

backend "s3" {
  bucket         = "microtodo-tf-state-XXXXXX"  # From bootstrap output
  key            = "microtodo/terraform.tfstate"
  region         = "eu-west-2"
  dynamodb_table = "microtodo-tf-state-lock"
  encrypt        = true
}

Step 4: Deploy AWS Infrastructure

cd terraform
terraform init
terraform plan
terraform apply

This will create:

  • VPC and networking components
  • EKS cluster and node groups
  • ECR repositories
  • IAM roles and policies
  • Secrets in AWS Secrets Manager
  • Install Helm charts (ALB Controller, ArgoCD, External Secrets)

Note: This process takes approximately 15-20 minutes.

Step 5: Configure kubectl

After the EKS cluster is created, configure kubectl:

aws eks update-kubeconfig --region eu-west-2 --name microtodo_eks_cluster

Verify the connection:

kubectl get nodes
kubectl get namespaces

Step 6: Deploy Application Resources

Apply the Kubernetes manifests:

# Create namespace
kubectl apply -f k8s/namespaces/

# Deploy all resources
kubectl apply -f k8s/

Step 7 (Optional): Configure ArgoCD (GitOps)

  1. Get the ArgoCD admin password:

    kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

  2. Port-forward to access ArgoCD UI:

    kubectl port-forward svc/argocd-server -n argocd 8080:443

  3. Access ArgoCD at https://localhost:8080

    • Username: admin
    • Password: (from step 1)

Step 8: Access the Application

Get the Load Balancer URL:

kubectl get ingress -n microtodo api-gateway-ingress

The application will be available at the ADDRESS shown in the output.

📁 Project Structure

infrastructure/
├── docker-compose.dev.yml          # Local development environment
├── scripts/
│   └── download-alb-policy.sh      # Helper script for ALB policy
├── terraform-bootstrap/            # Terraform state backend setup
│   └── main.tf                     # S3 bucket and DynamoDB table
├── terraform/                      # Main infrastructure code
│   ├── main.tf                     # AWS provider configuration
│   ├── terraform.tf                # Terraform and backend configuration
│   ├── variables.tf                # Input variables
│   ├── locals.tf                   # Local values
│   ├── vpc.tf                      # VPC and networking
│   ├── eks.tf                      # EKS cluster and node groups
│   ├── ecr.tf                      # Container registries
│   ├── iam.tf                      # IAM roles and policies
│   ├── secrets.tf                  # AWS Secrets Manager
│   ├── helm-charts.tf              # Helm releases (ALB, ArgoCD, External Secrets)
│   └── iam-policies/
│       └── aws-load-balancer-controller-policy.json
└── k8s/                            # Kubernetes manifests
    ├── namespaces/
    │   └── microtodo.yaml          # Application namespace
    ├── storage/
    │   └── gp3-storageclass.yaml   # EBS GP3 storage class
    ├── external-secrets/
    │   ├── secret-store.yaml       # AWS Secrets Manager integration
    │   └── external-secret.yaml    # Secret definitions
    ├── databases/
    │   ├── postgres-users.yaml     # Users database
    │   ├── postgres-tasks.yaml     # Tasks database
    │   ├── postgres-notifications.yaml  # Notifications database
    │   └── rabbitmq.yaml           # Message broker
    ├── services/
    │   ├── api-gateway/
    │   │   ├── deployment.yaml
    │   │   ├── service.yaml
    │   │   └── configmap.yaml
    │   ├── users-service/
    │   │   ├── deployment.yaml
    │   │   ├── service.yaml
    │   │   ├── configmap.yaml
    │   │   └── migrate.yaml        # Database migration job
    │   ├── tasks-service/
    │   │   ├── deployment.yaml
    │   │   ├── service.yaml
    │   │   ├── configmap.yaml
    │   │   └── migrate.yaml
    │   └── notifications-service/
    │       ├── deployment.yaml
    │       ├── service.yaml
    │       ├── configmap.yaml
    │       └── migrate.yaml
    ├── ingress/
    │   └── api-gateway-ingress.yaml  # ALB ingress
    └── argocd/
        └── application.yaml          # ArgoCD application definition

📖 Deployment Guide

Manual Deployment

Follow the steps in Getting Started for initial deployment.

GitOps Deployment with ArgoCD

Once ArgoCD is configured:

  1. Any changes pushed to the master branch in the k8s/ directory will be automatically detected by ArgoCD
  2. ArgoCD will sync the changes to the cluster based on the sync policy (automated with pruning and self-healing)
  3. Monitor deployments in the ArgoCD UI

CI/CD Pipeline (Recommended for Production)

A typical CI/CD workflow:

  1. Build Phase:

    • Build Docker images for each service
    • Tag images with commit SHA or version
    • Push to ECR repositories

  2. Update Manifests:

    • Update image tags in Kubernetes deployments
    • Commit changes to infrastructure repository

  3. ArgoCD Sync:

    • ArgoCD detects changes
    • Applies updates to the cluster
    • Verifies deployment health

Example GitHub Actions workflow (pseudo-code):

name: Deploy Service
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Build and push Docker image
        # Build image and push to ECR
      
      - name: Update Kubernetes manifests
        # Update image tags in infrastructure repo
      
      - name: Wait for ArgoCD sync
        # Monitor ArgoCD for successful deployment

🔧 Services

API Gateway

  • Purpose: Single entry point for all client requests, handles routing to microservices
  • Port: 80 (internal), exposed via ALB
  • Health Checks: TCP-based health probes
  • Scaling: 2 replicas (configurable)

Users Service

  • Purpose: User authentication, registration, and profile management
  • Port: 3000
  • Database: PostgreSQL (postgres-users)
  • Features:
    • JWT-based authentication
    • User CRUD operations
    • Password hashing and validation
  • Health Checks: TCP-based startup, liveness, and readiness probes

Tasks Service

  • Purpose: Task management (create, read, update, delete tasks)
  • Port: 3001
  • Database: PostgreSQL (postgres-tasks)
  • Message Queue: RabbitMQ (publishes task events)
  • Features:
    • Task CRUD operations
    • Task assignment and status tracking
    • Event publishing for notifications

Notifications Service

  • Purpose: Handle notifications triggered by task events
  • Port: 3002
  • Database: PostgreSQL (postgres-notifications)
  • Message Queue: RabbitMQ (consumes task events)
  • Features:
    • Notification creation and delivery
    • Event-driven architecture
    • Notification history

🔒 Security

Network Security

  • VPC Isolation: Services run in private subnets
  • Security Groups: (Should be implemented for production)
  • NAT Gateway: Controlled internet access for private subnets
  • ALB: Public-facing load balancer with optional SSL/TLS

Secrets Management

  • AWS Secrets Manager: Centralized secret storage
  • External Secrets Operator: Syncs secrets to Kubernetes
  • IRSA (IAM Roles for Service Accounts): Fine-grained AWS permissions
  • Environment Variables: Secrets injected as env vars, never in code

Authentication & Authorization

  • JWT: Token-based authentication
  • Service-to-Service: Internal communication (consider mTLS for production)

Best Practices Implemented

✅ Secrets stored in AWS Secrets Manager, not in code
✅ Terraform state encrypted and stored remotely
✅ ECR image scanning (commented out for cost reasons)
✅ Database credentials rotated through AWS Secrets Manager
✅ Principle of least privilege for IAM roles
✅ Private subnets for workloads

Production Security Enhancements (TODO)

⚠️ Implement Network Policies for pod-to-pod communication
⚠️ Add Web Application Firewall (WAF) to ALB
⚠️ Enable SSL/TLS with AWS Certificate Manager
⚠️ Implement Security Groups for EKS
⚠️ Enable VPC Flow Logs
⚠️ Add Pod Security Standards/Policies
⚠️ Implement secret rotation
⚠️ Add vulnerability scanning for container images (ECR)

📊 Monitoring and Logging

Current Setup

  • EKS Control Plane Logs: Enabled for API, audit, authenticator, controller manager, and scheduler
  • kubectl logs: View container logs
# View service logs
kubectl logs -f deployment/users-service -n microtodo
kubectl logs -f deployment/tasks-service -n microtodo
kubectl logs -f deployment/notifications-service -n microtodo

# View pod events
kubectl describe pod <pod-name> -n microtodo

Recommended Additions for Production

  1. Metrics Collection:

    • Prometheus for metrics scraping
    • Grafana for visualization
    • Custom dashboards for business metrics

  2. Distributed Tracing:

    • Jaeger or AWS X-Ray
    • Trace requests across microservices

  3. Log Aggregation:

    • ELK Stack (Elasticsearch, Logstash, Kibana)
    • Fluentd or Fluent Bit for log collection
    • CloudWatch Logs Insights

  4. Alerting:

    • Alertmanager (with Prometheus)
    • PagerDuty integration
    • Slack notifications

💻 Development

Local Development with Docker Compose

For local development, use the provided Docker Compose file:

docker-compose -f docker-compose.dev.yml up -d

This starts:

  • RabbitMQ with management UI (ports 5672, 15672)

Access RabbitMQ Management UI:

Testing Kubernetes Manifests

Validate Kubernetes manifests before applying:

# Dry-run
kubectl apply -f k8s/ --dry-run=client

# Validate with kubeval (if installed)
kubeval k8s/**/*.yaml

### Terraform Development

```bash
cd terraform

# Format code
terraform fmt -recursive

# Validate configuration
terraform validate

# Plan changes
terraform plan

# Apply changes
terraform apply

Updating Service Images

  1. Build and push new image to ECR:

    # Authenticate to ECR
aws ecr get-login-password --region eu-west-2 | docker login --username AWS --password-stdin <account-id>.dkr.ecr.eu-west-2.amazonaws.com

# Build and push
docker build -t microtodo/users-service:v1.2.3 .
docker tag microtodo/users-service:v1.2.3 <account-id>.dkr.ecr.eu-west-2.amazonaws.com/microtodo/users-service:v1.2.3
docker push <account-id>.dkr.ecr.eu-west-2.amazonaws.com/microtodo/users-service:v1.2.3

  2. Update the deployment manifest:

    spec:
  containers:
    - name: users-service
      image: <account-id>.dkr.ecr.eu-west-2.amazonaws.com/microtodo/users-service:v1.2.3

  3. Commit and push (ArgoCD will auto-sync if configured)

🏭 Production Considerations

High Availability

  • Multi-AZ NAT Gateways: Currently using single NAT Gateway (cost optimization). For production, deploy NAT Gateway per AZ.
  • Database High Availability: Consider Amazon RDS for PostgreSQL with Multi-AZ deployment
  • RabbitMQ Cluster: Deploy RabbitMQ in cluster mode for HA
  • Pod Disruption Budgets: Ensure minimum replicas during voluntary disruptions
  • HPA (Horizontal Pod Autoscaler): Auto-scale based on CPU/memory/custom metrics

Performance Optimization

  • CDN: CloudFront for static assets
  • Caching: Redis/Memcached for frequently accessed data
  • Database Indexing: Optimize database queries with proper indexes
  • Connection Pooling: Implement efficient database connection pooling
  • Resource Limits: Fine-tune CPU/memory requests and limits

Cost Optimization

Current cost-saving measures:

  • Single NAT Gateway (vs. 3 per AZ)
  • Smaller EC2 instance types (t3.small)
  • ECR image scanning disabled
  • Public EKS endpoint (vs. private only with VPN)

Production cost considerations:

  • Use Reserved Instances or Savings Plans for predictable workloads
  • Right-size EC2 instances based on actual usage
  • Enable cluster autoscaler
  • Set up cost alerts and budgets in AWS
  • Review and optimize EBS volume sizes

Disaster Recovery

  • Backup Strategy: Regular backups of databases and persistent volumes
  • Cross-Region Replication: Replicate critical data to another region
  • Disaster Recovery Plan: Document and test DR procedures
  • RTO/RPO: Define Recovery Time Objective and Recovery Point Objective

Security Hardening

  • TLS/SSL: Enable HTTPS with valid certificates
  • Network Policies: Restrict pod-to-pod communication
  • Pod Security Standards: Enforce security contexts
  • Image Scanning: Enable ECR vulnerability scanning
  • Audit Logging: Comprehensive audit trail
  • WAF: Web Application Firewall for ALB
  • Penetration Testing: Regular security assessments

Compliance

  • Data Encryption: At-rest and in-transit encryption
  • GDPR Compliance: Data privacy and user rights
  • Access Logs: Maintain comprehensive access logs
  • Regular Audits: Compliance and security audits

🐛 Troubleshooting

Common Issues

1. Terraform State Lock

Problem: Error acquiring the state lock

Solution:

# Release the lock (use with caution!)
terraform force-unlock <lock-id>

2. EKS Cluster Access Denied

Problem: error: You must be logged in to the server (Unauthorized)

Solution:

# Update kubeconfig
aws eks update-kubeconfig --region eu-west-2 --name microtodo_eks_cluster

# Verify AWS CLI credentials
aws sts get-caller-identity

3. Pod Not Starting

Problem: Pods stuck in Pending, CrashLoopBackOff, or ImagePullBackOff

Solution:

# Check pod events
kubectl describe pod <pod-name> -n microtodo

# Check logs
kubectl logs <pod-name> -n microtodo

# Common fixes:
# - Verify ECR authentication
# - Check resource requests/limits
# - Verify secrets are populated
kubectl get secrets -n microtodo
kubectl describe externalsecret microtodo-secrets -n microtodo

4. Database Connection Issues

Problem: Services can't connect to databases

Solution:

# Verify database pods are running
kubectl get pods -n microtodo | grep postgres

# Check database service
kubectl get svc -n microtodo | grep postgres

# Verify secrets
kubectl get secret microtodo-secrets -n microtodo -o yaml

# Test connection from a debug pod
kubectl run -it --rm debug --image=postgres:15 --restart=Never -n microtodo -- psql -h postgres-users-service -U postgres

5. ArgoCD Out of Sync

Problem: ArgoCD shows application as OutOfSync

Solution:

# Manual sync via CLI
argocd app sync microtodo-app

# Or via kubectl
kubectl -n argocd patch app microtodo-app -p '{"operation":{"sync":{}}}' --type merge

# Check sync status
argocd app get microtodo-app

6. Load Balancer Not Created

Problem: Ingress doesn't provision ALB

Solution:

# Check ALB controller logs
kubectl logs -n kube-system deployment/aws-load-balancer-controller

# Verify ingress class
kubectl get ingressclass

# Check IAM permissions for ALB controller
kubectl describe serviceaccount aws-load-balancer-controller -n kube-system

Useful Commands

# Check cluster health
kubectl get nodes
kubectl get componentstatuses

# View all resources in microtodo namespace
kubectl get all -n microtodo

# Check resource usage
kubectl top nodes
kubectl top pods -n microtodo

# View events
kubectl get events -n microtodo --sort-by='.lastTimestamp'

# Port forward to a service
kubectl port-forward svc/<service-name> -n microtodo <local-port>:<service-port>

# Execute command in pod
kubectl exec -it <pod-name> -n microtodo -- /bin/sh

# View Terraform state
cd terraform
terraform state list
terraform state show <resource>

# ArgoCD password recovery
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

📚 Additional Resources

Documentation

Tutorials

Note: This infrastructure is set up for learning and development purposes. For production deployments, review and implement the security hardening, high availability, and monitoring recommendations outlined in this document.

About

Infrastructure for the microtodo backend

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages