Distributed File System (DFS)

A distributed file system implementation in Go that demonstrates distributed systems concepts including consensus protocols, distributed locking, and fault-tolerant storage.

Features

File Operations: Create, read, write, delete files and directories
Distributed Coordination: Paxos consensus algorithm for replication
Distributed Locking: Concurrent access control across nodes
Caching: Local caching for improved performance
Fault Tolerance: Replication and consensus protocols

Architecture

The system consists of several microservices:

DFS Service: Main file system interface
Extent Service: Handles file data storage
Lock Service: Manages distributed locking
Paxos Service: Implements consensus protocol
Management Service: Node health monitoring

Recent Improvements

Critical Bug Fixes

Bounds Checking: Fixed potential panics when accessing empty strings
Variable Shadowing: Resolved variable redeclaration bugs
Error Handling: Improved error propagation and handling throughout
gRPC Connections: Fixed incorrect grpc.NewClient usage to grpc.Dial
Resource Management: Better connection cleanup and error handling

Code Quality Improvements

Input Validation: Added proper validation for all inputs
Error Messages: More descriptive error messages with context
Logging: Improved logging with structured information
Code Structure: Refactored long functions into smaller, focused ones
Documentation: Added comprehensive comments and documentation
Configuration: Centralized configuration management

Performance Improvements

Read-Write Mutex: Upgraded to RWMutex for better concurrent access
Context Management: Proper context cancellation and timeouts
Connection Pooling: Better connection management

Quick Start

1. Setup Development Environment

./scripts/setup-dev.sh

# Or manually
make deps
make all

2. Start All Services

./scripts/start-services.sh start

### 3. Monitor Services

```bash
# Check status
./scripts/start-services.sh status

# View logs
./scripts/start-services.sh logs

# Stop services
./scripts/start-services.sh stop

Scripts Overview

Service Management Scripts

`start-services.sh` (Linux/macOS)

Comprehensive service management script with the following commands:

start - Start all DFS services in the correct order
stop - Stop all services gracefully
restart - Restart all services
status - Show status of all services
logs - View service logs
clean - Clean up log files and PID files

Usage Examples:

# Start with default configuration
./scripts/start-services.sh start

# Start with custom ports
./scripts/start-services.sh start -p 3001 -e 3002 -l 2001

# Check service status
./scripts/start-services.sh status

# View logs for a specific service
./scripts/start-services.sh logs dfs

Development Scripts

`setup-dev.sh`

Sets up the complete development environment:

Checks and installs required tools (Go, protoc, etc.)
Sets up Go modules and dependencies
Generates protocol buffer code
Builds the project
Creates development configuration
Sets up Git hooks

Usage:

# Full setup
./scripts/setup-dev.sh

# Skip optional steps
./scripts/setup-dev.sh --skip

# Force setup (overwrite existing)
./scripts/setup-dev.sh --force

Configuration

Environment Variables

The system can be configured via environment variables:

# Service ports
export DFS_PORT=2001
export EXTENT_PORT=2002
export LOCK_PORT=1001

# Service addresses
export EXTENT_SERVICE=127.0.0.1:2002
export LOCK_SERVICE=127.0.0.1:1001

# Development settings
export DFS_DEBUG=true
export DFS_LOG_LEVEL=debug

Configuration File

A .env file is created during setup with default values:

# DFS Development Environment Configuration
DFS_PORT=2001
EXTENT_PORT=2002
LOCK_PORT=1001
REPLICA_LOCK_PORT=1002

EXTENT_SERVICE=127.0.0.1:2002
LOCK_SERVICE=127.0.0.1:1001

# Development settings
DFS_DEBUG=true
DFS_LOG_LEVEL=debug
DFS_MAX_CACHE_SIZE=1000
DFS_CACHE_TTL=300

Getting Started

Prerequisites

Go 1.23.1 or later
Protocol Buffers compiler (protoc)
Make (optional, for build automation)

Building

# Build all services
make all

# Build specific service
make dfsService
make extentService
make lockService

# Build with race detection
make race

# Build with debug symbols
make debug

# Clean build artifacts
make clean

Running Services

# Start all services using the script (recommended)
./scripts/start-services.sh start

# Or manually
./bin/extentService &
./bin/lockService &
./bin/dfsService -extentService=127.0.0.1:2002 -lockService=127.0.0.1:1001 &

API

The system exposes gRPC services for:

DFS Service: File operations (get, put, delete, mkdir, rmdir, dir)
Extent Service: Raw file data storage
Lock Service: Distributed locking (acquire, release)
Paxos Service: Consensus protocol (prepare, accept, decide)

Development

Project Structure

dfs/
├── cmd/                    # Service executables
│   ├── dfsService/        # Main DFS service
│   ├── extentService/     # File storage service
│   └── lockService/       # Distributed locking service
├── pkg/                   # Shared packages
│   ├── core/             # Core utilities and config
│   ├── paxos/            # Paxos consensus implementation
│   └── *Proto/           # Protocol buffer definitions
├── scripts/               # Build and deployment scripts
│   ├── start-services.sh  # Service management (Linux/macOS)
│   ├── start-services.ps1 # Service management (Windows)
│   ├── setup-dev.sh      # Development environment setup
│   └── test-client.sh    # Test client
└── logs/                  # Service logs (created at runtime)

Development Workflow

Setup Environment: ./scripts/setup-dev.sh
Make Changes: Edit code as needed
Build: make all
Test: ./scripts/test-client.sh test
Run: ./scripts/start-services.sh start
Monitor: ./scripts/start-services.sh logs
Stop: ./scripts/start-services.sh stop

Code Quality

The project follows Go best practices:

Proper error handling and propagation
Input validation and sanitization
Comprehensive logging
Clean separation of concerns
Proper resource management

Git Hooks

A pre-commit hook is automatically installed that:

Formats Go code using go fmt
Ensures all code is properly formatted before commits

Troubleshooting

Common Issues

Port Already in Use

# Check what's using the port
lsof -i :2001

# Kill the process
kill -9 <PID>

Service Won't Start

# Check logs
./scripts/start-services.sh logs dfs

# Check if dependencies are running
./scripts/start-services.sh status

Build Failures

# Clean and rebuild
make clean
make all

# Check Go version
go version

Protocol Buffer Issues

# Regenerate protobuf code
make proto

# Or use the setup script
./scripts/setup-dev.sh

Log Files

Logs are stored in the logs/ directory:

dfs.log - DFS service logs
extent.log - Extent service logs
lock.log - Lock service logs

Service Status

Check service status with:

./scripts/start-services.sh status

Contributing

Fork the repository
Create a feature branch
Make your changes
Add tests for new functionality
Ensure all tests pass
Submit a pull request

License

This is a hobby project for educational purposes.

Acknowledgments

Built with Go and gRPC
Implements Paxos consensus algorithm
Demonstrates distributed systems concepts

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
cmd		cmd
pkg		pkg
scripts		scripts
.gitignore		.gitignore
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum

pop-eax/DFS

Folders and files

Latest commit

History

Repository files navigation