Memory Consolidation System Guide

Version: 8.23.0+ | Last Updated: 2025-11-11 | Status: Production Ready

Overview

The Memory Consolidation System provides dream-inspired automatic memory maintenance with exponential decay scoring, creative association discovery, semantic clustering, compression, and controlled forgetting. As of v8.23.0, the consolidation scheduler runs 24/7 in the HTTP server, independent of MCP server or Claude Desktop.

What Consolidation Does

**Exponential Decay Scoringa: Prioritizes recent and frequently accessed memories
Creative Association Discovery: Finds semantic connections between memories (0.3-0.7 similarity)
Semantic Clustering: Groups related memories using DBSCAN algorithm
Compression: Summarizes redundant information while preserving originals
Controlled Forgetting: Archives low-relevance memories (90+ days inactive)

Architecture (v8.23.0+)

Consolidation Scheduler Location: HTTP Server

Runs 24/7 with HTTP server (independent of MCP server/Claude Desktop)
Uses APScheduler for automatic scheduling
Accessible via both HTTP API and MCP tools
Benefits: Persistent, reliable, no dependency on Claude Desktop being open

Token-Efficient Code Execution API

from mcp_memory_service.api import consolidate, scheduler_status

# Trigger consolidation (15 tokens vs 150 MCP tool - 90% reduction)
result = consolidate('weekly')

# Check scheduler (10 tokens vs 125 - 92% reduction)
status = scheduler_status()

Quick System Status Check

Verify Consolidation Scheduler is Running

# Check HTTP server status
systemctl --user status mcp-memory-http.service

# Health check
curl http://127.0.0.1:8000/api/health

# Scheduler status
curl http://127.0.0.1:8000/api/consolidation/status

Expected Output:

{
  "running": true,
  "next_daily": "2025-11-12T02:00:00",
  "next_weekly": "2025-11-16T03:00:00",
  "next_monthly": "2025-12-01T04:00:00",
  "jobs_executed": 0,
  "jobs_failed": 0
}

Configuration Check

# Verify consolidation is enabled
grep MCP_CONSOLIDATION_ENABLED .env
# Should output: MCP_CONSOLIDATION_ENABLED=true

# Verify HTTP server is enabled
grep MCP_HTTP_ENABLED .env
# Should output: MCP_HTTP_ENABLED=true

Real-World Performance Metrics

Based on actual test run: 2025-11-11, mcp-memory-service v8.23.1

Test Environment

Memory Count: 2,495 memories
Storage Backend: Hybrid (SQLite-vec + Cloudflare)
System: Linux 6.14.0-35-generic
Time Horizon: Weekly consolidation

Performance Results

Metric	Estimated	Actual	Notes
Processing Time	~25 seconds	4-6 minutes	First run with hybrid backend
Memory Updates	N/A	2,495 processed	All memories analyzed
Cloud Sync	N/A	Continuous	Cloudflare synchronization throughout

Why First Run Takes Longer

Hybrid Backend Impact:

Local SQLite operations: Fast (~5ms per memory)
Cloudflare synchronization: Network-dependent (~150ms per update)
Trade-off: Longer processing time for multi-device data persistence

Observed in Logs:

2025-11-11 11:14:08 mcp-memory-http[2283]: Successfully updated memory metadata: 735d2920...
2025-11-11 11:14:08 httpx - INFO - HTTP Request: POST https://api.cloudflare.com/.../query "HTTP/1.1 200 OK"

Performance by Backend

Backend	First Run	Subsequent Runs	Trade-offs
SQLite-Vec	5-25s	5-25s	Fast, local-only, single-device
Cloudflare	2-4min	1-3min	Network-dependent, cloud-only, multi-device
Hybrid	4-6min	2-4min	Slower, but local speed + cloud persistence

Recommendation: Hybrid backend is recommended for production despite longer processing time - the multi-device sync capability is worth the trade-off.

Manual Consolidation Methods

Method 1: HTTP API (Recommended for Manual Triggers)

Trigger Consolidation:

curl -X POST http://127.0.0.1:8000/api/consolidation/trigger \
  -H "Content-Type: application/json" \
  -d '{"time_horizon": "weekly", "immediate": true}'

Time horizons: daily, weekly, monthly, quarterly, yearly

Check Status:

curl http://127.0.0.1:8000/api/consolidation/status

Get Recommendations:

curl http://127.0.0.1:8000/api/consolidation/recommendations/weekly

Example Response:

{
  "recommendation": "consolidation_beneficial",
  "memory_count": 2495,
  "reasons": [
    "Consider running compression to reduce memory usage",
    "Many old memories present - consider forgetting/archival",
    "Good candidate for association discovery"
  ],
  "estimated_duration": 24.95
}

Method 2: MCP Tools (Available in Claude Code/Desktop)

# Trigger consolidation
mcp__memory__trigger_consolidation(time_horizon="weekly", immediate=true)

# Check status
mcp__memory__scheduler_status()

# Get recommendations
mcp__memory__consolidation_recommendations(time_horizon="weekly")

Method 3: Python Code Execution API (Most Token-Efficient)

from mcp_memory_service.api import consolidate, scheduler_status, consolidation_recommendations

# Trigger consolidation (15 tokens vs 150 MCP tool - 90% reduction)
result = consolidate('weekly')

# Check scheduler (10 tokens vs 125 - 92% reduction)
status = scheduler_status()

# Get recommendations
recommendations = consolidation_recommendations('weekly')

Monitoring and Troubleshooting

Real-Time Monitoring

Watch consolidation progress:

journalctl --user -u mcp-memory-http.service -f | grep consolidation

Check recent consolidation activity:

journalctl --user -u mcp-memory-http.service --since "10 minutes ago" | grep -i "consolidation\|complete\|finished"

Log Patterns

During consolidation (Hybrid backend):

INFO - Successfully updated memory metadata: 735d2920...
INFO - HTTP Request: POST https://api.cloudflare.com/.../query "200 OK"
INFO - Successfully updated memory metadata: 966c7015...
INFO - HTTP Request: POST https://api.cloudflare.com/.../query "200 OK"

Consolidation start:

INFO - Starting weekly consolidation...
INFO - Processing 2495 memories...

Consolidation complete:

INFO - Weekly consolidation completed in 245.3 seconds
INFO - Processed: 2495, Updated: 1230, Archived: 45, Compressed: 12

Consolidation Reports

Reports are saved to:

~/.local/share/mcp-memory-service/consolidation/reports/

Report Structure:

consolidation_weekly_2025-11-12_02-00-00.json
├── timestamp
├── time_horizon
├── duration_seconds
├── memories_processed
├── memories_updated
├── memories_archived
├── associations_discovered
└── compression_saved_bytes

Performance Considerations

SQLite-Vec Backend (Fast):

Processing: 5-25 seconds
Memory updates: <5ms each
No network latency
Best for: Single-device, speed-critical applications

Cloudflare Backend (Network-dependent):

Processing: 2-4 minutes
Memory updates: ~150ms each (network)
Cloud-only storage
Best for: Cloud-only deployments, team collaboration

Hybrid Backend (Recommended):

Processing: 4-6 minutes (first run), 2-4 minutes (subsequent)
Memory updates: Local fast + cloud sync
Multi-device persistence
Best for: Production, multi-device access, data durability

Automatic Scheduling

Default Schedule

Time Horizon	Schedule	Next Run Example	Purpose
Daily	Every day at 02:00	2025-11-12 02:00	Process recent memories, decay scoring
Weekly	Every Sunday at 03:00	2025-11-16 03:00	Broader pattern discovery, association discovery
Monthly	1st of month at 04:00	2025-12-01 04:00	Long-term consolidation, archival
Quarterly	Disabled by default	-	Strategic review (enable if needed)
Yearly	Disabled by default	-	Annual cleanup (enable if needed)

Configuration

Schedule configuration (src/mcp_memory_service/config.py):

CONSOLIDATION_SCHEDULE = {
    'daily': '02:00',              # Daily at 2 AM
    'weekly': 'SUN 03:00',         # Weekly on Sunday at 3 AM
    'monthly': '01 04:00',         # Monthly on 1st at 4 AM
    'quarterly': 'disabled',       # Disabled
    'yearly': 'disabled'           # Disabled
}

Enable/disable consolidation (.env):

# Enable consolidation (default: true)
MCP_CONSOLIDATION_ENABLED=true

# Enable HTTP server (required for consolidation)
MCP_HTTP_ENABLED=true

Check Next Scheduled Runs

curl http://127.0.0.1:8000/api/consolidation/status | jq '.next_daily, .next_weekly, .next_monthly'

Output:

"2025-11-12T02:00:00"
"2025-11-16T03:00:00"
"2025-12-01T04:00:00"

Best Practices

First Consolidation Run

✅ Plan for longer processing: 5-10 minutes with hybrid backend, large datasets (2000+ memories)

✅ Monitor first run: Watch logs to understand your system's performance baseline

journalctl --user -u mcp-memory-http.service -f | grep consolidation

✅ Check reports: Verify consolidation report was generated after first run

ls -lh ~/.local/share/mcp-memory-service/consolidation/reports/

✅ Verify improvements: Search should be faster after consolidation

Ongoing Maintenance

✅ Let scheduler run automatically: Manual triggers mainly for testing

✅ Monitor scheduler health: Weekly check of scheduler status

curl http://127.0.0.1:8000/api/consolidation/status

✅ Review reports periodically: Check what consolidation is doing

cat ~/.local/share/mcp-memory-service/consolidation/reports/consolidation_weekly_*.json | jq

✅ Adjust schedule if needed: Modify schedule in config.py for your usage patterns

Troubleshooting

❌ Consolidation not running:

# Check HTTP server is running
systemctl --user status mcp-memory-http.service

# Check consolidation is enabled
grep MCP_CONSOLIDATION_ENABLED .env

# Restart server if needed
systemctl --user restart mcp-memory-http.service

❌ Consolidation takes too long:

Check storage backend (hybrid is slower but more durable)
Consider SQLite-only for single-device speed
Verify network connectivity for Cloudflare backend

❌ No reports generated:

# Check report directory exists
mkdir -p ~/.local/share/mcp-memory-service/consolidation/reports/

# Check server logs for errors
journalctl --user -u mcp-memory-http.service | grep -i error

Migration from MCP-only Mode (v8.22.x → v8.23.0+)

No action required - Consolidation automatically runs in HTTP server if enabled.

For users without HTTP server:

# Enable HTTP server in .env
echo "MCP_HTTP_ENABLED=true" >> .env

# Restart service
systemctl --user restart mcp-memory-http.service

# Verify consolidation scheduler
curl http://127.0.0.1:8000/api/consolidation/status

MCP tools continue working (backward compatible via internal API calls).

API Reference

HTTP API Endpoints

Endpoint	Method	Description	Response Time
`/api/consolidation/trigger`	POST	Trigger consolidation	~10-30s initial response
`/api/consolidation/status`	GET	Scheduler status	<5ms
`/api/consolidation/recommendations/{horizon}`	GET	Get recommendations	~50ms

Code Execution API

# Import functions
from mcp_memory_service.api import (
    consolidate,                        # Trigger consolidation
    scheduler_status,                   # Check scheduler
    consolidation_recommendations,      # Get recommendations
    pause_consolidation,                # Pause scheduler
    resume_consolidation                # Resume scheduler
)

# Trigger consolidation
result = consolidate('weekly')
# Returns: {'status': 'started', 'time_horizon': 'weekly', 'estimated_duration': 24.95}

# Check scheduler status
status = scheduler_status()
# Returns: {'running': True, 'next_daily': '2025-11-12T02:00:00', ...}

# Get recommendations
recs = consolidation_recommendations('weekly')
# Returns: {'recommendation': 'consolidation_beneficial', 'memory_count': 2495, ...}

# Pause consolidation (maintenance mode)
pause_consolidation()  # Pause all jobs
pause_consolidation('weekly')  # Pause specific time horizon

# Resume consolidation
resume_consolidation()  # Resume all jobs
resume_consolidation('weekly')  # Resume specific time horizon

Summary

The Memory Consolidation System provides automatic, intelligent memory maintenance with minimal manual intervention. As of v8.23.0, it runs reliably in the HTTP server with automatic scheduling. The hybrid backend provides the best balance of performance and data persistence, with first-run processing of 4-6 minutes for large datasets being an acceptable trade-off for multi-device synchronization.

Key Takeaways:

✅ Consolidation runs automatically - no manual intervention needed
✅ First run with hybrid backend: 4-6 minutes (normal, expected)
✅ Three manual trigger methods available for testing
✅ Monitor via logs, check reports for detailed results
✅ Hybrid backend recommended for production despite longer processing time

Related Documentation:

Uh oh!

Memory Consolidation System Guide

Memory Consolidation System Guide

Overview

What Consolidation Does

Architecture (v8.23.0+)

Token-Efficient Code Execution API

Quick System Status Check

Verify Consolidation Scheduler is Running

Configuration Check

Real-World Performance Metrics

Test Environment

Performance Results

Why First Run Takes Longer

Performance by Backend

Manual Consolidation Methods

Method 1: HTTP API (Recommended for Manual Triggers)

Method 2: MCP Tools (Available in Claude Code/Desktop)

Method 3: Python Code Execution API (Most Token-Efficient)

Monitoring and Troubleshooting

Real-Time Monitoring

Log Patterns

Consolidation Reports

Performance Considerations

Automatic Scheduling

Default Schedule

Configuration

Check Next Scheduled Runs

Best Practices

First Consolidation Run

Ongoing Maintenance

Troubleshooting

Migration from MCP-only Mode (v8.22.x → v8.23.0+)

API Reference

HTTP API Endpoints

Code Execution API

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally