LmdbJava Benchmarks

Just want the latest results?

• Library Comparisons: Full Benchmark | Smoketest
• Version Regressions: Full Benchmark | Smoketest
• Historical: 2016 Results

This is a JMH benchmark of open source, embedded, memory-mapped, key-value stores available from Java:

• LmdbJava (with fast ByteBuffer, safe ByteBuffer and an Agrona buffer)
• LMDBJNI
• Lightweight Java Game Library (LMDB API)
• LevelDBJNI
• RocksDB
• MVStore (pure Java)
• MapDB (pure Java)
• Xodus (pure Java)
• Chroncile Map (pure Java) (**)

(**) does not support ordered keys, so iteration benchmarks not performed

The benchmark itself is adapted from LMDB's db_bench_mdb.cc, which in turn is adapted from LevelDB's benchmark.

The benchmark includes:

• Writing data
• Reading all data via each key
• Reading all data via a reverse iterator
• Reading all data via a forward iterator
• Reading all data via a forward iterator and computing a CRC32 (via JDK API)
• Reading all data via a forward iterator and computing a XXH32 hash

Byte arrays (byte[]) are always used for the keys and values, avoiding any serialization library overhead. For those libraries that support compression, it is disabled in the benchmark. In general any special library features that decrease latency (eg batch modes, disable auto-commit, disable journals, hint at expected data sizes etc) were used. While we have tried to be fair and consistent, some libraries offer non-obvious tuning settings or usage patterns that might further reduce their latency. We do not claim we have exhausted every tuning option every library exposes, but pull requests are most welcome.

Build

Clone this repository and build:

mvn clean package

Usage

This benchmark uses POSIX calls to accurately determine consumed disk space and only depends on Linux-specific native library wrappers where a range of such wrappers exists. Operation on non-Linux operating systems is unsupported.

Running Benchmarks

Library Comparison Benchmarks

Use the run-libs.sh script to compare different key-value store libraries:

# Quick smoke test (1K entries, fast verification)
./run-libs.sh smoketest

# Full benchmark using 25% of system RAM (default)
./run-libs.sh benchmark

# Full benchmark using 50% of system RAM
./run-libs.sh benchmark 50

The benchmark auto-scales based on available RAM and caps at 1 million entries. Results are written to target/benchmark-libs/.

Version Regression Testing

Use the run-vers.sh script to test LmdbJava performance across versions:

# Quick smoke test (1K entries, fast verification)
./run-vers.sh smoketest

# Full benchmark using 25% of system RAM (default)
./run-vers.sh benchmark

# Full benchmark using 50% of system RAM
./run-vers.sh benchmark 50

This tests selected LmdbJava versions from Maven Central plus current development branches to identify performance regressions. Results are written to target/benchmark-vers/.

Running Both Benchmark Suites

Use the run-both.sh script to run both library and version benchmarks sequentially (designed for overnight runs):

# Run both benchmarks using 25% of system RAM (default)
./run-both.sh

# Run both benchmarks using 50% of system RAM
./run-both.sh 50

This will run library comparison benchmarks followed by version regression benchmarks, both in full benchmark mode with 120s iterations. Expect several hours of runtime depending on your system.

Generating Reports

After running library comparison benchmarks, generate a comprehensive report:

./report-libs.sh

After running version regression tests, generate a version comparison report:

./report-vers.sh

Reports generate:

• target/benchmark/README.md - Full markdown report with charts
• target/benchmark/index.html - HTML viewer with embedded charts (open in browser)
• Various SVG charts and supporting files

Publishing Reports

After generating a report, you can publish it to Cloudflare Pages:

export CLOUDFLARE_API_TOKEN="your-token"
export CLOUDFLARE_ACCOUNT_ID="your-account-id"
./publish-results.sh

The script automatically detects:

• Type: Library comparison or version regression (from README heading)
• Mode: Smoketest (3s) or benchmark (120s) (from smoketest warning)

Reports are published to:

• libraries-smoketest.lmdbjava.org - Fast daily library verification
• libraries-benchmark.lmdbjava.org - Full library comparison analysis
• versions-smoketest.lmdbjava.org - Fast daily version monitoring
• versions-benchmark.lmdbjava.org - Full version regression analysis

Workflow for curated reports:

1. Run full benchmarks: ./run-both.sh (or individually with ./run-libs.sh benchmark and ./run-vers.sh benchmark)
2. Generate reports: ./report-libs.sh and ./report-vers.sh
3. Review and edit commentary in report scripts if needed, then re-run
4. Publish: ./publish-results.sh (run twice, once after each report generation)

Version Management

Update all dependency and plugin versions:

mvn versions:update-properties

Support

Issues are disabled for this repository. Please report any issues or questions on the LmdbJava issue tracker.

Contributing

Contributions are welcome! Please see the LmdbJava project's Contributing Guidelines.

License

This project is licensed under the Apache License, Version 2.0.