Many Databases 1 LSM Engine - OpenData
Summary
Host KV interviews Almog Gavra, co-founder of Responsive, about Open Data and SlateDB. They discuss unifying vector, time series, and log databases on a single object-storage-native LSM foundation to reduce operational costs. The episode covers how abstracting distributed systems complexity allows multiple database types to share tuning knobs and failure modes.
Topics discussed
Introduction to Almok Gavra and Responsive
Background at LinkedIn and Kafka Streams challenges
Origin of SlateDB and the operational tax of distributed systems
The economic and operational benefits of object storage
Amortizing operational expertise across multiple data systems
Mental model: Specialized databases vs. Open Data infrastructure
SlateDB architecture: Object-native LSM tree foundation
Open Data project: Mapping diverse workloads to key-value storage
Tuning LSM trees for different workloads (Time Series vs Vector)
The Cloud Triad: Balancing latency, cost, and durability
Cache strategies and reader/writer separation in Open Data
Single writer simplicity and stateless ingest patterns
Failure modes, fencing, and simplified recovery
Compaction strategies and snapshot management
Conclusion: When to use Open Data vs traditional databases
Listen ad-free on Castria