PiLSMer
A data-free key-value store. Pi for storage. Regret for reads.
PiLSMer writes your data normally, then uses planning or compaction to replace stored values with instructions for finding equivalent byte sequences inside a deterministic stream. Reads still work. Everything else gets worse.
The value survives. The storage does not.
The canonical demo writes a JSON invoice, explains the raw envelope, rewrites it into a plan, explains the damage, and reads the original bytes back.
$ cargo run -q --bin pilsmer -- init /tmp/pilsmer-demo/db $ printf '{"total":49.99,"status":"paid"}' > /tmp/pilsmer-demo/invoice.json $ cargo run -q --bin pilsmer -- put /tmp/pilsmer-demo/db invoice:123 /tmp/pilsmer-demo/invoice.json $ cargo run -q --bin pilsmer -- explain /tmp/pilsmer-demo/db invoice:123 storage_class: Raw logical_user_bytes: 31 physical_value_bytes: 56 $ cargo run -q --bin pilsmer -- plan-key /tmp/pilsmer-demo/db invoice:123 planned: invoice:123 $ cargo run -q --bin pilsmer -- get /tmp/pilsmer-demo/db invoice:123 {"total":49.99,"status":"paid"}
After planning
Compaction into nonexistence.
PiLSMer keeps SlateDB as the storage engine and moves the absurdity into envelopes, stream indexes, planner output, reconstruction, and the compaction filter.
The joke has tests.
The public surface is deadpan, but the repo has the normal boring pieces: codecs, decode limits, reconstruction checks, metrics, and CI.
Envelope codec
Raw bytes and reconstruction plans are framed as typed PiLSMer values before SlateDB sees them.
envelope.rsPlanner
Dynamic programming converts logical bytes into located chunks inside a deterministic stream prefix.
planner.rsReconstruction
Reads rebuild bytes from plan metadata and verify the logical hash before returning user data.
reconstruct.rsSlateDB wrapper
All application reads and writes go through the wrapper. Direct SlateDB reads return envelopes.
pilsmer-slateCompaction filter
Raw values can be rewritten into plans during app-level planning, vacuuming, or compaction.
compaction_filter.rsCI
The repository checks formatting, all targets, and tests through the Rust workflow.
ci.ymlEverything else gets worse.
The metrics are intentionally incriminating. The system can report zero philosophical user bytes stored while physical metadata grows.
$ cargo run -q --bin pilsmer -- metrics /tmp/pilsmer-demo/db
pilsmer_raw_values_total 1
pilsmer_planned_values_total 2
pilsmer_logical_bytes_total 98
pilsmer_physical_value_bytes_total 3184
pilsmer_plan_metadata_bytes_total 2960
pilsmer_stream_prefix_bytes_indexed 65536
pilsmer_metadata_amplification_ratio 30.20
pilsmer_philosophical_compression_ratio infinity
pilsmer_philosophical_purity_violations_total 0
Useful damage
| Metric | Value |
|---|---|
| raw values | 1 |
| planned values | 2 |
| logical bytes | 98 |
| physical value bytes | 3,184 |
| plan metadata bytes | 2,960 |
| stream prefix indexed | 64 KiB |
| metadata amplification | 30.20x |
| purity violations | 0 |
Properties of a worse storage engine.
SlateDB-backed LSM
PiLSMer keeps the object-storage-native LSM and wraps the value semantics.
Value envelopes
Raw and planned values use typed envelopes so direct engine reads stay honest.
Stream registry
Plans refer to versioned stream identities and fingerprints, not a private cache file.
Dynamic planner
The planner chooses located chunks across indexed deterministic stream prefixes.
Hash verification
Reconstruction checks logical hashes before bytes leave the wrapper.
Guarded planning
`plan-key` rewrites a single key after checking the source envelope has not changed.
VACUUM MEANING
Existing values can be replanned under a budget to improve the shape of regret.
Standalone compactor
The compactor path makes compaction into nonexistence observable in demos.