Search

Find posts by keyword, tag, or both.

NoteSeptember 16, 20264 min read

Visualizing DataFusion physical plans

A staycation project that turns DataFusion EXPLAIN output into an Excalidraw diagram — paste a plan, click Visualize, and see sort order, pushdown, and parallel streams in color.

EssaySeptember 13, 20267 min read

(Almost) linear scaling of a star-schema query with Distributed DataFusion

How a two-table product/sales query — sales every 30 minutes for grocery products — stays streaming on one node and scales near-linearly across workers.

NoteAugust 3, 20261 min read

A week at Dagstuhl on robust queries and stream processing

Notes from Seminar 26311 — a useful week of industry and academia on hard database problems, and a small-group project I plan to keep building.

EssayNovember 14, 20241 min read

Optimizing queries in InfluxDB 3 using progressive evaluation

Which SQL qualifies for ProgressiveEvalExec, how to see it in EXPLAIN, and how mixed overlapping files are still evaluated newest-first.

EssayMarch 18, 20241 min read

Making most-recent-value queries hundreds of times faster

How databases optimize queries — and how a ProgressiveEval operator made most-recent-value queries in InfluxDB 3 hundreds of times faster.

EssayJanuary 29, 20241 min read

How to read InfluxDB 3 query plans

How to read a query plan in InfluxDB 3.0 — logical versus physical plans, EXPLAIN, and a leading-edge example.

EssayJanuary 19, 20241 min read

Partitioning data for query performance in InfluxDB 3

Partitioning schemes supported by InfluxDB 3.0 and the trade-offs between large and small numbers of partitions.

EssayJune 27, 20231 min read

InfluxDB 3 system architecture

The architecture of InfluxDB IOx (InfluxDB 3.0) — a cloud-scalable database for high-performance load and query on time series.

EssayMarch 27, 20231 min read

Compactor — a hidden engine of database performance

A hidden database server that handles post-ingestion and pre-query workloads so ingest and query do not share CPU and memory with compaction.

EssayMarch 13, 20231 min read

Using deduplication for eventually consistent transactions

Deduplication as an alternative to transactions for eventually consistent use cases of a distributed database with defined primary keys.

EssayNovember 18, 20221 min read

Partitioning for performance in a sharding database system

How partitioning inside a shard adds pruning, intra-node parallelism, and fast deletion, and how to manage hot versus cold partitions.

EssayNovember 16, 20221 min read

Scaling throughput and performance in a sharding database system

Understand the two dimensions of scaling for database query and ingest workloads, and how sharding can make scaling elastic — or not.

EssayFebruary 1, 202114 min read

Distributed cloud columnar databases: Vertica Eon vs Snowflake

How Vertica Eon and Snowflake both separate storage from compute — and where their architectures, caches, and scaling models diverge.