Specialized Parquet Compression for Databases

ISEF Category: Systems Software

Subcategory: Databases  ·  Difficulty: Advanced  ·  Setup: University Lab  ·  Time: Full Year

The Hook

A database file can waste more space than it needs to, just because every column gets treated the same. But a timestamp column does not behave like a user ID column, and a category list does not behave like either one. If you match the compression method to the data pattern, you can often shrink files without slowing reads much. That tradeoff is the heart of this project.

What Is It?

This project studies compression for Parquet files, a common format for storing table-like data. Compression means turning data into a smaller file. The catch is that not all columns behave the same way. Some columns are almost sorted, like timestamps. Some are messy and repetitive, like user IDs that follow a Zipfian pattern, where a few values appear a lot and many appear rarely. Some columns have only a few categories, like city names or status labels.

Think of it like packing a suitcase. You would not fold a winter coat the same way you pack socks. A learned codec does that kind of matching for data. It looks at the type of a column, then chooses or learns a compression strategy that fits that pattern. The goal is simple, smaller files with fast decoding, so databases can read data quickly when people query it.

Why This Is a Good Topic

This is a strong science fair topic because you can measure it in a clear way. You can compare file size, compression time, and decode speed across real public datasets. You also get a real-world link to how databases store and serve data at scale. A student can learn data profiling, benchmarking, and careful evaluation, which are all useful research skills.

Research Questions

• How does a column-specific compression strategy change file size compared with Zstd-level-9 on Parquet tables?
• What is the effect of column type, such as monotonic timestamps or Zipfian IDs, on decode CPU cost?
• Does training a separate codec for low-cardinality categorical columns improve compression ratio without slowing reads?
• To what extent does a learned model generalize from Stack Overflow data to NYC taxi data?
• Which column features best predict whether a custom codec will beat a general-purpose compressor?
• How does the number of distinct values in a column affect the tradeoff between compression ratio and decode speed?

Basic Materials

• Laptop or desktop computer with at least 16 GB RAM.
• Python 3.10 or newer.
• Parquet datasets from public sources such as Stack Overflow and NYC taxi records.
• Command-line tools for running compression benchmarks.
• External storage or enough free disk space for multiple dataset copies.
• Spreadsheet software for logging results.
• Basic plotting library in Python, such as matplotlib or seaborn.

Advanced Materials

• University Linux workstation or server with multiple CPU cores.
• Profiling tools such as perf or py-spy.
• Apache Arrow and Parquet libraries for column-level inspection.
• A development build of a compression library or codec prototype.
• Benchmark harness for repeated read and write tests.
• Dataset preprocessing scripts for feature extraction and sampling.
• Git for version control and experiment tracking.

Software & Tools

• Python: Runs preprocessing, benchmark scripts, and result analysis.
• Apache Arrow: Reads Parquet columns and helps inspect column types and layouts.
• PyArrow: Lets you load, write, and benchmark Parquet data from Python.
• pandas: Summarizes column statistics and helps compare dataset properties.
• matplotlib: Plots compression ratio, decode speed, and CPU cost across tests.

Experiment Steps

1. Define which column patterns you will target, such as monotonic, Zipfian, or low-cardinality columns.
2. Choose public datasets and profile their columns so you can group them by statistical type.
3. Select one baseline compressor and one column-aware strategy so your comparison stays fair.
4. Design benchmarks that measure file size, encode time, decode time, and CPU cost under the same conditions.
5. Plan controls that keep schema, sampling method, and hardware settings consistent across runs.
6. Decide how you will score success, then test whether the custom codec still wins on a second dataset.

Common Pitfalls

• Mixing different column types in one benchmark batch, which hides whether the codec helps only one pattern.
• Comparing against a weak baseline instead of a strong general-purpose compressor, which makes the result less convincing.
• Measuring file size on one machine and decode speed on another, which breaks the fairness of the comparison.
• Ignoring schema differences between Stack Overflow and NYC taxi data, which can make generalization claims look stronger than they are.
• Testing only one sample split, which can make a lucky column sample look like a real win.

What Makes This Competitive

A strong version of this project does more than say one codec is smaller. It explains when and why the codec works, then backs that up with clean benchmarks across multiple column types. You can make the project stronger by adding a predictor that chooses the best compression strategy from column statistics, or by testing whether the method still helps on a new public dataset. Careful error bars, fair baselines, and a clear generalization test can push the work much closer to research style.

Project Variations

• Test whether the same column-aware compression strategy works on JSON-to-Parquet conversion pipelines instead of native Parquet dumps.
• Compare a learned codec against dictionary encoding, run-length encoding, and delta encoding on the same columns.
• Train the model on one public dataset and measure how much performance drops on a different dataset with similar column shapes.

Learn More

• Apache Parquet documentation: Read the file format overview and column encoding details in the official docs.
• Apache Arrow documentation: Learn how columnar data is stored and moved between tools in the Arrow docs.
• PubMed: Search for review articles on data compression in databases and analytical systems.
• arXiv: Search for recent preprints on learned compression, columnar storage, and database encoding.
• MIT OpenCourseWare: Look for database systems lectures that cover storage engines, indexing, and columnar design.
• USGS Science Data tutorials: Study how public data systems organize large datasets for efficient access.