Understanding Parquet Format for beginners

A walk through of the most important file format to ever exist

There’s been a lot of talk about open table formats like Iceberg and Delta over the last few years. While these formats are awesome, many of the underlying efficiency and performance gains can be attributed to Parquet, with Iceberg/Delta serving as a nice management layer on top.

Think of Iceberg/Delta as the middle manager who gets all the credit, while Parquet is the senior engineer doing most of the work!

In this article, we are going to talk about:

• Why Parquet matters?

• The two compression techniques

  • run-length encoding

  • dictionary encoding

• Comparing Parquet to CSV for data compression

• The interplay between Parquet and compression algorithms like zstd

We talk all about data compression in the FREE data engineering boot camp you can join here

Why Parquet Matters

Parquet is a columnar storage format.

Instead of storing rows like this:

| user_id | country | status | amount |
|---------|---------|--------|--------|
| 1       | US      | active | 100    |
| 2       | US      | active | 200    |
| 3       | CA      | churn  | 150    |

Row-based formats (like CSV or JSON) store this row by row.

Parquet stores it column by column:

user_id: [1, 2, 3]
country: [US, US, CA]
status:  [active, active, churn]
amount:  [100, 200, 150]

This unlocks:

• Column Pruning

  • If your query only needs amount , Parquet doesn’t read the other columns.

• Predicate Pushdown

  • If you’re filtering on country = 'US', Parquet can skip entire row groups using metadata statistics.

• Massive Compression

  • Columns contain similar data types and repeated values — perfect for compression.

You’ll notice a theme with Parquet: maximizing data skipping and compression!

The more data we skip, the lower our cloud bill!

Run-Length Encoding (RLE)

RLE compresses sequences of repeated values.

Example column:

country: [US, US, US, US, CA, CA, CA, US]

Instead of storing:

US, US, US, US, CA, CA, CA, US

RLE stores:

(US, 4), (CA, 3), (US, 1)

The more sorted your data is, the better the compression!

You could imagine if we sorted the data above, we would get:

country: [CA, CA, CA, US, US, US, US, US]

And RLE would store:

(CA, 3), (US, 5)

Just by sorting, we reduced our storage footprint 33%!

This is extremely effective for:

• Sorted columns

• Partition columns

• Low-cardinality values

• Boolean columns

• Status flags

Example in PySpark

Let’s create data where RLE will shine:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()

data = [
    (1, “US”, “active”),
    (2, “US”, “active”),
    (3, “US”, “active”),
    (4, “CA”, “churn”),
    (5, “CA”, “churn”),
    (6, “CA”, “churn”),
]

df = spark.createDataFrame(data, [”user_id”, “country”, “status”])

df = df.orderBy(”country”) 

df.write \
    .option(”compression”, “zstd”) \
    .mode(”overwrite”) \
    .parquet(”/tmp/rle_example”)

Because country and status are clustered together, Parquet’s internal RLE encoding compresses them efficiently.

Columnar data would look like this:

user_id: [1,2,3,4,5,6]
country: [US, US, US, CA, CA, CA]
status: [active, active, active, churn, churn, churn]

Run Length encoding Compressed Parquet would look like:

user_id: [1,2,3,4,5,6]
country: [(US, 3), (CA, 3)]
status: [(active, 3), (churn, 3)]

Run-length encoding is one of the most important compression techniques! We cover all of this in a ton of detail in our free data engineering boot camp

Dictionary Encoding

Dictionary encoding works differently.

If a column has limited unique values:

country: [US, US, CA, CA, US]

Parquet builds a dictionary:

Dictionary:
0 -> US
1 -> CA

Then stores:

[0, 0, 1, 1, 0]

Instead of repeating strings, it stores small integers.

This is extremely powerful for:

• Categorical columns

• Enums

• Status fields

• Country codes

• Device types

• Event types

This is not very useful for:

• UUIDs

• User Ids

• Timestamps

Example: Forcing Dictionary Encoding

You can control Parquet behavior in Spark:

df.write \
    .option(”parquet.enable.dictionary”, “true”) \
    .option(”parquet.dictionary.page.size”, “1048576”) \
    .parquet(”/tmp/dict_example”)

Dictionary encoding is usually enabled by default, but it may fall back if:

• Cardinality is too high

• The dictionary grows too large (set by parquet.dictionary.page.size defaulting to 1MB per page)

Remember, dictionary encoding works best for low-cardinality columns!

Comparing CSV to Parquet

Imagine we had this CSV dataset

| country | status |
|---------|---------|
| US      | active | 
| US      | active | 
| US      | active | 
| US      | active | 
| US      | active | 
| US      | active | 
| US      | active | 
| US      | active | 
| CA      | active  |
| CA      | active  |
| CA      | active  |
| CA      | active  |
| CA      | churn  |
| CA      | churn  |
| CA      | churn  |
| CA      | churn  |
| CA      | churn  |
| CA      | churn  |
| CA      | churn  |
| CA      | churn  |

Let’s approximate bytes:

• String sizes:

  • US / CA = 2 bytes

  • active = 6 bytes

  • churn = 5 bytes

  Country bytes:

  • 8×2 (US) = 16

  • 12×2 (CA) = 24
    Total = 40

  Status bytes:

  • 12×6 (active) = 72

  • 8×5 (churn) = 40
    Total = 112

Total (values only):

CSV ≈ 40 + 112 = 152 bytes

Parquet Size (dictionary + RLE, approx)

Dictionary

We store each unique string once:

• US (2) → is stored as 0

• CA (2) → is stored as 1

• active (6) → is stored as 0

• churn (5) → is stored as 1

Dictionary bytes = 15

RLE over dictionary IDs

After dictionary encoding:

country: [(0, 8), (1, 12)]
status: [(0, 12), (1, 8)]

If we approximate each run pair (value_id, run_length) as ~2 bytes (tiny ints), then:

Runs ≈ 4 × 2 = 8 bytes

So:

Parquet (dict + RLE) ≈ 15 + 8 = 23 bytes!

CSV takes up 152 bytes. Parquet takes up 23 bytes

152 / 23 ≈ 6.6× smaller!

And that’s before zstd/snappy compression, and ignoring Parquet metadata overhead (which matters at tiny sizes but becomes negligible at real scale).

Now let’s add zstd to make it even smaller!

Here’s where it gets interesting.

Parquet uses two layers:

1. Encoding layer (RLE, dictionary)

2. Compression layer (zstd, snappy, gzip)

Think of encoding as restructuring data for compression.

Think of zstd as the final squeeze.

Before we add it to the CSV example above, let’s do a quick example of how zstd works:

Imagine this byte string:

ACTIVE_ACTIVE_ACTIVE_ACTIVE

Raw size:

“ACTIVE_” = 7 bytes
4 repetitions = 28 bytes

zstd scans and sees repetition.

It might encode this as:

Literal: “ACTIVE_”
Backreference: (distance=7, length=21)

Meaning:

• Write "ACTIVE_"

• Then repeat the previous 7-byte chunk 3 more times

Instead of storing 28 bytes, it stores:

• 7 literal bytes

• A small pointer describing repetition

That’s how dictionary + RLE inside Parquet creates ideal input for zstd — lots of repetition.

Putting zstd + run length encoding + dictionary encoding

Take our Parquet toy RLE output:

country: (0 × 8), (1 × 12)
status: (0 x 12), (1 x 8)

Internally, this might look like:

0 8 1 12 0 12 1 8

Now zstd sees the repeated 0s, 1s, 12s, and 8s:

• Very small integers

• Very few distinct values

• Repetition patterns

It compresses this to something like:

[compressed block header + compact bitstream]

Shrinking 8 bytes → 5-6 bytes.

Not dramatic here — because dictionary encoding and RLE already removed most redundancy.

The dictionary-encoded, run-length-encoded, zstd-compressed version of this dataset is ~20-21 bytes! This is 8 times smaller than CSV!

Which technique is actually doing the work here?

In this example, we can see that run-length encoding is doing 70-80% of the compression. Dictionary encoding is doing another 10%, and zstd compression is doing another 10%.

The problem with this is that run-length encoding requires sorting to be the most effective. Often, the sorting required to compress the data optimally costs MORE than the storage costs of storing the data non-ideally. Remember, zstd can compress repeated patterns even if the data is not sorted.

This illustrates the painful storage and compute tradeoff that doesn’t have a correct solution. Optimizing the storage almost always costs compute!

How do you compress your data most effectively? Can we vow to never use CSV ever again, at least? Please share this with your friends if you found it useful!

You can use code PARQUET for 30% off the DataExpert.io academy, where we show how these techniques actually work in production!