Where the Memory Runs Out: pandas, Polars and DuckDB up to 160 Million Rows

You may know the conversation. A notebook dies halfway through a group-by, somebody says the laptop is too small, and somebody else starts pricing a bigger machine.

Before anybody spends money, it would help to know where “too small” actually starts. So we went looking for the edge, on the same 2-core machine as our five-million-row race, by giving it more and more rows until something gave way.

The ladder and the fence

The plan was simple: one question all the way up, revenue by region and month, the group-by from the race. The data is the same kind of synthetic orders, saved as Parquet (we took our own doorway advice), in seven sizes from 1 million rows to 160 million. The smallest file is 13.6 MB. The biggest is 2.2 GB.

Then the safety rail. The machine has about 8 GB of memory, and a run that eats all of it takes everything else down with it. So every run got its own process inside a hard fence of 4 GiB (4.3 GB), half the machine. Cross the fence and the operating system stops that one run and nothing else. A stopped run isn’t an error in the experiment. It’s the answer.

Here are the contestants, written the plain way:

cols = ['region', 'order_date', 'qty', 'unit_price']
month = pl.col('order_date').dt.truncate('1mo')
rev = (pl.col('qty') * pl.col('unit_price')).sum()

# pandas: load four columns, then group
o = pd.read_parquet(path, columns=cols)
d = o.assign(rev=o.qty * o.unit_price, m=o.order_date.dt.to_period('M'))
d.groupby(['region', 'm'], observed=True)['rev'].sum()

# Polars, eager: load four columns, then group
pl.read_parquet(path, columns=cols).group_by(['region', month]).agg(rev)

# Polars, lazy: describe the whole query, then collect()
pl.scan_parquet(path).group_by(['region', month]).agg(rev).collect()

# DuckDB: SQL straight on the file
con.sql(f"""SELECT region, date_trunc('month', order_date) AS m,
                  SUM(qty * unit_price)
           FROM read_parquet('{path}') GROUP BY 1, 2""").fetchall()

You’ll notice Polars is in there twice, once eager and once lazy. There’s a reason for that, and it turns out to be most of the story.

A word on the measuring: peak memory here means the most memory the process held at any moment, Python and the library included. Every run’s 192 region-by-month totals were compared across all the engines that finished, at every size, and rounded to two decimal places they were identical.

The easy floors

At the bottom of the ladder, nobody breaks a sweat. One million rows took every engine under a quarter of a second and about 290 MB at most. At five million, pandas and eager Polars each held about 0.8 GB. Nothing to worry about on this machine.

By twenty million, though, the shapes had split apart:

Look at the peaks. pandas reached 2.3 GB, about 8.5 times the size of the 273 MB file it was reading. Eager Polars peaked higher still, at 3.2 GB: speed and thrift aren’t the same thing. Meanwhile lazy Polars sat at 471 MB and DuckDB at 125 MB.

The wall

Then forty million rows, a 545 MB file. pandas climbed to the fence and was stopped. Eager Polars climbed to the fence and was stopped. On an 8 GB machine with half of it to play with, a file of about half a gigabyte was too much for both.

The other two didn’t notice. Lazy Polars and DuckDB went on through 80 million rows and then 160 million, a 2.2 GB file, and their memory barely moved. Lazy Polars never went above 535 MB. DuckDB never went above 143 MB, which is less than pandas needed for one million rows. DuckDB had set its own memory limit to 3.1 GiB, under the fence, and never came close to that either.

And time scaled the way you’d hope: roughly in step with the rows. At 160 million rows lazy Polars took 6.82 s and DuckDB 9.69 s. Lazy Polars was the fastest at every size; DuckDB was the leanest at every size.

Why Polars twice

Here’s the reason. Eager Polars and lazy Polars are the same library on the same machine reading the same files. One was stopped at 40 million rows. The other finished 160 million in about half a gigabyte. The difference is that the lazy query describes the whole job before anything runs, so a streaming engine can read the file in pieces instead of all at once, and with only 192 groups to remember it never needs much more.

But “lazy” isn’t quite the whole answer either. Since Polars 2.0, a lazy query runs on the streaming engine by default; before, it ran in memory unless you asked. So we ran the identical lazy code on Polars 1.44.2, in the same fence. It used 846 MB at 5 million rows and 3.2 GB at 20 million, just like eager Polars, and at 40 million it was stopped too.

One version number moved the wall from 40 million rows to past 160 million.

So the edge of a modest machine isn’t where we expected. It isn’t set by the size of the file, or even by which library you picked. It’s set by whether the engine has to hold the data before it can answer. Hold the data, and a 2-core box with a 4 GiB fence runs out somewhere between 20 and 40 million rows. Stream it, and we stopped at 160 million without finding the edge at all.

None of this is the last word, and the limits are real. This is one aggregation with only 192 groups, which is the friendliest possible job for a streaming engine: a big join, a sort or a group-by with millions of groups will need more memory, and the Polars 2.0 announcement says spilling to disk does not yet cover joins and group-bys. Peak memory includes Python and the library itself; importing alone took about 105 MB for pandas, 46 MB for Polars and 68 MB for DuckDB. Memory peaks repeated closely run to run, except for both Polars engines: eager Polars ranged from 1.6 GB to 1.8 GB at 10 million rows, and lazy Polars from 382 MB to 471 MB at 20 million and 419 MB to 527 MB at 80 million. Times are medians of three on a shared cloud machine. We used pandas 3.0.5, Polars 2.0.0 and DuckDB 2.0.0 with PyArrow 26.0.0 on Python 3.13.16.

The edge we didn’t find

We stopped at 160 million rows because the point was made, not because anything broke. Somewhere above that, or with a harder job than a 192-group total, the streaming engines will meet a wall of their own. We never reached it.

So where does that wall stand for the job you actually run?