pandas vs Polars vs DuckDB: We Raced Them on a Middle-Seat Computer

Every few months the internet holds a race between pandas, Polars and DuckDB, and every few months the race is run on a machine that costs more than a small car. The results are thrilling, and about as relevant to your Tuesday as a Formula 1 lap time is to the school run.

So we ran it on the kind of computer most of us actually get: a cloud box with 2 cores and about 7 GB of memory. The computing equivalent of a middle seat. Same data, same questions, the current release of each tool, and one rule that most benchmarks skip: every engine had to produce the same results, checked in full, or its time didn’t count.

What exactly did we race?

Five million made-up orders from 200,000 made-up customers, saved as a 210 MB CSV. Then four questions you’d recognise from any week of real work: open the file, total revenue by region and month, join the customers to get revenue by segment, and find each customer’s single biggest order.

We used pandas 3.0.5, Polars 2.0.0 and DuckDB 2.0.0 on Python 3.13.16, with default settings, because default settings are what most code runs on. Each task ran once to warm up and then five times (three for the CSV read); the number below is the median. Then we compared the answers in full: all 192 region-by-month totals and all four segment totals matched to the cent, and all three engines picked the same winning order for every one of the 200,000 customers. Fancier tuning would change the numbers. It would also change the article from “what you’ll see” to “what you could see if you read the docs on a Sunday”.

Round one: the group-by, and a polite disappointment

If you came for a bloodbath, the first round will let you down. Revenue by region and month took pandas 0.56 s, Polars 0.44 s and DuckDB 0.19 s. All three, sub-second. You’d need a stopwatch and a grudge to care.

Which raises the obvious question: if everything is fast, why does anyone argue about this at all?

Round two: the join

Here’s the number that made us run the whole thing again, just in case. Joining the 200,000 customers onto five million orders and totalling revenue by segment took pandas 0.68 s. Polars did it in 0.18 s. DuckDB did it in 0.09 s.

About 7 times faster.

Same answer, same machine, same afternoon. The gap isn’t magic: Polars and DuckDB both work column by column and spread the work across every core they can find, while classic pandas mostly does one thing at a time, very reliably, the way it has for over fifteen years.

Before you go build a slide deck about this

Look at the units. Seven times faster than under a second saves you about half a second. You could blink twice. You will not get a promotion out of that blink.

So we added everything up, and the real story wasn’t in the join at all. It was at both ends of the job: getting the file open, and the one question that has to look at every customer’s orders one by one.

pandas didn’t lose in the join.
It lost at the door, and at the window.

Reading the CSV took pandas 3.52 s, Polars 0.84 s and DuckDB 1.10 s. Finding each customer’s biggest order took 3.94 s for pandas, against 1.19 s for Polars and 1.15 s for DuckDB. Those two tasks account for 84% of the gap. In total, pandas spent 8.70 s on work the other two finished in under three seconds.

Is 8.70 s versus 2.53 s a crisis? On one run, no. On a notebook you re-run forty times a day, a dashboard that refreshes every minute, or a file ten times bigger, the difference stops being a blink and starts being your lunch break.

The caveats, honestly: the data is synthetic, it is one modest machine, and pandas has faster options we deliberately left switched off (its pyarrow CSV engine among them). Run it on your own data before you rewrite anything. That last sentence is the whole job of a benchmark, and the bit most of them leave out.

The question we can’t answer for you

The fastest tool in this race is the one some of your team would have to learn. The slowest is the one everyone already knows.

So how slow does the familiar one have to get before learning becomes the cheaper option, and who on your team is quietly keeping count?