pandas 3, Polars 2, DuckDB 2: We Ran the Release Notes

A major version number is a promise and a threat in the same digit. The promise is that things got better. The threat is in the small print, and the small print is long.

And this year, three of the tools data people lean on every day each went up a whole number: pandas to 3, Polars to 2 and DuckDB to 2. Each arrived with official release notes and upgrade advice, and each says, in its own polite way, that some of your code will now behave differently.

Release notes are claims, though, and this section has a habit with claims. So we built two small laboratories on the same 2-core cloud machine: one with the versions you probably have installed (pandas 2.3.3, Polars 1.44.2, DuckDB 1.5.6) and one with the new ones (pandas 3.0.5, Polars 2.0.0, DuckDB 2.0.0), both on Python 3.13.16. Then we turned the official notes into 55 small experiments and ran every one of them on both sides. A change made this list only if we watched it happen.

Which leaves the question everyone asks before an upgrade: what breaks? We’ll come back to it, because it turns out to be slightly the wrong question.

pandas 3: text finally gets a type of its own

Start with the change you’ll see first. A column of text used to be object, the dtype that means “some Python things, good luck”. In pandas 3 it is str, backed by PyArrow when PyArrow is installed. On the five million orders from our earlier race, the region column went from 269.4 MB as object to 64.4 MB as str. Same eight region names, 24% of the memory. The new type is also stricter: try to put the number 5 into it and you get a TypeError instead of a column that is quietly mostly text.

The second change is Copy-on-Write, now the only mode. Any subset you take behaves as a copy, so the old two-step “chained assignment” stops writing to the original:

df = pd.DataFrame({'foo': [1, 2, 3], 'bar': [4, 6, 8]})
df['foo'][df['bar'] > 5] = 100       # chained assignment
df['foo'].tolist()
# pandas 2.3.3: [1, 100, 100]  (with a FutureWarning)
# pandas 3.0.5: [1, 2, 3]  (with a ChainedAssignmentError warning)

df.loc[df['bar'] > 5, 'foo'] = 100   # one step: works in both
# [1, 100, 100]

The third is easy to miss. Dates parsed from text are now stored in microseconds rather than nanoseconds, so the integer you get from a date is a thousand times smaller than it used to be:

s = pd.to_datetime(pd.Series(['2024-01-01']))
s.astype('int64')[0]
# pandas 2.3.3: 1704067200000000000   (nanoseconds)
# pandas 3.0.5: 1704067200000000      (microseconds)
s.dt.as_unit('ns').astype('int64')[0]   # say the unit: same in both

And nothing fails there. A number that was right yesterday is a thousand times off today. File that one away.

There’s good news too: pd.col('a') + pd.col('b') now builds an expression you can hand to assign, which saves a lot of lambda df:. And time zones are now the standard library’s zoneinfo, so pytz is no longer required.

Polars 2: the streaming engine takes the wheel

Over in Polars, the big move is that every lazy query now runs on the streaming engine by default. It is built to be fast and to handle data bigger than memory, and it does not promise to keep rows in the order they came in. The upgrade guide says so for join, group_by and unpivot. We wanted to see how much that matters, so we joined our 5,000,000 orders to 200,000 customers, lazily, in both versions.

Look closely, though: the same 5,000,000 rows, and the same revenue total, 1,067,372,417.25, on every run. But the rows came back in a few hundred interleaved blocks, arranged differently each time: one run started with order 1,699. Ask for maintain_order='left' and the order comes back, with 0 breaks. Code that takes “the first row after a join” now takes whichever block the engine finished first.

The rest of Polars 2 is mostly tidying, and most of it is loud. Casting text straight to a date now fails (use str.to_date()), read_csv no longer accepts n_threads, a side-by-side concat needs equal heights, and the long-deprecated names are gone, with an error that tells you what to do instead:

AttributeRemovedError:
`melt` was removed in version 2.0; use `DataFrame.unpivot`
instead, with `index` instead of `id_vars` and `on`
instead of `value_vars`

A few are quieter. A CSV without a header now gets columns named column_0, column_1 rather than column_1, column_2, so code that asks for column_1 now gets a different column. An empty list now explodes into no row at all instead of one null row. And Int64 + UInt64 gives Int128, where it used to give a lossy Float64.

DuckDB 2: the change that bit us

Then there’s DuckDB, and the change that actually bit us. While we were building our next benchmark, a one-line query that read a DuckDB setting with fetchone() stopped the whole run. Relations no longer have fetchone(), and the error explains why in one line: Relations no longer have fetchone(): a relation keeps no result between calls, so every call runs it again. Use rel.limit(1).fetchall() instead. We switched to fetchall()[0] and started again.

The Arrow calls moved too. In 1.x, the result of con.execute() could hand you an Arrow table directly. In 2.0 those methods are gone from execute() results, and the official migration guide sends you to the relational API:

con = duckdb.connect()
con.execute(q).fetch_arrow_table()   # 1.5.6: works, with a DeprecationWarning
                                     # 2.0.0: AttributeError
con.sql(q).to_arrow_table()          # the 2.0 way

Call to_arrow_table() on an execute() result anyway and it still works, with a warning that it makes an extra copy of the rows.

The SQL changed in places as well. Integer division by zero, 1 // 0, is now an error where it used to be NULL. The old x -> x + 1 lambda is switched off in favour of lambda x: x + 1. And a database file written by 2.0 can’t be opened by 1.5.6, which told us: IO Error: Trying to read a database file with version number 69, but we can only read versions between 64 and 68. Files go forwards, not back, unless you ask for storage_compatibility_version.

Then there were the ones that changed an answer without a word. SELECT 'abc' ~ 'b' was false and is now true, because ~ now matches part of a string rather than the whole of it. And SELECT 2 * 3! was 720. It is now 12. Both are arguably more correct. Neither will tell you.

Sorting the pile

By now we had 25 changes we’d seen happen with our own eyes. So we sorted them by the only question that matters on the morning after an upgrade: how would you find out?

Read down the last column. Some raise an error. A couple print a warning and carry on. And the rest do neither.

Thirteen of the twenty-five changes make no noise at all.

Which turns the usual worry upside down. The ten errors are the friendly ones: they stop the job on the line that needs fixing, and several of them name the fix. The silent thirteen are where an upgrade actually costs you: a date that is a thousand times smaller, a regular expression that suddenly matches, a join whose first row is now a lottery, a column_1 that is now a different column. “What breaks?” was the wrong question. The useful one is “what changes and keeps running?”

And there is one more thing hiding in our results. The old versions saw some of this coming: six of the twenty-five changes printed a warning in pandas 2.3.3, Polars 1.44.2 or DuckDB 1.5.6 before the new version made it real. pandas’ own advice is to upgrade to 2.3 first and clear every warning before moving to 3.0. Warnings are only useful if somebody reads them.

A word on the limits of all this. These are small, direct experiments, one per change: they show that a behaviour changed, not how often your code relies on it. The count depends on how you group things, too: we counted melt(), with_row_count() and profile() as one change, and // and % by zero as another. Split them and the errors grow to thirteen of twenty-eight; the silent thirteen stay thirteen. We tested pandas 3.0.5, and the changes above arrived with 3.0.0. And the lists in the official guides are longer than ours, because we kept only what we could watch happen.

The morning after

So picture tonight. Somewhere, a scheduled job will run on a freshly upgraded machine. It will finish on time, exit with code zero and write its numbers to the same table as yesterday.

If yours did that, is it because nothing changed, or because nothing told you?