The Polars Python package

Polars dataframe

Polars is a fast dataframe library written in Rust. As well as an eager API similar to pandas, it has a lazy API which optimises your queries and can run them in parallel.

Note that unlike pandas, Polars dataframes have no index; rows are addressed by position.

The Polars website has a quickstart guide and a user guide.

Datasets

You can download these and import them as Polars dataframes

movie tags (228MB)
wine data (106kB)
flight data (11kB)
Data on income inequality (Wikipedia)

Imports

import polars as pl

print(pl.__version__)

1.44.1

Dataframe

df = pl.DataFrame(
    {
        "X": [78, 85, 96, 80, 86],
        "Y": [84, 94, 89, 83, 86],
        "Z": [86, 97, 96, 72, 83],
    }
)
df

shape: (5, 3)
┌─────┬─────┬─────┐
│ X   ┆ Y   ┆ Z   │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 78  ┆ 84  ┆ 86  │
│ 85  ┆ 94  ┆ 97  │
│ 96  ┆ 89  ┆ 96  │
│ 80  ┆ 83  ┆ 72  │
│ 86  ┆ 86  ┆ 83  │
└─────┴─────┴─────┘

Series

s = pl.Series([2, 4, 6, 8, 10])
s

shape: (5,)
Series: '' [i64]
[
	2
	4
	6
	8
	10
]

Lazy and eager

The examples so far have used the eager API: each operation on a DataFrame is executed immediately. Polars also has a lazy API, which works on a LazyFrame. Lazy operations are not executed immediately but recorded in a logical query plan, and the plan is only executed when you ask for the result. This allows Polars to optimise the query as a whole — for example by pushing filters and projections down so that less data is read — and to run independent parts of the query in parallel.

Use lazy() to go from an eager dataframe to a lazy one:

lf = df.lazy()
print(lf)

naive plan: (run LazyFrame.explain(optimized=True) to see the optimized plan)

DF ["X", "Y", "Z"]; PROJECT */3 COLUMNS

Nothing is computed yet: lf just holds the plan. Use collect() to run the query and get the result as an eager DataFrame:

lf = lf.filter(pl.col("X") > 80).select(["Y", "Z"])
lf.collect()

shape: (3, 2)
┌─────┬─────┐
│ Y   ┆ Z   │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 94  ┆ 97  │
│ 89  ┆ 96  │
│ 86  ┆ 83  │
└─────┴─────┘

You can inspect the plan before and after optimisation with explain():

print(lf.explain(optimized=False))
print(lf.explain())

SELECT [col("Y"), col("Z")]
  FILTER col("X") > 80
  FROM
    DF ["X", "Y", "Z"]; PROJECT */3 COLUMNS
simple π 2/2 ["Y", "Z"]
  FILTER col("X") > 80
  FROM
    simple π 3/3 ["Y", "Z", "X"]
      DF ["X", "Y", "Z"]; PROJECT */3 COLUMNS

Lazy reading also applies to files: pl.scan_csv(filename) returns a LazyFrame and only reads the data (and only the columns and rows that the query needs) when you call collect().

Creating

`pl.DataFrame(np.random.rand(20,5))`	5 columns and 20 rows of random floats
`pl.Series(my_list)`	Create a series from an iterable my_list
`df.with_row_index("idx")`	Add a row number column (Polars has no index)

Viewing and Inspecting

`df.head(n)`	First n rows of the DataFrame
`df.tail(n)`	Last n rows of the DataFrame
`df.shape`	Number of rows and columns
`df.schema`	Column names and data types
`df.estimated_size()`	Memory usage of the DataFrame
`df.describe()`	Summary statistics for numerical columns
`s.value_counts()`	View unique values and counts
`df.select(pl.all().n_unique())`	Number of unique values for all columns

Selecting

`df[col]`	Returns column with label col as Series
`df[[col1, col2]]`	Returns columns as a new DataFrame
`s[0]`	Selection by position
`df.filter(pl.col(id) == 'index_one')`	Selection by the value of a column
`df[0,:]`	First row
`df[0,0]`	First element of first column

Cleaning

`df.columns = ['a','b','c']`	Rename columns
`s.is_null()`	Checks for null values, returns boolean Series
`s.is_not_null()`	Opposite of is_null()
`df.drop_nulls()`	Drop all rows that contain null values
`df.drop_nulls(subset=['a','b'])`	Drop rows that are null in the given columns
`df.drop([c for c in df.columns if df[c].null_count() > 0])`	Drop all columns that contain null values
`df.with_columns(pl.all().fill_null(x))`	Replace all null values with x
`s.fill_null(s.mean())`	Replace all null values with the mean
`s.cast(pl.Float64)`	Convert the datatype of the series to float
`s.replace({1:'one'})`	Replace all values equal to 1 with 'one'
`s.replace({2:'two', 3:'three'})`	Replace all 2 with 'two' and 3 with 'three'
`df.rename(lambda name: name + '_new')`	Mass renaming of columns
`df.rename({'old_name': 'new_name'})`	Selective renaming

Filtering, Sorting, Grouping

`df.filter(pl.col(col) > 0.6)`	Rows where the column col is greater than 0.6
`df.filter((pl.col(col) > 0.6) & (pl.col(col) < 0.8))`	Rows where 0.8 > col > 0.6
`df.sort(col1)`	Sort values by col1 in ascending order
`df.sort(col2, descending=True)`	Sort values by col2 in descending order
`df.sort([col1, col2], descending=[False, True])`	Sort values by col1 in ascending order then col2 in descending
`df.group_by(col)`	Returns a group_by object for values from one column
`df.group_by(col1, col2)`	Returns group_by object for values from multiple columns
`df.group_by(col1).agg(pl.col(col2).mean())`	Returns the mean of col2, grouped by the values in col1
`df.pivot(col2, index=col1, values=col3, aggregate_function="mean")`	Pivot: rows from col1, a column per value of col2, mean of col3
`df.group_by(col1).agg(pl.all().mean())`	Find the average across all columns for every unique col1 group
`df.select(pl.all().mean())`	Compute the mean across each column
`df.with_columns(pl.max_horizontal(pl.all()).alias("max"))`	Compute the max across each row

Joining, Combining

`pl.concat([df1, df2])`	Add the rows in df2 to the end of df1 (columns should
	be identical)
`pl.concat([df1, df2], how="horizontal")`	Add the columns in df2 to the end of df1 (rows should
	be identical)
`df1.join(df2, on=col1, how="inner")`	SQL-style join the columns in df1 with the columns in
	df2 where the rows for col1 have identical values. The
	'how' can be 'left', 'full', 'semi', 'anti' or 'cross'

Statistics

`df.describe()`	Summary statistics for numerical columns
`df.mean()`	Returns the mean of all columns
`df.select(pl.corr(col1, col2))`	Returns the correlation between two columns
`df.select(pl.all().count())`	Returns the number of non-null values in each column
`df.max()`	Returns the highest value in each column
`df.min()`	Returns the lowest value in each column
`df.median()`	Returns the median of each column
`df.std()`	Returns the standard deviation of each column

Importing

`pl.read_csv(filename)`	From a CSV file
`pl.read_csv(filename, separator="\t")`	From a delimited text file (like TSV)
`pl.read_excel(filename)`	From an Excel file
`pl.read_database(query, connection)`	Read from a SQL table/database
`pl.read_json(filename)`	Read from a JSON file
`pl.read_ndjson(filename)`	Read from a newline-delimited JSON file
`pl.scan_csv(filename)`	Lazily scan a CSV file (query optimisation)
`pl.DataFrame(dict)`	From a dict, keys for columns names, values
	for data as lists

Exporting

`df.write_csv(filename)`	Write to a CSV file
`df.write_excel(filename)`	Write to an Excel file
`df.write_database(table_name, connection)`	Write to a SQL table
`df.write_json(filename)`	Write to a file in JSON format
`df.write_ndjson(filename)`	Write to a newline-delimited JSON file
`df.write_parquet(filename)`	Write to a Parquet file