Backends

Liken runs on seven DataFrame backends: pandas, polars, modin, dask, ray, pyspark and pyarrow. You do not need choose a backend, Liken detects it from the DataFrame you pass to lk.dedupe: a pandas DataFrame gets the pandas backend, a pyarrow.Table gets the pyarrow backend etc. Passing an unsupported dataframe type raises a ValueError: Unsupported dataframe type.

The deduplication API is the same on every backend. What differs is where the work runs and what you get back. This page lays out those differences; the First Steps tabs show the instantiation code for each.

Installing

pandas, polars and pyarrow ship with Liken itself. The other backends are optional extras:

pip install 'liken[dask]'     # deduplicate dask dataframes
pip install 'liken[modin]'    # deduplicate modin dataframes
pip install 'liken[ray]'      # deduplicate ray datasets
pip install 'liken[pyspark]'  # deduplicate pyspark dataframes
pip install 'liken[all]'      # deduplicate with any of the above

uv pip install 'liken[dask]'    # deduplicate dask dataframes
uv pip install 'liken[modin]'   # deduplicate modin dataframes
uv pip install 'liken[ray]'     # deduplicate ray datasets
uv pip install 'liken[pyspark]' # deduplicate pyspark dataframes
uv pip install 'liken[all]'     # deduplicate with any of the above

Instantiating

Each backend is instantiated by passing its DataFrame to lk.dedupe — see the Instantiating tabs for all seven.

PySpark requires a SparkSession. Pass it explicitly, or Liken raises ValueError: spark_session arg must be provided for a spark dataframe:

import liken as lk

df = lk.dedupe(spark_df, spark_session=spark).apply(lk.fuzzy()).drop_duplicates("address")

For every other backend the spark_session argument is ignored.

Local vs Distributed

The first split is execution scope:

• pandas, polars, modin run over the whole DataFrame in one process. modin is a drop-in for pandas — its DataFrames are pandas-like, and Liken treats them locally.
• pyarrow runs over the whole table in one process. Arrow tables are columnar in memory, the same format Liken uses internally.
• dask, ray, pyspark run over partitions — over batches, on ray. Deduplication executes per partition or batch, on the worker holding it.

The partitioned execution has a consequence you must plan around: records are only matched within a partition. Two identical rows in different partitions will not be deduplicated, and each partition's duplicates get ids from that partition's own numbering. Partition your data so that likely duplicates land together, or use repartition on the columns your rules match against — see Use Partitioned Data.

What You Get Back

collect() returns the same DataFrame type you passed in, with the work already staged by drop_duplicates or canonicalize:

collect() itself computes nothing — on the lazy backends the work executes when you trigger it, for example with .compute() (dask) or a Spark action.

Three methods pull data to the driver, so they trigger a collect on the distributed backends:

• canonicalize(id=None) — computing the auto-increment ids requires the whole frame's row positions.
• canonicals() — counts groups by materialising the canonical ids. On PySpark this needs Spark 4 or above, as documented on the method.
• synthesize() — builds the synthetic records from collected groups.

Passing an existing id column (canonicalize(id="uid")) avoids the auto-increment collect on dask and ray.

Restrictions

One API difference to know before you write code against a distributed backend:

• explore runs on the pandas, polars, modin and pyarrow backends only. On dask, ray or pyspark it raises ValueError. Profile a sample locally first, then apply the chosen rules to the full distributed DataFrame. On pandas and modin the result is indexed by metric; on polars and pyarrow there is no index and metric is a regular column.
• Custom dedupers receive a plain Python list built per partition on the worker (see Data Size and Distributed Backends). Keep the function importable so it can be shipped to workers.

Which One Do I Pick?

• pandas — in-memory frames. The reference backend: everything works, and results are the easiest to inspect.
• polars — in-memory frames on a modern, multithreaded execution engine.
• pyarrow — in-memory Arrow tables, when your data is already Arrow (for example loaded from Parquet with pyarrow.parquet).
• modin — pandas code that has outgrown one core. Scales pandas across cores while keeping the pandas API.
• dask — partitioned, pandas-like processing on one machine or a small cluster, when data does not fit in memory.
• ray — dataset-oriented pipelines with ray already in your stack.
• pyspark — data already lives in Spark. Deduplicate where the data is, rather than moving it.

All seven accept the same dedupers, dict collections and pipelines. The choice is about where your data lives and how far it must scale, not about which rules you can write.