How to Remove Duplicate Rows Across Multiple Columns in Pandas?

Removing Duplicates Across Multiple Columns in Python Pandas

The drop_duplicates function in Pandas provides a convenient way to remove duplicate rows based on specified columns. However, what if you want to drop duplicates not across a single column but rather a subset of multiple columns?

To achieve this, we can harness the power of drop_duplicates along with the subset parameter. By specifying the list of columns to check for duplicates in, you can ensure that rows matching on any combination of those columns are eliminated.

Consider the following example:

    A   B   C
0   foo 0   A
1   foo 1   A
2   foo 1   B
3   bar 1   A

Our goal is to drop rows that match on both columns A and C. This would remove rows 0 and 1, as they have the same values in both columns.

Using drop_duplicates, we can accomplish this with the following code:

import pandas as pd

df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.drop_duplicates(subset=['A', 'C'], keep=False)

The subset parameter specifies the columns to use for duplicate detection. The keep parameter, set to False, ensures that all duplicate rows are removed.

The resulting DataFrame will be as follows:

    A   B   C
0   foo 0   A
2   foo 1   B
3   bar 1   A

Rows 0 and 1 have been dropped because they matched on both A and C, effectively uniquifying the DataFrame based on those columns.

The above is the detailed content of How to Remove Duplicate Rows Across Multiple Columns in Pandas?. For more information, please follow other related articles on the PHP Chinese website!