How to Remove Consecutive Duplicates in Pandas?

Removing Consecutive Duplicates in Pandas

While Pandas' drop_duplicates() method is effective for eliminating all duplicate values, it does not discern consecutive occurrences. To address this limitation, there are efficient methods to selectively drop only consecutive duplicates.

One approach employs the shift function to compare the current value to the previous one:

a.loc[a.shift() != a]

This logic returns a mask where consecutive duplicates are characterized by False values. The loc method then selects only the rows with True values, effectively removing the consecutive duplicates.

Another method utilizes the diff function to detect changes:

a.loc[a.diff() != 0]

However, this approach is less efficient for large datasets due to the overhead associated with the differentiation calculation.

Update

It's worth noting that the default shift period is 1, so shift() and shift(1) produce equivalent results:

a.loc[a.shift(1) != a]

This ensures that the first consecutive value is correctly identified as a duplicate.

The above is the detailed content of How to Remove Consecutive Duplicates in Pandas?. For more information, please follow other related articles on the PHP Chinese website!