How to Remove Duplicate Dictionaries from a List in Python While Preserving Order?

Removing Duplicate Dictionaries in a List in Python

When working with lists of dictionaries, it may be necessary to remove duplicates based on identical key-value pairs. This article will provide detailed guidance on how to achieve this using Python's built-in functions and a more efficient approach that preserves dictionary order.

Using Sets for Deduplication

One approach involves converting each dictionary into a tuple of its items. Since tuples are hashable, they can be used as keys in a set to identify duplicate dictionaries. The following Python code demonstrates this method:

new_list = [dict(t) for t in {tuple(d.items()) for d in old_list}]

This code iterates over the original list of dictionaries (old_list) and creates a set of tuples representing the items in each dictionary. The set automatically removes duplicates. The resulting set is then converted back to a list of dictionaries using a dictionary comprehension.

Preserving Order with Custom Logic

However, if the order of the dictionaries is important, the above method will not preserve it. To achieve this, the following alternative approach is recommended:

seen = set()
new_list = []
for d in old_list:
    t = tuple(sorted(d.items()))
    if t not in seen:
        seen.add(t)
        new_list.append(d)

This code creates a set of tuples representing the ordered items in each dictionary. The sorted() function ensures that the items are in a consistent order. The set then serves to exclude duplicate tuples, and the resulting list retains the original dictionary order.

Handling Nested Dictionaries

If the dictionaries contain nested dictionaries, the code provided requires slight modification to flatten the nested structures. The following code snippet illustrates this:

def flatten_dict(d):
    new_d = {}
    for k, v in d.items():
        if isinstance(v, dict):
            v = flatten_dict(v)
            for k2, v2 in v.items():
                new_d[f'{k}.{k2}'] = v2
        else:
            new_d[k] = v
    return new_d

new_list = [dict(t) for t in {tuple(flatten_dict(d).items()) for d in old_list}]

Conclusion

In this article, we have explored two ways to remove duplicate dictionaries from a list in Python: using sets for deduplication and preserving the original order with custom logic. The choice of method depends on the specific requirements of the task.

The above is the detailed content of How to Remove Duplicate Dictionaries from a List in Python While Preserving Order?. For more information, please follow other related articles on the PHP Chinese website!