How Can I Efficiently Chunk an Iterator in Python?

Iterating Iterators in Chunks

Consider a scenario where you possess an iterator l = [1, 2, 3, 4, 5, 6, 7] and wish to partition it into chunks of size 3, resulting in an iterator containing [[1, 2, 3], [4, 5, 6], [7]].

itertools Solution

The Python standard library offers a convenient solution within the itertools module:

import itertools

def grouper(iterable, n, *, incomplete='fill', fillvalue=None):
    """Collect data into non-overlapping fixed-length chunks or blocks."""

    args = [iter(iterable)] * n
    if incomplete == 'fill':
        return zip_longest(*args, fillvalue=fillvalue)
    elif incomplete == 'strict':
        return zip(*args, strict=True)
    elif incomplete == 'ignore':
        return zip(*args)
    else:
        raise ValueError('Expected fill, strict, or ignore')

chunks = list(grouper(l, 3))  # Output: [[1, 2, 3], [4, 5, 6], [7]]

Standard Library Enhanced Version

A specialized function, batched, introduced in recent versions of the itertools recipes, precisely addresses this chunking requirement:

from itertools import batched

chunks = list(batched(l, 3))  # Output: [[1, 2, 3], [4, 5, 6], [7]]

Non-General Solution

For sequences specifically, a simpler approach that handles last chunks effectively is:

chunks = [my_list[i:i + chunk_size] for i in range(0, len(my_list), chunk_size)]

The above is the detailed content of How Can I Efficiently Chunk an Iterator in Python?. For more information, please follow other related articles on the PHP Chinese website!