Detailed introduction to iterators and iterator slicing in Python

This article brings you a detailed introduction to iterators and iterator slices in Python. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

In the first two articles about Python slicing, we learned the basic usage, advanced usage, misunderstandings of slicing, and how custom objects implement slicing usage (see the end of the article for related links). This article is the third in the slicing series, and its main content is iterator slicing.

Iterator is a unique advanced feature in Python, and slicing is also an advanced feature. What will be the result of combining the two?

1. Iteration and iterators

First of all, there are several basic concepts to clarify: iteration, iterable objects, and iterators.

Iteration is a way of traversing container type objects (such as strings, lists, dictionaries, etc.). For example, when we say iterate a string "abc", it refers to starting from The process of extracting all its characters one by one from left to right. (PS: The word "iteration" in Chinese means repeated cycles and layer-by-layer progression, but in Python this word should be understood as One-way horizontal linear. If you are not familiar with it, I suggest you use it directly. Understood as traversal.)

So, how to write instructions for iterative operations? The most common writing syntax is the for loop.

# for循环实现迭代过程
for char in "abc":
    print(char, end=" ")
# 输出结果：a b c

The for loop can implement the iterative process, but not all objects can be used in the for loop. For example, if the string "abc" is replaced by any integer in the above example, an error will be reported: 'int' object is not iterable .

The word "iterable" in this error message refers to "iterable", that is, the int type is not iterable. The string type is iterable, and similarly, types such as lists, tuples, and dictionaries are all iterable.

How to determine whether an object is iterable? Why are they iterable? How to make an object iterable?

To make an object iterable, you need to implement the iterable protocol, that is, you need to implement the __iter__() magic method. In other words, as long as the object that implements this magic method is an iterable object .

How to determine whether an object implements this method? In addition to the for loop mentioned above, I know of four other methods:

# 方法1：dir()查看__iter__
dir(2)     # 没有，略
dir("abc") # 有，略

# 方法2：isinstance()判断
import collections
isinstance(2, collections.Iterable)     # False
isinstance("abc", collections.Iterable) # True

# 方法3：hasattr()判断
hasattr(2,"__iter__")     # False
hasattr("abc","__iter__") # True

# 方法4：用iter()查看是否报错
iter(2)     # 报错：'int' object is not iterable
iter("abc") # <str_iterator>

### PS：判断是否可迭代，还可以查看是否实现__getitem__，为方便描述，本文从略。</str_iterator>

The most noteworthy of these methods is the iter() method, which is a built-in method of Python and its function isTurn iterable objects into iterators. This sentence can be parsed into two meanings: (1) iterable objects and iterators are two different things; (2) iterable objects can become iterators.

In fact, an iterator must be an iterable object, but an iterable object is not necessarily an iterator. How big is the difference between the two?

As shown in the blue circle in the above figure, the most critical difference between ordinary iterable objects and iterators can be summarized as: one is the same and the other is different. The so-called "one is the same" ", that is, both are iterable (__iter__). The so-called "two differences" mean that after the iterable object is converted into an iterator, it will lose some attributes (__getitem__) and also add some attributes (__next__).

First look at the added attribute __next__. It is the key to the reason why an iterator is an iterator. In fact, we define an object that implements both the __iter__ method and the __next__ method as an iterator. of.

With this additional attribute, iterable objects can implement their own iteration/traversal process without resorting to external for loop syntax. I invented two concepts to describe these two traversal processes (PS: For ease of understanding, it is called traversal here, but it can actually be called iteration): traversal refers to traversal implemented through external grammar, and self-traversal refers to Traversal implemented through its own methods.

With the help of these two concepts, we say that an iterable object is an object that can be "traversed by it", and an iterator is an object that can also be "self-traversed" on this basis.

ob1 = "abc"
ob2 = iter("abc")
ob3 = iter("abc")

# ob1它遍历
for i in ob1:
    print(i, end = " ")   # a b c
for i in ob1:
    print(i, end = " ")   # a b c
# ob1自遍历
ob1.__next__()  # 报错： 'str' object has no attribute '__next__'

# ob2它遍历
for i in ob2:
    print(i, end = " ")   # a b c    
for i in ob2:
    print(i, end = " ")   # 无输出
# ob2自遍历
ob2.__next__()  # 报错：StopIteration

# ob3自遍历
ob3.__next__()  # a
ob3.__next__()  # b
ob3.__next__()  # c
ob3.__next__()  # 报错：StopIteration

As can be seen from the above example, the advantage of iterator is that it supports self-traversal. At the same time, it is characterized by one-way non-loop. Once the traversal is completed, an error will be reported when called again.

In this regard, I think of an analogy: an ordinary iterable object is like a bullet magazine. It traverses by taking out the bullet and putting it back after completing the operation, so it can be traversed repeatedly (that is, calling the for loop multiple times, Returns the same result); and the iterator is like a gun loaded with a magazine and not detachable. To traverse or self-traverse it is to fire bullets. This is a consumable traversal and cannot be reused (that is, the traversal will have an end. ).

I have written so much, let me summarize it a little: Iteration is a way of traversing elements. It is divided according to the implementation method. There are two types: external iteration and internal iteration. Objects that support external iteration (it traverses) It is an iterable object, and an object that also supports internal iteration (self-traversal) is an iterator; according to the consumption method, it can be divided into reusable iteration and one-time iteration. Ordinary iterable objects are reusable, and iterators The device is disposable.

2、迭代器切片

前面提到了“一同两不同”，最后的不同是，普通可迭代对象在转化成迭代器的过程中会丢失一些属性，其中关键的属性是 __getitem__ 。在《Python进阶：自定义对象实现切片功能》中，我曾介绍了这个魔术方法，并用它实现了自定义对象的切片特性。

那么问题来了：为什么迭代器不继承这个属性呢？

首先，迭代器使用的是消耗型的遍历，这意味着它充满不确定性，即其长度与索引键值对是动态衰减的，所以很难 get 到它的 item ，也就不再需要 __getitem__ 属性了。其次，若强行给迭代器加上这个属性，这并不合理，正所谓强扭的瓜不甜......

由此，新的问题来了：既然会丢失这么重要的属性（还包括其它未标识的属性），为什么还要使用迭代器呢？

这个问题的答案在于，迭代器拥有不可替代的强大的有用的功能，使得 Python 要如此设计它。限于篇幅，此处不再展开，后续我会专门填坑此话题。

还没完，死缠烂打的问题来了：能否令迭代器拥有这个属性呢，即令迭代器继续支持切片呢？

hi = "欢迎关注公众号：Python猫"
it = iter(hi)

# 普通切片
hi[-7:] # Python猫

# 反例：迭代器切片
it[-7:] # 报错：'str_iterator' object is not subscriptable

迭代器因为缺少__getitem__ ，因此不能使用普通的切片语法。想要实现切片，无非两种思路：一是自己造轮子，写实现的逻辑；二是找到封装好的轮子。

Python 的 itertools 模块就是我们要找的轮子，用它提供的方法可轻松实现迭代器切片。

import itertools

# 例1：简易迭代器
s = iter("123456789")
for x in itertools.islice(s, 2, 6):
    print(x, end = " ")   # 输出：3 4 5 6
for x in itertools.islice(s, 2, 6):
    print(x, end = " ")   # 输出：9

# 例2：斐波那契数列迭代器
class Fib():
    def __init__(self):
        self.a, self.b = 1, 1

    def __iter__(self):
        while True:
            yield self.a
            self.a, self.b = self.b, self.a + self.b
f = iter(Fib())
for x in itertools.islice(f, 2, 6):
    print(x, end = " ")  # 输出：2 3 5 8
for x in itertools.islice(f, 2, 6):
    print(x, end = " ")  # 输出：34 55 89 144

itertools 模块的 islice() 方法将迭代器与切片完美结合，终于回答了前面的问题。然而，迭代器切片跟普通切片相比，前者有很多局限性。首先，这个方法不是“纯函数”（纯函数需遵守“相同输入得到相同输出”的原则，之前在《来自Kenneth Reitz大神的建议：避免不必要的面向对象编程》提到过）；其次，它只支持正向切片，且不支持负数索引，这都是由迭代器的损耗性所决定的。

那么，我不禁要问：itertools 模块的切片方法用了什么实现逻辑呢？下方是官网提供的源码：

def islice(iterable, *args):
    # islice('ABCDEFG', 2) --> A B
    # islice('ABCDEFG', 2, 4) --> C D
    # islice('ABCDEFG', 2, None) --> C D E F G
    # islice('ABCDEFG', 0, None, 2) --> A C E G
    s = slice(*args)
    # 索引区间是[0,sys.maxsize]，默认步长是1
    start, stop, step = s.start or 0, s.stop or sys.maxsize, s.step or 1
    it = iter(range(start, stop, step))
    try:
        nexti = next(it)
    except StopIteration:
        # Consume *iterable* up to the *start* position.
        for i, element in zip(range(start), iterable):
            pass
        return
    try:
        for i, element in enumerate(iterable):
            if i == nexti:
                yield element
                nexti = next(it)
    except StopIteration:
        # Consume to *stop*.
        for i, element in zip(range(i + 1, stop), iterable):
            pass

islice() 方法的索引方向是受限的，但它也提供了一种可能性：即允许你对一个无穷的（在系统支持范围内）迭代器进行切片的能力。这是迭代器切片最具想象力的用途场景。

除此之外，迭代器切片还有一个很实在的应用场景：读取文件对象中给定行数范围的数据。

在《给Python学习者的文件读写指南（含基础与进阶，建议收藏）》里，我介绍了从文件中读取内容的几种方法：readline() 比较鸡肋，不咋用；read() 适合读取内容较少的情况，或者是需要一次性处理全部内容的情况；而 readlines() 用的较多，比较灵活，每次迭代读取内容，既减少内存压力，又方便逐行对数据处理。

虽然 readlines() 有迭代读取的优势，但它是从头到尾逐行读取，若文件有几千行，而我们只想要读取少数特定行（例如第1000-1009行），那它还是效率太低了。考虑到文件对象天然就是迭代器 ，我们可以使用迭代器切片先行截取，然后再处理，如此效率将大大地提升。

# test.txt 文件内容
'''
猫
Python猫
python is a cat.
this is the end.
'''

from itertools import islice
with open('test.txt','r',encoding='utf-8') as f:
    print(hasattr(f, "__next__"))  # 判断是否迭代器
    content = islice(f, 2, 4)
    for line in content:
        print(line.strip())
### 输出结果：
True
python is a cat.
this is the end.

The above is the detailed content of Detailed introduction to iterators and iterator slicing in Python. For more information, please follow other related articles on the PHP Chinese website!