python指定生成器迭代的位置开始

Question

一个爬虫程序需要断点恢复的功能, 之前把循环位置作为[a, b, c]保存在json当中

每次进程启动后先从json读取历史位置然后继续遍历

但因为想改成多线程版本, 就试着换成生成器

但这样一来, 每次进程退出重启, 迭代不能记忆位置了

有何良策?

爬去的网页url格式如

-init-page-line

所以我开始是这样的三层循环, (我在学校C/C++为主...我知道这个很不pythonic)

while self.index[0] <= self.limit[0]:
    while self.index[1] <= self.limit[1]:
        while self.index[2] <= self.limit[1]:
           # get page and download someting

里面的index是一个包含历史位置的列表, 每次程序开始会从一个JSON文件读取
然后每次完成一个页面的读取后就会把当前位置更新到JSON文件

因为整体的页面爬取的量十分大, 页面请求次数在千万次级别
目前IO等待占了90%以上的时间, 就像把它改成多线程的
当然也是第一次尝试多线程..

自己的思路是这样的, 首先类维护一个生成器

# 这里的范围是根据网站本身设的
self.generator = ([i, p, l, r]
                    for i in xrange(1, 27)
                    for p in xrange(1, 101)
                    for l in xrange(1, 101)

然后, 比如说JSON里读进来的历史位置, 是一个这样的列表

[5, 5, 5, 5]

再然后.. 思路有点乱了
但应该是每次在发出请求报并等待响应包的时候, 就继续去取得下一个url并发包
尽可能做到并发..

Answer 1

I don’t quite understand what you want to express, but the iterator can specify the starting position in this way:

from itertools import islice
for x in islice(iterms, 3, None)        # 这里跳过了前面3个元素

Answer 2

I still don’t quite understand what you mean, but I’ll give you a rough guess. Please tell me if I’m wrong.

I guess you want to crawl this form of url:

-init-page-line

So you have three lists that might look like this:

self.lst[0] = ['init', 'a', 'b', 'end']
self.lst[1] = ['page', 'paragraph', 'row']
self.lst[2] = ['line', 'face', 'point']

Then you have index 和 limit to record where you are now and the maximum index of each lst:

self.index = [0, 0, 0]  # in the begining
self.limit = [3, 2, 2]

Then you combine all the urls:

while self.index[0] <= self.limit[0]:
    while self.index[1] <= self.limit[1]:
        while self.index[2] <= self.limit[2]: # P.S. 你這裡是不是打錯了
             name1 = self.lst[0][self.index[0]]
             name2 = self.lst[1][self.index[1]]
             name3 = self.lst[2][self.index[2]]
             # get page "-name1-name2-name3" and download someting
             # update self.index

The above is my personal guess. If you want to complete this thing, I suggest you do this:

from itertools import product, dropwhile

def gen_url(self):
    return '-' + '-'.join(product(*self.lst))

for i, url in enumerate(gen_url())
    # get page by url and downloading something
    # if you want to stop, save i+1 to your json as a save point s

# next time you conitnue the task, just load s from json as start point
for i, url in dropwhile(lambda t: t[0]<s enumerate(gen_url()):
    # get page by url and downloading something

---

• itertools.product

• itertools.dropwhile

---

Questions I answered: Python-QA