Python：在多线程中使用进度条（progressbar）碰到的问题

Question

Python线程池代码

# !/usr/bin/env python
# -*- coding:utf-8 -*-
# ref_blog:http://www.open-open.com/home/space-5679-do-blog-id-3247.html

import Queue
import threading
import time

class WorkManager(object):
    def __init__(self, work_num=1000,thread_num=2):
        self.work_queue = Queue.Queue()
        self.threads = []
        self.__init_work_queue(work_num)
        self.__init_thread_pool(thread_num)

    """
        初始化线程
    """
    def __init_thread_pool(self,thread_num):
        for i in range(thread_num):
            self.threads.append(Work(self.work_queue))

    """
        初始化工作队列
    """
    def __init_work_queue(self, jobs_num):
        for i in range(jobs_num):
            self.add_job(do_job, i)

    """
        添加一项工作入队
    """
    def add_job(self, func, *args):
        self.work_queue.put((func, list(args)))#任务入队，Queue内部实现了同步机制
    """
        检查剩余队列任务
    """
    def check_queue(self):
        return self.work_queue.qsize()

    """
        等待所有线程运行完毕
    """   
    def wait_allcomplete(self):
        for item in self.threads:
            if item.isAlive():item.join()

class Work(threading.Thread):
    def __init__(self, work_queue):
        threading.Thread.__init__(self)
        self.work_queue = work_queue
        self.start()

    def run(self):
        #死循环，从而让创建的线程在一定条件下关闭退出
        while True:
            try:
                do, args = self.work_queue.get(block=False)#任务异步出队，Queue内部实现了同步机制
                do(args)
                self.work_queue.task_done()#通知系统任务完成
            except Exception,e:
                print str(e)
                break

#具体要做的任务
def do_job(args):
    print args
    time.sleep(0.1)#模拟处理时间
    print threading.current_thread(), list(args)

if __name__ == '__main__':
    start = time.time()
    work_manager =  WorkManager(10, 2)
    work_manager.wait_allcomplete()
    end = time.time()
    print "cost all time: %s" % (end-start)

这里，我使用的官网的progressbar组件，打算在多线程中进行使用。

主要方法和属性如下：

• currval: current value of the progress, 0 <= currval <= maxval
• maxval: maximum (and final) value of the progress
• finished: True if the bar is have finished (reached 100%), False o/w
• start_time: first time update() method of ProgressBar was called
• seconds_elapsed: seconds elapsed since start_time
• percentage(): percentage of the progress (this is a method)

---

我打算在wait_allcomplete()函数中进行修改

    def wait_for_complete( self ):
        for item in self.threads:
            if item.isAlive():
                item.join()

        pbar = ProgressBar(maxval = len(self.threads))
        pbar.start()#开始显示进度条
        step = 0
        i=0
        while True:
            for td in self.threads:
                if td.isAlive() == False:
                    step += 1
                    pbar.update(step)#进度更新
        pbar.finish()

主要思路就是：当有一个线程退出的时候就更新下进度条，但执行的时候，以上代码没能达到预期。
希望大家帮忙看下问题出在哪里

Answer 1

There is an attribute in Queue: unfinished_tasks, which stores the current number of unfinished tasks.
Every time a task (task_done) is completed, the value of unfinished_tasks will be reduced by one.
For details, you can view the task_done function in Queue.
In addition, someone has already combined the thread pool and the progress bar. For details, please see starcluster.threadpool
Focus on the wait function in the ThreadPool class.

---

You can also learn about this article: Notes on using the task_done method in Queue