python - pyspider scheduler 停止调度,重启时间长.
大家讲道理
大家讲道理 2017-04-18 10:04:38
0
2
610
大家讲道理
大家讲道理

光阴似箭催人老,日月如移越少年。

reply all(2)
巴扎黑

When scheduler stops scheduling, does it stop all projects or the one you are trying to restart?

Trace the contents of the scheduler log about project %s updated, status:%s, paused:%s, %d tasks to see if schduler knows that the project status has changed.

unknown project 如果 project 确实存在,是不应该出现的
not processing pack This is normal. After the scheduler restarts, the previously distributed tasks cannot be tracked.
When starting, the scheduler needs to restore the status of all active tasks from the database. If there are many tasks, it will indeed be time-consuming.

巴扎黑

This problem has been found. In the source code of pyspider, the status_count query of statusdb under mongodb under database is very slow when the amount of data is extremely large, which will cause the scheduler to start up very long

Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template