本篇文章主要介紹了詳解python並發獲取snmp資訊及性能測試,小編覺得挺不錯的,現在分享給大家,也給大家做個參考。一起跟著小編過來看看吧
python & snmp
#用python取得snmp資訊有多個現成的函式庫可以使用,其中比較常用的是netsnmp
和pysnmp
兩個函式庫。網路上有較多的關於兩個函式庫的例子。
本文重點在於如何並發的獲取snmp的數據,即同時獲取多台機器的snmp資訊。
netsnmp
先說netsnmp。 python的netsnmp,其實是來自於net-snmp套件。
python透過一個c檔案呼叫net-snmp的介面來取得資料。
因此,在並發取得多台機器的時候,不能夠使用協程取得。因為使用協程,在get數據的時候,協程會一直等待net-snmp介面返回數據,而不會像socket使用時那樣在等待數據時把CPU切換給其他協程使用。從這點上來說,使用協程和串行獲取沒有區別。
那麼要如何解決並發取得的問題呢?可以使用線程,多線程獲取(當然也可以使用多進程)。多個執行緒同時呼叫net-snmp的介面取得數據,然後cpu在多個執行緒之間不停切換。當一個執行緒取得一個結果後,可以繼續呼叫介面來取得下一個snmp資料。
這裡我寫了一個範例程式。先把所有的host和oid做成任務放到佇列裡,然後啟動多個線程,去執行獲取任務。程式樣例如下:
import threading import time import netsnmp import Queue start_time = time.time() hosts = ["192.20.150.109", "192.20.150.110", "192.20.150.111", "192.20.150.112", "192.20.150.113", "192.20.150.114", "192.20.150.115", "192.20.150.116", "192.20.150.117", "192.20.150.118", "192.20.150.119", "192.20.150.120", "192.20.150.121", "192.20.80.148", "192.20.80.149", "192.20.96.59", "192.20.82.14", "192.20.82.15", "192.20.82.17", "192.20.82.19", "192.20.82.12", "192.20.80.139", "192.20.80.137", "192.20.80.136", "192.20.80.134", "192.20.80.133", "192.20.80.131", "192.20.80.130", "192.20.81.141", "192.20.81.140", "192.20.82.26", "192.20.82.28", "192.20.82.23", "192.20.82.21", "192.20.80.128", "192.20.80.127", "192.20.80.122", "192.20.81.159", "192.20.80.121", "192.20.80.124", "192.20.81.151", "192.20.80.118", "192.20.80.119", "192.20.80.113", "192.20.80.112", "192.20.80.116", "192.20.80.115", "192.20.78.62", "192.20.81.124", "192.20.81.125", "192.20.81.122", "192.20.81.121", "192.20.82.33", "192.20.82.31", "192.20.82.32", "192.20.82.30", "192.20.81.128", "192.20.82.39", "192.20.82.37", "192.20.82.35", "192.20.81.130", "192.20.80.200", "192.20.81.136", "192.20.81.137", "192.20.81.131", "192.20.81.133", "192.20.81.134", "192.20.82.43", "192.20.82.45", "192.20.82.41", "192.20.79.152", "192.20.79.155", "192.20.79.154", "192.25.76.235", "192.25.76.234", "192.25.76.233", "192.25.76.232", "192.25.76.231", "192.25.76.228", "192.25.20.96", "192.25.20.95", "192.25.20.94", "192.25.20.93", "192.24.163.14", "192.24.163.21", "192.24.163.29", "192.24.163.6", "192.18.136.22", "192.18.136.23", "192.24.193.2", "192.24.193.19", "192.24.193.18", "192.24.193.11", "192.20.157.132", "192.20.157.133", "192.24.212.232", "192.24.212.231", "192.24.212.230"] oids = [".1.3.6.1.4.1.2021.11.9.0",".1.3.6.1.4.1.2021.11.10.0",".1.3.6.1.4.1.2021.11.11.0",".1.3.6.1.4.1.2021.10.1.3.1", ".1.3.6.1.4.1.2021.10.1.3.2",".1.3.6.1.4.1.2021.10.1.3.3",".1.3.6.1.4.1.2021.4.6.0",".1.3.6.1.4.1.2021.4.14.0", ".1.3.6.1.4.1.2021.4.15.0"] myq = Queue.Queue() rq = Queue.Queue() #把host和oid组成任务 for host in hosts: for oid in oids: myq.put((host,oid)) def poll_one_host(): while True: try: #死循环从队列中获取任务,直到队列任务为空 host, oid = myq.get(block=False) session = netsnmp.Session(Version=2, DestHost=host, Community="cluster",Timeout=3000000,Retries=0) var_list = netsnmp.VarList() var_list.append(netsnmp.Varbind(oid)) ret = session.get(var_list) rq.put((host, oid, ret, (time.time() - start_time))) except Queue.Empty: break thread_arr = [] #开启多线程 num_thread = 50 for i in range(num_thread): t = threading.Thread(target=poll_one_host, kwargs={}) t.setDaemon(True) t.start() thread_arr.append(t) #等待任务执行完毕 for i in range(num_thread): thread_arr[i].join() while True: try: info = rq.get(block=False) print info except Queue.Empty: print time.time() - start_time break
netsnmp除了支援get操作之外,還支援walk操作,即遍歷某個oid。
但是walk使用的時候需要謹慎,以免導致高延時等問題,具體可以參見之前的一篇snmpwalk高延時問題分析的博客。
pysnmp
pysnmp是用python實作的一套snmp協定的函式庫。其自身提供了對於非同步的支持。
import time import Queue from pysnmp.hlapi.asyncore import * t = time.time() myq = Queue.Queue() #回调函数。在有数据返回时触发 def cbFun(snmpEngine, sendRequestHandle, errorIndication, errorStatus, errorIndex, varBinds, cbCtx): myq.put((time.time()-t, varBinds)) hosts = ["192.20.150.109", "192.20.150.110", "192.20.150.111", "192.20.150.112", "192.20.150.113", "192.20.150.114", "192.20.150.115", "192.20.150.116", "192.20.150.117", "192.20.150.118", "192.20.150.119", "192.20.150.120", "192.20.150.121", "192.20.80.148", "192.20.80.149", "192.20.96.59", "192.20.82.14", "192.20.82.15", "192.20.82.17", "192.20.82.19", "192.20.82.12", "192.20.80.139", "192.20.80.137", "192.20.80.136", "192.20.80.134", "192.20.80.133", "192.20.80.131", "192.20.80.130", "192.20.81.141", "192.20.81.140", "192.20.82.26", "192.20.82.28", "192.20.82.23", "192.20.82.21", "192.20.80.128", "192.20.80.127", "192.20.80.122", "192.20.81.159", "192.20.80.121", "192.20.80.124", "192.20.81.151", "192.20.80.118", "192.20.80.119", "192.20.80.113", "192.20.80.112", "192.20.80.116", "192.20.80.115", "192.20.78.62", "192.20.81.124", "192.20.81.125", "192.20.81.122", "192.20.81.121", "192.20.82.33", "192.20.82.31", "192.20.82.32", "192.20.82.30", "192.20.81.128", "192.20.82.39", "192.20.82.37", "192.20.82.35", "192.20.81.130", "192.20.80.200", "192.20.81.136", "192.20.81.137", "192.20.81.131", "192.20.81.133", "192.20.81.134", "192.20.82.43", "192.20.82.45", "192.20.82.41", "192.20.79.152", "192.20.79.155", "192.20.79.154", "192.25.76.235", "192.25.76.234", "192.25.76.233", "192.25.76.232", "192.25.76.231", "192.25.76.228", "192.25.20.96", "192.25.20.95", "192.25.20.94", "192.25.20.93", "192.24.163.14", "192.24.163.21", "192.24.163.29", "192.24.163.6", "192.18.136.22", "192.18.136.23", "192.24.193.2", "192.24.193.19", "192.24.193.18", "192.24.193.11", "192.20.157.132", "192.20.157.133", "192.24.212.232", "192.24.212.231", "192.24.212.230"] oids = [".1.3.6.1.4.1.2021.11.9.0",".1.3.6.1.4.1.2021.11.10.0",".1.3.6.1.4.1.2021.11.11.0",".1.3.6.1.4.1.2021.10.1.3.1", ".1.3.6.1.4.1.2021.10.1.3.2",".1.3.6.1.4.1.2021.10.1.3.3",".1.3.6.1.4.1.2021.4.6.0",".1.3.6.1.4.1.2021.4.14.0", ".1.3.6.1.4.1.2021.4.15.0"] snmpEngine = SnmpEngine() #添加任务 for oid in oids: for h in hosts: getCmd(snmpEngine, CommunityData('cluster'), UdpTransportTarget((h, 161), timeout=3, retries=0,), ContextData(), ObjectType(ObjectIdentity(oid)), cbFun=cbFun) time1 = time.time() - t #执行异步获取snmp snmpEngine.transportDispatcher.runDispatcher() #打印结果 while True: try: info = myq.get(block=False) print info except Queue.Empty: print time1 print time.time() - t break
pysnmp本身只支援最基礎的get和getnext指令,因此如果想要使用walk,就需要自己實作。
效能測試
在同一個環境下,對兩者進行了效能測試。兩者對198個host,10個oid進行採集。
測試群組 | 耗時(sec) |
---|---|
#netsnmp(20執行緒) | 6.252 |
netsnmp(50執行緒) | 3.269 |
netsnmp(200執行緒) | 3.265 |
pysnmp | 4.812 |
可以看到netsnmp的擷取速度跟線程數有關。當執行緒數增大到一定程度,擷取時間不再縮短。因為開闢執行緒同樣會消耗時間。而已有的線程已經夠處理。
pysnmp效能較之略差一下。詳細分析pysnmp在新增任務(執行getCmd時)消耗了約1.2s,之後的採集約消耗3.3秒。
在增加了oid數,在進行實驗。 host仍然是198個,oid是42個。
測試群組 | 耗時(sec) |
---|---|
#netsnmp(20執行緒) | 30.935 |
netsnmp(50執行緒) | 12.914 |
netsnmp(200執行緒) | 4.044 |
pysnmp | #11.043 |
可以看到差距被進一步拉大。在執行緒夠多的情況下,netsnmp的效率要明顯強於pysnmp。
因為二者都支援可以並行擷取多個host,從易用性來說,netsnmp更為簡單一些,且netsnmp支援walk功能。本文更加推薦netsnmp。
安裝netsnmp需要安裝net-snmp。如果centos,則使用yum會較為方便。
以上是詳解python並發取得snmp資訊及效能測試方法的詳細內容。更多資訊請關注PHP中文網其他相關文章!