http - python requests post 大文件和获取进度条

Question

准备用HTTP 在局域网内 上传一些文件，文件一般都在1G左右

r = requests.post('****',
                 data={'path':'2016/07/08/5ASD5SDFASDFASDF/cad.zip'},
                 files={'file': open(filename, 'rb')}
                 )
                     

这样上传小文件可以， 但是上传大文件时候会py会报 memeryError的内存错误。
如何解决此问题呢？

Answer 1

from requests_toolbelt import  *

m = MultipartEncoder(fields={'file': ('filename',open(f, 'rb'))},

                     boundary='---------------------------7de1ae242c06ca'
                    )

import time

def my_callback(monitor):
    # Your callback function
    print monitor.bytes_read

m = MultipartEncoderMonitor(m, my_callback)

req_headers = {'Content-Type': m.content_type,
               'path':'2016/07/09/5ASD5SDFASDFASDF/{}.zip'.format(time.time()),}

r = requests.post(url, data=m, headers=req_headers)

Using an extension library, requests_toolbelt can perfectly solve 2 problems

Answer 2

The building owner uses the streaming upload of requests.
Here are some instructions

Streaming Uploads

Requests supports streaming uploads, which allow you to send large streams or files without reading them into memory. To stream and upload, simply provide a file-like object for your body:

with open('massive-body', 'rb') as f:
    requests.post('http://some.url/streamed', data=f)

http://www.python-requests.org/en/master/user/advanced/#streaming-uploads

Answer 3

It is recommended that you use the poster module
https://atlee.ca/software/poster/