jQuery webuploader uploads large files in parts

Generally when uploading files, the files to be uploaded are uploaded to the server through the client. At this time, the uploaded files are all in the server memory. If large files such as videos are uploaded, the server memory will be very tight, and Generally, we use flash or HTML5 for asynchronous upload. If the file is relatively large, even if the client shows that the file has been uploaded 100%, there will still be a long wait, and the current page's request to the server will also be blocked. block.

Under normal circumstances, it is usually saved directly on the server after the long transfer is completed.

public void ProcessRequest(HttpContext context)
  {
   context.Response.ContentType = "text/plain";
   //保存文件
   context.Request.Files[0].SaveAs(context.Server.MapPath("~/1/" + context.Request.Files[0].FileName));
   context.Response.Write("Hello World");
  }

In recent projects, Baidu’s open source upload component webuploader has been used. Officially, webuploader supports multi-part uploading.

var uploader = WebUploader.create({
   auto: true,
   swf:'/webuploader/Uploader.swf',
   // 文件接收服务端。
   server: '/Uploader.ashx',
   // 内部根据当前运行是创建，可能是input元素，也可能是flash.
   pick: '#filePicker',
   chunked: true,//开启分片上传
   threads: 1,//上传并发数
   //由于Http的无状态特征，在往服务器发送数据过程传递一个进入当前页面是生成的GUID作为标示
   formData: {guid:"<%=Guid.NewGuid().ToString()%>"}
  });

webuploader's multi-part upload is to divide the file into several parts, and then post the data to the file receiving end you define. If the uploaded file is larger than the fragment size, it will be divided into parts, and then it will be posted Add two form elements chunk and chunks to the data. The former indicates the order of the current fragment in the uploaded fragment (starting from 0), and the latter represents the total number of fragments.

After selecting a file, it was divided into 7 shards, so the process of posting data to Uploader.ashx was performed 7 times.

The form elements chunk and chunks in each request and the GUID to indicate that they are fragments of the same file

After receiving the data on the server side, it can be processed according to these parameters.

　1. Press the GUID to create a temporary file

　　2. Append the received fragment data to the file corresponding to the GUID.

　3. Rename the temporary file according to the uploaded file name

　4. If there is no sharding, save it directly

public void ProcessRequest(HttpContext context)
  {
   context.Response.ContentType = "text/plain";
   //如果进行了分片
   if (context.Request.Form.AllKeys.Any(m => m == "chunk"))
   {
    //取得chunk和chunks
    int chunk =Convert.ToInt32(context.Request.Form["chunk"]);
    int chunks = Convert.ToInt32(context.Request.Form["chunks"]);
 
     
    //根据GUID创建用该GUID命名的临时文件
    string path = context.Server.MapPath("~/1/" + context.Request["guid"]);
    FileStream addFile = new FileStream(path, FileMode.Append, FileAccess.Write);
    BinaryWriter AddWriter = new BinaryWriter(addFile);
    //获得上传的分片数据流
    HttpPostedFile file = context.Request.Files[0];
    Stream stream = file.InputStream;
 
    BinaryReader TempReader = new BinaryReader(stream);
    //将上传的分片追加到临时文件末尾
    AddWriter.Write(TempReader.ReadBytes((int)stream.Length));
    //关闭BinaryReader文件阅读器
    TempReader.Close();
    stream.Close();
    AddWriter.Close();
    addFile.Close();
 
    TempReader.Dispose();
    stream.Dispose();
    AddWriter.Dispose();
    addFile.Dispose();
    //如果是最后一个分片，则重命名临时文件为上传的文件名
    if (chunk == (chunks - 1))
    {
     FileInfo fileinfo = new FileInfo(path);
     fileinfo.MoveTo(context.Server.MapPath("~/1/" + context.Request.Files[0].FileName));
    }
   }
   else//没有分片直接保存
   {
    context.Request.Files[0].SaveAs(context.Server.MapPath("~/1/" + context.Request.Files[0].FileName ));
   }
   context.Response.Write("ok");
  }

There are still some unresolved problems, although the needs are temporarily met:

1. If the upload concurrency exceeds 1 At this time, there will be a split upload server that has not finished processing, and the second split will arrive at the same time, so there will be an error that the file is occupied.
2. If locking solves the first problem, then locking will definitely affect the efficiency (at the same time, only one process can access the code that saves the file).
3. There is a problem with the order of files. It is possible that the second fragment arrives first, and then the first one arrives. Then you cannot append the stream to the temporary file at once. You can only create multiple temporary files and wait for all fragments to be processed. After the upload is completed, it is spliced ​​into one file.

It’s just a demo, I hope someone can help solve the existing problems.