Let's talk in depth about the File module in Node

When talking about Stream/Buffer, we have already started using require("fs") to introduce files The module does some operations

The file module is an encapsulation of underlying file operations, such as file reading and writing/opening and closing/delete adding, etc.

The biggest feature of the file module is that all methods are provided There are two versions of Synchronous and Asynchronous . Methods with sync suffix are all synchronous methods, and those without are all asynchronous methods

File common sense

File Permissions

Because you need to operate the file, you need to set the corresponding permissions. [Related tutorial recommendations: nodejs video tutorial, Programming teaching]

is mainly divided into three roles, file owner, The group to which the file belongs and other users

File permissions are divided into read, write, and execute. The numbers are expressed as 4/2/1 respectively. When there is no permission, it is expressed as 0

If execution is canceled Permissions mean that no files in the folder can be accessed, and you cannot cd to the folder

Use Linux commandsllYou can check the permissions of the files/folders in the directory

The first d represents the folder, - represents the file, followed by the file permissions // TODO: What does @ represent

File identification

In Node, the identification bit represents the operation mode of the file, readable/writable/readable and writable, etc., which can be arranged and combined

File descriptor

As mentioned in the previous content, the operating system will assign a numerical identifier called a file descriptor to each open file, and use these values ​​​​to track specific document.

File descriptors generally start from 3, 0/1/2 represent standard input/standard output/error output respectively

Commonly used API

Some practices

Filtering the appropriate files in your project

const fs = require("fs");
const path = require("path");
const { promisify } = require("util");
const reg = new RegExp("(.ts[x]*|.js[x]*|.json)$");
const targetPath = path.resolve(__dirname, "../mini-proxy-mobx");

const readDir = (targetPath, callback) => {
    fs.readdir(targetPath, (err, files) => {
        if (err) callback(err);
        files.forEach(async (file) => {
            const filePath = path.resolve(__dirname, `${targetPath}/${file}`);
            const stats = await promisify(fs.stat)(filePath);
            if (stats.isDirectory()) {
                await readDir(filePath);
            } else {
                checkFile(filePath);
            }
        });
    });
};
const checkFile = (file) => {
    if (reg.test(file)) {
        console.log(file);
    }
};

readDir(targetPath, (err) => {
    throw err;
});

File copy

Problem: Need to copy the contents of file 1 to file 2

File API

You can use fs.readFile to read the file content, and then use fs.writeFile to write a new file

const fs = require("fs");
const path = require("path");

const sourceFile = path.resolve(__dirname, "../doc/Mobx原理及丐版实现.md");
const targetFile = path.resolve(__dirname, "target.txt");

fs.readFile(sourceFile, (err, data) => {
    if (err) throw err;
    const dataStr = data.toString();
    fs.writeFile(targetFile, dataStr, (err) => {
        if (err) throw err;
        console.log("copy success~");
        process.exit(1);
    });
});

? Is there any problem with this? As we said in Stream, it requires a Click a little bit, otherwise the memory will be overwhelmed when the file is large.

Buffer Use

Use the fs.open method to open the file, obtain the file descriptor, and then call the fs.read/fs.write method to read and write to a specific location. Amount of data

const copyFile = (source, target, size, callback) => {
    const sourceFile = path.resolve(__dirname, source);
    const targetFile = path.resolve(__dirname, target);

    const buf = Buffer.alloc(size);
    let hasRead = 0; // 下次读取文件的位置
    let hasWrite = 0; // 下次写入文件的位置
    fs.open(sourceFile, "r", (err, sourceFd) => {
        if (err) callback(err);
        fs.open(targetFile, "w", (err, targetFd) => {
            if (err) throw callback(err);
            function next() {
                fs.read(sourceFd, buf, 0, size, hasRead, (err, bytesRead) => {
                    if (err) callback(err);
                    hasRead += bytesRead;
                    if (bytesRead) {
                        fs.write(targetFd, buf, 0, size, hasWrite, (err, bytesWrite) => {
                            if (err) callback(err);
                            hasWrite += bytesWrite;
                            next();
                        });
                        return;
                    }
                    fs.close(sourceFd, () => { console.log("关闭源文件"); });
                    fs.close(targetFd, () => { console.log("关闭目标文件"); });
                });
            }
            next();
        });
    });
};

Stream uses

const fs = require("fs");
const path = require("path");
const readStream = fs.createReadStream(
    path.resolve(__dirname, "../doc/Mobx原理及丐版实现.md")
);
const writeStream = fs.createWriteStream(path.resolve("target.txt"));
readStream.pipe(writeStream);

File upload

Small file upload

// 上传后资源的URL地址
const RESOURCE_URL = `http://localhost:${PORT}`;
// 存储上传文件的目录
const UPLOAD_DIR = path.join(__dirname, "../public");

const storage = multer.diskStorage({
    destination: async function (req, file, cb) {
        // 设置文件的存储目录
        cb(null, UPLOAD_DIR);
    },
    filename: function (req, file, cb) {
        // 设置文件名
        cb(null, `${file.originalname}`);
    },
});

const multerUpload = multer({ storage });

router.post(
    "/uploadSingle",
    async (ctx, next) => {
        try {
            await next();
            ctx.body = {
                code: 1,
                msg: "文件上传成功",
                url: `${RESOURCE_URL}/${ctx.file.originalname}`,
            };
        } catch (error) {
            console.log(error);
            ctx.body = {
                code: 0,
                msg: "文件上传失败",
            };
        }
    },
    multerUpload.single("file")
);

Large file upload

Main steps

1. 前端接收大文件，并进行切片处理
2. 将每份切片进行上传处理
3. 后端接收到所有的切片，存储所有切片到一个文件夹中
4. 将文件夹中的切片做合并，并对切片做删除
5. 再次上传统一文件时，能够快速上传

具体实现

1. 前端切片

   const BIG_FILE_SIZE = 25 * 1024 * 1024;
   const SLICE_FILE_SIZE = 5 * 1024 * 1024;
   
   const uploadFile = async () => {
       if (!fileList?.length) return alert("请选择文件");
       const file = fileList[0];
       const shouldUpload = await verifyUpload(file.name);
       if (!shouldUpload) return message.success("文件已存在，上传成功");
       if (file.size > BIG_FILE_SIZE) {
           // big handle
           getSliceList(file);
       }
       // // normal handle
       // upload("/uploadSingle", file);
   };
   const getSliceList = (file: RcFile) => {
       const sliceList: ISlice[] = [];
       let curSize = 0;
       let index = 0;
       while (curSize < file.size) {
           sliceList.push({
               id: shortid.generate(),
               slice: new File(
                   [file.slice(curSize, (curSize += SLICE_FILE_SIZE))],
                   `${file.name}-${index}`
               ),
               name: file.name,
               sliceName: `${file.name}-${index}`,
               progress: 0,
           });
           index++;
       }
       uploadSlice(sliceList);
       setSliceList(sliceList);
   };

   file 是一种特殊的 Blob 对象，可以使用 slice 进行大文件分割

   

2. 上传切片

   const uploadSlice = async (sliceList: ISlice[]) => {
     const requestList = sliceList
         .map(({ slice, sliceName, name }: ISlice, index: number) => {
             const formData = new FormData();
             formData.append("slice", slice);
             formData.append("sliceName", sliceName);
             formData.append("name", name);
             return { formData, index, sliceName };
         })
         .map(({ formData }: { formData: FormData }, index: number) =>
             request.post("/uploadBig", formData, {
                 onUploadProgress: (progressEvent: AxiosProgressEvent) =>
                     sliceUploadProgress(progressEvent, index),
             })
         );
     await Promise.all(requestList);
   };

   根据切片构建每个切片的 formData，将二进制数据放在 slice 参数中，分别发送请求。

   onUploadProgress 来处理每个切片的上传进度

   // Client
   const storage = multer.diskStorage({
     destination: async function (req, file, cb) {
         const name = file?.originalname.split(".")?.[0];
         const SLICE_DIR = path.join(UPLOAD_DIR, `${name}-slice`);
         if (!fs.existsSync(SLICE_DIR)) {
             await fs.mkdirSync(SLICE_DIR);
         }
         // 设置文件的存储目录
         cb(null, SLICE_DIR);
     },
     filename: async function (req, file, cb) {
         // 设置文件名
         cb(null, `${file?.originalname}`);
     },
   });
   
   // Server
   router.post(
       "/uploadBig",
       async (ctx, next) => {
           try {
               await next();
               const slice = ctx.files.slice[0]; // 切片文件
               ctx.body = {
                   code: 1,
                   msg: "文件上传成功",
                   url: `${RESOURCE_URL}/${slice.originalname}`,
               };
           } catch (error) {
               ctx.body = {
                   code: 0,
                   msg: "文件上传失败",
               };
           }
       },
       multerUpload.fields([{ name: "slice" }])
   );

3. 切片合并

   当我们所有的切片上传成功之后，我们依旧希望是按着原始文件作为保存的，所以需要对切片进行合并操作

   // Client
   const uploadSlice = async (sliceList: ISlice[]) => {
   		// ...和上述 uploadSlice 一致
   		mergeSlice();
   };
   
   const mergeSlice = () => {
       request.post("/mergeSlice", {
           size: SLICE_FILE_SIZE,
           name: fileList[0].name,
       });
   };
   
   // Server
   router.post("/mergeSlice", async (ctx, next) => {
       try {
           await next();
           const { size, name } = ctx.request.body ?? {};
           const sliceName = name.split(".")?.[0];
           const filePath = path.join(UPLOAD_DIR, name);
           const slice_dir = path.join(UPLOAD_DIR, `${sliceName}-slice`);
           await mergeSlice(filePath, slice_dir, size);
           ctx.body = {
               code: 1,
               msg: "文件合并成功",
           };
       } catch (error) {
           ctx.body = {
               code: 0,
               msg: "文件合并失败",
           };
       }
   });
   
   // 通过 stream 来读写数据，将 slice 中数据读取到文件中
   const pipeStream = (path, writeStream) => {
       return new Promise((resolve) => {
           const readStream = fs.createReadStream(path);
           readStream.on("end", () => {
               fs.unlinkSync(path);   // 读取完成之后，删除切片文件
               resolve();
           });
           readStream.pipe(writeStream);
       });
   };
   
   const mergeSlice = async (filePath, sliceDir, size) => {
       if (!fs.existsSync(sliceDir)) {
           throw new Error("当前文件不存在");
       }
       const slices = await fs.readdirSync(sliceDir);
       slices.sort((a, b) => a.split("-")[1] - b.split("-")[1]);
       try {
           const slicesPipe = slices.map((sliceName, index) => {
               return pipeStream(
                   path.resolve(sliceDir, sliceName),
                   fs.createWriteStream(filePath, { start: index * size })
               );
           });
           await Promise.all(slicesPipe);
           await fs.rmdirSync(sliceDir);  // 读取完成之后，删除切片文件夹
       } catch (error) {
           console.log(error);
       }
   };

4. 上传文件校验

   当我们上传一个文件的时候，先去判断在服务器上是否存在该文件，如果存在则直接不做上传操作，否则按上述逻辑进行上传

   // Client
   const verifyUpload = async (name: string) => {
       const res = await request.post("/verify", { name });
       return res?.data?.data;
   };
   
   const uploadFile = async () => {
       if (!fileList?.length) return alert("请选择文件");
       const file = fileList[0];
       const shouldUpload = await verifyUpload(file.name);
       if (!shouldUpload) return message.success("文件已存在，上传成功");
       if (file.size > BIG_FILE_SIZE) {
           // big handle
           getSliceList(file);
       }
       // // normal handle
       // upload("/uploadSingle", file);
   };
   
   // Server
   router.post("/verify", async (ctx, next) => {
       try {
           await next();
           const { name } = ctx.request.body ?? {};
           const filePath = path.resolve(UPLOAD_DIR, name);
           if (fs.existsSync(filePath)) {
               ctx.body = {
                   code: 1,
                   data: false,
               };
           } else {
               ctx.body = {
                   code: 1,
                   data: true,
               };
           }
       } catch (error) {
           ctx.body = {
               code: 0,
               msg: "检测失败",
           };
       }
   });

   上述直接使用文件名来做判断，过于绝对，对文件做了相关修改并不更改名字，就会出现问题。更应该采用的方案是根据文件相关的元数据计算出它的 hash 值来做判断。

   const calculateMD5 = (file: any) => new Promise((resolve, reject) => {
       const chunkSize = SLICE_FILE_SIZE
       const fileReader = new FileReader();
       const spark = new SparkMD5.ArrayBuffer();
       let cursor = 0;
       fileReader.onerror = () => {
           reject(new Error('Error reading file'));
       };
       fileReader.onload = (e: any) => {
           spark.append(e.target.result);
           cursor += e.target.result.byteLength;
           if (cursor < file.size) loadNext();
           else resolve(spark.end());
           
       };
       const loadNext = () => {
           const fileSlice = file.slice(cursor, cursor + chunkSize);
           fileReader.readAsArrayBuffer(fileSlice);
       }
       loadNext();
   });

   本文所有的代码可以github上查看

   总结

   本文从文件常识/常用的文件 API 入手，重点讲解了 Node 中 File 的相关实践，最后使用相关内容实现了大文件上传。

   更多node相关知识，请访问：nodejs 教程！

   The above is the detailed content of Let's talk in depth about the File module in Node. For more information, please follow other related articles on the PHP Chinese website!