How Do You Efficiently Copy Sparse Files in Go Without Bloating?

Sparse Files Bloating with io.Copy()

In copying files with io.Copy(), large sparse files tend to inflate disproportionately at their destination. This behavior stems from io.Copy()'s handling of raw bytes, which lacks the means to convey information on file holes.

Handling Holes

To manage holes, you must delve deeper into the syscall package and tinker with the SEEK_HOLE and SEEK_DATA values for lseek(2). Unfortunately, these values are not present in the standard syscall package or its golang.org/x/sys counterpart.

Solution

To overcome this, follow these steps:

1. Map the syscall.Seek() to lseek(2) on supported platforms.
2. Determine the correct values for SEEK_HOLE and SEEK_DATA for your target platforms, as they can vary.

For instance, on Linux, you can use:

// +build linux

const (
    SEEK_DATA = 3
    SEEK_HOLE = 4
)

Reading and Transferring Sparse Files

Detect data regions and read from them while reading sparse files. For transferring files with sparseness intact, the process is more involved and requires platform-specific considerations.

Linux-Specific Approach

On Linux, use fallocate(2) with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE to punch a hole at the file's end. If it fails, shovel zeroed blocks to cover the hole.

Additional Notes

• Some filesystems, such as FAT, do not support holes.
• Check if the source and destination are on the same filesystem to consider using syscall.Rename() for faster file movement without copying data.

The above is the detailed content of How Do You Efficiently Copy Sparse Files in Go Without Bloating?. For more information, please follow other related articles on the PHP Chinese website!