How Can I Get the Real-Time File Size of a Server-Bound Upload?

Question:

Obtain the real-time file size of an uploaded file while it is being written to the server without blocking both the server and client.

Context:

File upload progress in a client's browser while writing to the server using fetch()'s POST request with a File or Blob body.

Requirement:

Display the file size as text/event-stream while it's being written to the server filesystem. Stop when all bytes provided as a query string parameter during the file upload have been written. The file size is currently retrieved from a separate script, which is called after the file has been written to the server.

Implementation:

Initially attempted using PHP but faced errors due to undefined HTTP_LAST_EVENT_ID and incorrect file size being reported. Also experimented with different approaches and languages like bash, c, nodejs, and python.

Solution:

1. Clear file stats cache to obtain real-time file size:

<code class="php">clearstatcache(true, $upload);
$data = filesize($upload);</code>

2. Modified stream.php with error handling and using usleep for better performance:

<code class="php">// Check if the header's been sent to avoid `PHP Notice:  Undefined index: HTTP_LAST_EVENT_ID in stream.php on line `
// php 7+
//$lastId = $_SERVER["HTTP_LAST_EVENT_ID"] ?? 0;
// php < 7
$lastId = isset($_SERVER["HTTP_LAST_EVENT_ID"]) ? intval($_SERVER["HTTP_LAST_EVENT_ID"]) : 0;

$upload = $_GET["filename"];
$data = 0;
// if file already exists, its initial size can be bigger than the new one, so we need to ignore it
$wasLess = $lastId != 0;
while ($data < $_GET["filesize"] || !$wasLess) {
    // system calls are expensive and are being cached with assumption that in most cases file stats do not change often
    // so we clear cache to get most up to date data
    clearstatcache(true, $upload);
    $data = filesize($upload);
    $wasLess |= $data < $_GET["filesize"];
    // don't send stale filesize
    if ($wasLess) {
        sendMessage($lastId, $data);
        $lastId++;
    }
    // not necessary here, though without thousands of `message` events will be dispatched
    //sleep(1);
    // millions on poor connection and large files. 1 second might be too much, but 50 messages a second must be okay
    usleep(20000);
}</code>

3. Include fileId and fileSize as part of the POST request, and add in-memory storage using redis or memcache to store file metadata.
4. Utilize EventSource in the client-side JavaScript:

<code class="javascript">const [fileId, request, source] = [
    Math.random().toString(36).substr(2),
    new Request(`${url}?fileId=${fileId}&amp;size=${filesize}`, {
        method: "POST",
        headers: headers,
        body: file
    }),
    new EventSource(`${stream}?fileId=${fileId}`)
];</code>

5. Customize setUnique and updateProgress functions based on your chosen storage mechanism:

<code class="php">function setUnique(string $id, int $size) {
    // implement with your storage of choice
}

function updateProgress(string $id, int $processed) {
    // implement with your storage of choice
}</code>

6. Obtain progress from the storage:

<code class="php">list($progress, $size) = getProgress($_GET["fileId"]);</code>

Important Notes:

• The provided solution prioritizes functionality over security and is not recommended for production use without implementing additional security measures.
• The number of open connections may need to be adjusted based on server configuration and resource utilization.
• Instead of EventSource, using polling can be considered to reduce the number of open connections, but it may impact responsiveness.
• The optimal sleep time in usleep() may vary depending on the desired update frequency and performance trade-offs.

The above is the detailed content of How Can I Get the Real-Time File Size of a Server-Bound Upload?. For more information, please follow other related articles on the PHP Chinese website!