A small change can save 70% of json_decode memory?

json_decode decodes the JSON format string, accepts a JSON format string and converts it into a PHP variable. But when running json_decode, the memory may be exceeded. What should I do?

If you use PHP's json_decode function to parse a JSON string, and the JSON string contains an array with a large number of elements, then you need to be careful that PHP exceeds the memory limit during the parsing process. .

The author encountered a JSON file that needed to be parsed during development. The JSON contained an array composed of many MAC addresses, like this:

{
    "name": "MAC File",
    "date": "2017-11-08",
    "macList": [
        "11-11-11-11-11-11",
        "22-22-22-22-22-22",
        ...
    ]
}

As a result, the json_decode process exceeded the PHP default 128M memory limit.

WHAT, over the limit? ! This JSON file is only 10M!

After cursing "Is there a bug in this function?", after careful consideration, I found that the problem lies in the array composed of MAC addresses. You must know that PHP arrays consume a lot of memory.

How much memory does PHP array use? You can do a simple experiment by putting 500,000 MAC addresses into the array and printing the memory usage:

$a = [];
for ($i = 0; $i !== 500000; $i++) {
    $a[] = '11-11-11-11-11-11';
}
echo memory_get_usage() . PHP_EOL;

If you write these MAC addresses in A file theoretically only occupies 9.6M of disk space, but the PHP array maintains the same information, but occupies 72.4M of memory.

Is there a way to solve the memory overrun during json_deocde? Of course, to be simple and crude, just increase the memory limit:

ini_set('memory_limit','1024M');

Although it is feasible, it will cause a problem, that is:

may be laughed at by engineers of other languages ​​​​for PHP’s memory footprint .

Is there a smarter way to solve the memory overrun problem?

have. Because PHP arrays take up a lot of memory, we need to avoid json_decode from generating huge arrays when decoding. How to do it? This starts with the JSON encoding format. For example, you can modify a huge JSON array into a string:

{
    "name": "MAC File",
    "date": "2017-11-08",
    "macList": "11-11-11-11-11-11，22-22-22-22-22-22，...",
}

I converted macList from an array to a comma-separated string. This prevents json_decde from generating a huge array and replaces it with an extremely long string.

The amount of memory occupied by strings is much smaller than that of arrays. The 500,000 MAC addresses just now occupied only 9.7M of memory. After the modification, json_decode was parsed successfully and the parsing speed was faster.

Originally macList was an array, and the elements in it could be traversed through foreach. Now it is a string. How to traverse it?

It’s not difficult, you can use strtok:

$tok = strtok($macList, ',');
while ($tok !== false) {
    $mac = $tok;
    $tok = strtok(',');
}

The difficulty of traversal doesn’t increase much, right?

You may ask, this method can deal with simple JSON arrays. What if each element of a JSON array is a JSON object?

We can construct the string like this:

{
    "list": '{"name":"obj1"}###{"name":"obj2"}###...'
}

The string consists of small JSONs, separated by special marks

. During parsing, JSON objects are segmented according to special tags, and then parsed one by one using json_decode:

$tok = strtok($objectList, '###'); // 按###切割
while ($tok !== false)
{
    $objectStr = $tok;
     // 每切割出一个JSON对象就解码
    $object = json_decode($objectStr, true);
    $tok = strtok('###');
}

You can also create your own method of encoding/parsing this very long string. In short, the ultimate goal is Avoid json_decode generating very large arrays during the decoding process.

Through this article, you should have a glimpse of PHP Array’s ability to eat memory. Replacing arrays in JSON with string representations can save a lot of memory. I also ran a comparative data for your reference:

Parse 500,000 MACs: Save MAC address methodArray MethodString methodJSON file size9.6M8.6M (each The element omits a pair of quotation marks) Average memory usage72.4M8.7M Average json_decode parsing time0.73s0.41s

Parsing 1 million MACs:Save MAC address methodArray methodString methodJSON file size20M18MAverage memory usage204.6M54.2MAverage json_decode parsing time1.61s0.81s

##Parsing 2 million MACs:

Save MAC address method	Array method	String method
JSON file size	40M	36M
Average memory usage	409.0M	108.2M
Average json_decode parsing time	3.05s	1.53s

Recommended learning: php video tutorial

The above is the detailed content of A small change can save 70% of json_decode memory?. For more information, please follow other related articles on the PHP Chinese website!