Create PHP parallel processing from getResult principle request

Question

I want to do some kind of multi-threading in a PHP8/Symfony6 environment and it works like this:

I'm making a principle request that serves about a million rows

I want to create multiple parallel processes to process these rows at the same time. My request is made via getResult() which serves the doctrine object. Is there a way to create a thread with an array of objects as parameter. Then create threads whenever my "queue" is filled with data?

I found some information about "pthreads" or "popen" but I'm not sure if it's still relevant since the topic is a bit old

Answer 1

In this case, one of the things Doctrine does is hydrate a class with the data content, so all those millions of records are converted to classes and hold them in memory, so a couple of suggestions: p>

1. Avoid using getResult in large queries. Instead, if you want to do something with this object, try paginating and clearing the entity manager, like:

public function paginate(int $start = 0, int $offset = 10): Doctrine\ORM\Tools\Pagination\Paginator
{
    $qb = $this->createQueryBuilder('student');
    $qb
       ->where($qb->expr()->isNull('student.schoolRef'))
       ->setFirstResult($start)
       ->setMaxResults($offset);
    return new Paginator($qb, true);
}

src/Service.php

$start = 0;
$offset = 100;
do {
    $i = 0;
    $students = $this->studentRepository->paginate($start, $offset);
    foreach($students as $student) {
    //do stuff
      ++$i;
    }
    $this->entityManager->flush();
    $this->entityManager->clear();
} while ($i == $offset);

2. Use while to query directly:

<?php

$sql = <<<SQL
SELECT
    *
FROM user u
SQL;

$statement = $connection->prepare($sql);

$statement->execute();

while ($user = $statement->fetch()) {
    // print_r($user);
}

You can also use the fetchAll method to fetch all users at once, but be aware that you may run out of memory if your query returns too many users. Using the while loop above, you are only fetching one user into memory at a time, so memory performance is better.

<?php

// ...

// all users are in memory here
$users = $connection->fetchAll($sql);

foreach ($users as $user) {
    // print_r($user);
}

As someone commented, you can use

• Symfony Process
• Symfony Messenger