Batch processing matters on almost every Drupal project, and it matters even more when we have to work through a large amount of data. The idea is to split one heavy job into small chunks, each run as its own page request, so we never ask the server to do everything in a single load.
That is what keeps the process from dying on a PHP timeout, and it lets the user watch a progress bar instead of a frozen screen. A few typical jobs for the Batch API:
- import or migrate data from an external source,
- clean up internal data,
- run an action on a set of nodes,
- talk to an external API for each item.
A batch is usually launched from a form. But what if we want a *nix crontab to run it on a schedule, with nobody sitting in front of the browser? One of the cleanest answers is to fire the batch from a custom Drush command, and let the crontab call that command.
In this post we'll build a custom Drush command that loads every node of a content type passed as an argument (page, article, and so on), then runs a batch that simulates a long operation on each node. After that we'll see how to launch the command from a crontab. The original Drupal 8 example lives at github.com/KarimBoudjema/Drupal8-ex-batch-with-drush9-command; the code below is rewritten for Drupal 11.
Here is the tree of the module. Note how much simpler it is than the Drupal 8 version: no drush.services.yml and no composer.json service declaration anymore.
web/modules/custom/ex_batch/
|-- ex_batch.info.yml
`-- src
|-- BatchService.php
`-- Drush
`-- Commands
`-- ExBatchCommands.phpWe'll proceed in three steps:
- a
BatchServiceclass to host the two batch callbacks (BatchService.php); - a custom Drush 13 command that loads the nodes and fires the batch (
ExBatchCommands.php); - a crontab task that runs the command automatically at set times.
Let's scaffold the module with Drush 13's generator (Drupal Console is long gone):
drush generate module1. A BatchService class for the batch callbacks
A batch is built around two callbacks: one that processes each chunk, and one that runs at the end. It's good practice to keep them out of the .module file, so here they live as two static methods on a small class we could reuse later: processNode() for each item, and finished() for the wrap-up.
<?php
declare(strict_types=1);
namespace Drupal\ex_batch;
/**
* Hosts the batch operation and finished callbacks.
*/
final class BatchService {
/**
* Processes one node. Called once per batch operation.
*
* @param int $nid
* The node ID to process.
* @param string $operationDetails
* A short message describing the operation.
* @param array $context
* The batch context, passed by reference and persisted between chunks.
*/
public static function processNode(int $nid, string $operationDetails, array &$context): void {
// Simulate a long operation. Here we would load the node, call an
// external API, rewrite a field, and so on.
usleep(100000);
// Anything stored under 'results' is handed to finished() at the end.
$context['results'][] = $nid;
// Shown under the progress bar while the batch runs.
$context['message'] = t('Processing node @nid: @details', [
'@nid' => $nid,
'@details' => $operationDetails,
]);
}
/**
* Runs once, after every operation is done.
*
* @param bool $success
* TRUE if no operation threw an uncaught exception.
* @param array $results
* Everything the operations pushed onto $context['results'].
* @param array $operations
* Any operations left unprocessed (only on failure).
*/
public static function finished(bool $success, array $results, array $operations): void {
$messenger = \Drupal::messenger();
if ($success) {
$messenger->addStatus(t('@count node(s) processed.', ['@count' => count($results)]));
return;
}
// On failure, $operations holds what was left to do.
$failed = reset($operations);
$messenger->addError(t('An error occurred while processing: @op', [
'@op' => print_r($failed, TRUE),
]));
}
}processNode() is where the real work would go: here we just wait 100 milliseconds with usleep() to stand in for a slow task. Two lines are worth pointing at. $context['results'][] collects a value per item, and that whole array reaches finished() as its $results argument. $context['message'] is the line Drupal shows under the progress bar. In finished() we simply report how many nodes went through, or surface what was left over on failure.
2. The custom Drush 13 command
This is the heart of the module. In Drupal 8 a Drush 9 command needed three files: a drush.services.yml, a composer.json with an extra.drush.services section, and the command class itself with its @command annotations. None of that is needed anymore. In Drush 13 a command is just a class under src/Drush/Commands/, and Drush discovers it from the #[CLI\Command] PHP attribute: annotations were dropped along with the old service file.
<?php
declare(strict_types=1);
namespace Drupal\ex_batch\Drush\Commands;
use Drupal\Core\Batch\BatchBuilder;
use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\Core\Logger\LoggerChannelFactoryInterface;
use Drupal\ex_batch\BatchService;
use Drush\Attributes as CLI;
use Drush\Commands\DrushCommands;
use Symfony\Component\DependencyInjection\ContainerInterface;
/**
* Runs a batch over every node of a given content type.
*/
final class ExBatchCommands extends DrushCommands {
public function __construct(
private readonly EntityTypeManagerInterface $entityTypeManager,
private readonly LoggerChannelFactoryInterface $loggerFactory,
) {
parent::__construct();
}
/**
* {@inheritdoc}
*/
public static function create(ContainerInterface $container): self {
return new self(
$container->get('entity_type.manager'),
$container->get('logger.factory'),
);
}
/**
* Processes every published node of a content type in a batch.
*/
#[CLI\Command(name: 'exbatch:update-nodes', aliases: ['exbatch'])]
#[CLI\Argument(name: 'type', description: 'The content type (node bundle) to process.')]
#[CLI\Usage(name: 'drush exbatch:update-nodes article', description: 'Run the batch over every published article.')]
public function updateNodes(string $type = 'article'): void {
$this->loggerFactory->get('ex_batch')->info('Batch update started for @type nodes.', ['@type' => $type]);
// 1. Load every published node of this type.
$storage = $this->entityTypeManager->getStorage('node');
$nids = $storage->getQuery()
->condition('type', $type)
->condition('status', 1)
->accessCheck(FALSE)
->execute();
if (!$nids) {
$this->logger()->warning(dt('No published @type node(s) found.', ['@type' => $type]));
return;
}
// 2. Build the batch: one operation per node.
$batch = (new BatchBuilder())
->setTitle(dt('Processing @count @type node(s)', ['@count' => count($nids), '@type' => $type]))
->setFinishCallback([BatchService::class, 'finished']);
foreach ($nids as $nid) {
$batch->addOperation(
[BatchService::class, 'processNode'],
[(int) $nid, dt('Updating node @nid', ['@nid' => $nid])],
);
}
// 3. Register the batch, then let Drush run it to completion.
batch_set($batch->toArray());
drush_backend_batch_process();
$this->logger()->success(dt('Batch update finished.'));
}
}We inject two core services in the constructor: entity_type.manager to load the nodes, and logger.factory to log the start and end. The create() method wires them from the container, exactly as a controller would.
The command itself is the updateNodes() method, and three attributes describe it:
#[CLI\Command(name: 'exbatch:update-nodes', aliases: ['exbatch'])]
#[CLI\Argument(name: 'type', description: 'The content type (node bundle) to process.')]
#[CLI\Usage(name: 'drush exbatch:update-nodes article', description: 'Run the batch over every published article.')]#[CLI\Command] gives the command its name (and an alias); #[CLI\Argument] documents the $type argument; #[CLI\Usage] shows an example in the command help. This replaces the old @command, @aliases and @usage annotations one-for-one.
The interesting part is the batch itself. In Drupal 8 we hand-wrote a $batch array; in Drupal 11 we build it fluently with BatchBuilder:
$batch = (new BatchBuilder())
->setTitle(dt('Processing @count @type node(s)', ['@count' => count($nids), '@type' => $type]))
->setFinishCallback([BatchService::class, 'finished']);
foreach ($nids as $nid) {
$batch->addOperation([BatchService::class, 'processNode'], [(int) $nid, $details]);
}addOperation() queues one call to our processNode() callback per node, with its arguments; setFinishCallback() points at finished(). When the batch is ready we register it with batch_set($batch->toArray()) and then call drush_backend_batch_process(), the Drush helper that actually drives a Drupal batch from the command line by spawning the requests itself.
That's it. Clear the cache with drush cr and run the command:
drush exbatch:update-nodes article3. Run the Drush command from a crontab
Now we want the crontab to run the command on its own, on a schedule. The exact steps depend on the server's operating system; on Linux, macOS and Unix we edit a crontab that runs jobs at set intervals.
Open your crontab for editing (this opens your default editor):
crontab -eAdd a scheduled line with our command (replace [docroot] with your site's docroot path). The first line sets a PATH so cron can find the drush binary:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
*/5 * * * * cd [docroot] && drush exbatch:update-nodes articleThis runs the command every five minutes. Since we log the start and end inside the command, we can watch the Drupal log to confirm it fires, or read cron's own mail:
sudo tail -f /var/mail/rootRecap. We wrote a BatchService class with two static callbacks, processNode() for each item and finished() for the wrap-up. We wrote a custom Drush 13 command that loads every published node of a content type, builds the batch with BatchBuilder, registers it with batch_set() and runs it with drush_backend_batch_process(). Finally we scheduled the command in a crontab. Modernised for Drupal 11, the real change from the Drupal 8 original is the Drush command itself: no drush.services.yml, no composer.json service block, just a class under src/Drush/Commands/ with the #[CLI\Command] attribute, and a batch built with BatchBuilder instead of a raw array.
This way we can run heavy jobs on a regular basis without ever putting the server under undue stress. How do you launch your own batches, from a form or from the CLI? I'd love to hear about it in the comments.
More info