n8n batch processing
In this posting I’ll describe how I set up my n8n workflow to update documents in Paperless NGX. My initial workflow was ok, but left room for improvement.
Initial Workflow
I’m using Paperless NGX to store all my analog and digital documents for archival. While Paperless does an OK job to recognise the document type and tags automatically with its built-in machine learning models, I wanted to also have a summary of the document and better document titles.
Since I’m scanning the analog documents with my printer all those documents are titled “EPSON WF-4830”.
I’ve set up a workflow in Paperless which call’s a web hook whenever a new document arrives or a special tag is added to a document.
The web hook is the trigger of a workflow on my N8N instance. The workflow would consume the document from Paperless, generate the summary, generate the title and update said document with the results.
To make it a little more interesting the LLM that N8N is using sits on my laptop, which n8n reaches via a Tailscale tunnel. That is not always on obviously.
First improvement attempt
In a first attempt to improve the workflow I wanted to use a queue for the documents. So that if my laptop isn’t available the documents from Paperless would just be added to the queue and processed later.
With the initial setup, the n8n workflow would just fail if it can’t reach the LLM for processing.
I create a separate workflow in n8n which takes the document id from the Paperless and inserts that into a simple data table in n8n.
A second workflow, which is triggered by a timer every 5 minutes, gets all the entries from the data table and processes them.
Most N8N nodes can deal with input arrays. Means if their input contains multiple elements, they will automatically iterate of all the elements in the input and execute their function on each item. The output of the node will again be an array.
This works fine if the process of the node is very robust. Means if every item is processed for sure. If a single items causes a problem, the node execution will fail entirely.
Subsequent steps are never reached. In my case that meant for longer batches there was a high chance that the whole batch failed because of a network issue or me shutting down my laptop.
The Improvement
N8N also has a dedicated “Loop” node. This one will take an array as input and pass a single item onto the next node until all items have been processed.
Using this node my workflow would only ever process a single document, update it and remove it from the data table. Failure to process one document will not stop the entire workflow.
Thus even if my laptop dies in the middle of the processing the remaining queue would be process next time it comes online and timer kicks off the processing.
Of course I had to add a mutex to prevent the processing queue from running twice at the same time. The entire processing workflow in N8N now looks something like this:
