Potentially that would be a good application of federation and distributed computing

An Internet archive like distributed tool, that then feeds into local tokenization and indexing.

Alternatively a centralized service that generates indices and then locally they are queried would save a lot of energy.