How to load balance across multiple indexers
If you you want to configure the Simple Relay ingester to load balance such that if Simple Relay generates 100 entries, you'd like indexer1 to store 50 of them and indexer2 to store the other 50, you've come to the right spot. Here is how to splitting the load across the two of them, without duplication.
We have actually built that in to the default configs. Just specify multiple Cleartext-Backend-Target or Encrypted-Backend-Target statements in the ingester's [Global] section, e.g.:
Encrypted-Backend-Target=10.0.0.1:4024
Encrypted-Backend-Target=10.0.0.2:4024
This will work provided both indexers are configured to use the same
Ingest-Secret, which should usually be the case.
Note that this is not strict "round-robin"; we push entries to
indexers as fast as they can accept them, so if your indexers are on
vastly disparate hardware one may end up receiving more entries than
the other.