Lets review some common errors and logs to check
If you are running searches and receiving errors about indexes being unavailable there are a couple things to check.
Take a look at error.log on web server:
07-08-2020 15:03:03.231 [80000079] search/client_commands.go:489 GOB decode search_api.ServerResponse failure: read tcp 192.168.100.1:58498->192.168.100.2:9404: i/o timeout
07-08-2020 15:03:03.231 [80000287] search/manager.go:444 Error from client net:192.168.100.2:9404: net:192.168.100.2:9404 : read tcp 192.168.100.1:58498->192.168.100.2:9404: i/o timeout
07-08-2020 15:03:03.332 [80000076] webserver/indexPool.go:611 Failed to RunQuery: net:192.168.100.2:9404 : read tcp 192.168.100.1:58498->192.168.100.2:9404: i/o timeout
07-08-2020 15:03:08.235 [8000012d] webserver/searchSocket.go:514 Failed to connect to to indexers: No indexes are available
The first few errors come when the indexer takes too long to acknowledge the request to start a search. The later errors are because the web server cannot seem to connect to any of the indexers. Unfortunately we can't really tell from these messages if the indexer process went *down* or if it just got so loaded it took too long to respond.
Take a look at the error.log and warn.log for the *indexer* at around that time period. It may shed some light on what happened. Checking how long the indexer process has been running as well should help us know if the process died or not.
If you're still having some trouble please reach out with the above mentioned logs to support@gravwell.io, we're here to help.