Scheduled Searches: Best Practices

Scheduled Searches are the most basic form of automation within Gravwell, and will be one of the most important tools in your toolkit.  In combination with Alerts and Flows, Scheduled searches will be something you will find yourself using quite extensively.

Because of how useful Schedules Searches are in a production environment, we wanted to provide some Best Practices in their usage which will help you get the most out of them.

1. Utilize the Query Library

While you can schedule any search, including ad-hoc ones, as a best practice we recommend using your Query Library as the basis for any Scheduled Search.  We find using the Query library for scheduled searches allows for a lot a flexibility, and simplifies the management of the query as it evolves and is refined over time.

By using the Query Library instead of having your search directly within the Scheduled search, it provides a single point which you can modify a query, instead of potentially having multiple locations which must be touched.  It also makes it easier to take the underlying query and run it ad-hoc or include it in a dashboard.

2.  Take advantage of the Offset setting

The primary use case around a Scheduled search is to look over a repeating window in the past for reporting or alerting purposes.  In theory, simply running a search over the past 5min, 15min, 1 hr, or even 1 day, is pretty straightforward, and you'd be able to catch anything that came in as your search window and scheduled search timing would line up.  Unfortunately, the reality of networking or complex system don't always have things working as you'd expect on paper.

For example, let's say you have a query of syslog data coming from a dozen systems running at the top of every hour looking at the past hour's data.  Under ideal conditions, any events that happened in the previous 60min would have been received by the indexers and be in the search results.  However, network delays or hiccups in the system sending the data, can sometimes cause the information to not be received by the indexer until a few minutes after the event occurred.  As data is indexed based on the event time (usually included in the syslog message), and not simply on when it is received, then it's possible that you can have events that are never identified in a search because it's added to a previous scheduled search's window.

Utilizing the offset allows you to include a delay between when the search is ran and the window it is searching, to allow for any delayed events to be received before the search is run.  As an example, this will allow you to run a search that runs at 01:05 searching all the events that occurred between 00:00 and 01:00.  If there was a network or other event which delayed the delivery of the data to your Gravwell cluster, the offset would give you a 5-minute cushion for the data delivery.

3.  Enable the Backfill option

The Backfill option helps ensure that no matter what happens, you don't miss an important alert.  There may be times, such as during maintenance windows, or system outages, where for one reason or another a Scheduled Search doesn't run as expected.  The Backfill option tells the system that if a scheduled invocation of your Automation does not run, or fails, for any reason, to go back and re-run the automation with the time windows in place as if it had run as originally scheduled.

In practice, this will help you feel confident that something as simple as scheduled maintenance which could bring down parts of your Gravwell cluster, will not create a blind spot within your system monitoring where an event could happen that you'd never be alerted on.

4.  Develop Output Standards

In most cases, the results from a scheduled search are used to detect activity that will then be used to trigger one or more actions: send an email notification, run a historical search, update a lookup resource, reingest data from later analysis, pull data from an intel source, etc.; this is accomplished by pairing one or more scheduled searches with an alert that passes the results to one or more flows.  Whether you're developing a novel search or tuning an existing one, it is important to plan the results from your searches that share similar data sources, flow actions, etc. to simplify the creation, tuning of searches and keep content tidy.  For instance, if you have 20 Windows searches that report user activity and send an email, it would be advised to group those with a single alert rather than 20 scheduled searches each to a single alert.

This list is by no means a definitive list of best practices, as is intended only as a starting point to help you develop your own best practices.  As your deployment matures, you will likely find additional ways of doing things which will simplify both the overall management and upkeep of the system, as well as your day-to-day usage.