Cisco IOS Syslog: Logging options to help standardize output

Modified on Wed, 25 Jun at 1:51 PM

Aggregating logs coming from your Cisco switches can be an amazing benefit to help monitor and alert on potential issues within your network. Unfortunately, the documentation on the output log formatting configurations can be difficult to locate within Cisco's documentation. Because aggregating network device logs is a very popular use case which can offer a lot of benefits with minimal effort, We wanted to provide some helpful tips, tricks, and settings options which can help make your deployment successful. This includes helping to understand some of the logging options which may need to be set to help keep your logs looking the same between different switch models and IOS versions, that may have different default settings. Cisco has detailed documentation on how to configure logging, so please refer to their documentation for additional information or for questions not answered in this Gravwell focused tips and tricks article.

Timestamp field

The Timestamp field within the Syslog messages is often going to be one of the most important data points, as it can be used to help identify when issues occurred. When aggregating logs, having unified and correct timestamps can have an extra importance as it can help you in syncing network events across multiple devices. There are several settings and important things to be aware of which can help ensure your data is being logged correctly. Most of the below information can be found in Cisco's Documentation

The default setting is to not include timestamps in the messages. To enable the timestamps, you would use the service command within the switch's cli.

service timestamps log datetime

This example will enable the timestamps in logging messages using UTC time in the MMM DD HH:MM:SS format. You can optionally add the msec, localtime, and/or show-timezone options to add milliseconds, display the time using localtime, and/or include the local timezone within the stamp output.

If the timestamp appears with an Asterisk * at the start of the time, ie  *Jan 02 15:04:05, it is an indication that the device's system clock has not been set and therefore the output timestamp is likely incorrect.

Being aware of how the time is formatted can be very important when configuring the Simple Relay listener for your Cisco logs as there are several timezone and timestamp related configuration options.

Sequence Numbers

Sequence Numbers can be thought of as a log ID or Event ID within the device. They can be useful in helping identify the order in which events occurred when timestamps are not granular enough. (IE, when multiple events happen at the same time). They can also be helpful in locating the event within the device's internal logs.

dret

service sequence-numbers

Syslog Origin identification

A common issue as seen online when doing log aggregation on Cisco devices, is ensuring that you can identify which device the logs are coming from. Within Gravwell, we automatically log the IP address we receive the messages from, which you can display using the SRC enumerated value in your queries. You can also use the src search module to filter based on that IP.

But as useful as knowing the IP address the log came from can be, many people are more comfortable and familiar with the hostnames they use for their network devices than the IP's. As such, Cisco provides a way to include the hostname of the device within the Syslog messages it sends to an external syslog receiver. The origin-id command allows you to configure this field within the syslog message.

logging origin-id hostname

This command will set the origin-id sent in the syslog message to the device's hostname. There are other options, including ip, ipv6, and string which can be configured if the hostname isn't what you'd prefer to use.

Gravwell Autoextractor

Cisco's logging format, while called syslog, doesn't quite fit the RFC Syslog standards. Because of this, using Regex in your auto extractor configuration is likely going to be the best option. Due to the amount of variability in how you can configure the syslog messages, it is almost impossible to provide a solid autoextractor that will work for every use case or combination of options. The important thing will be to standardize in your deployment to a standard configuration so that whatever autoextractor you deploy will be able to properly handle the structure of the logs for all your devices.

That said, we want to provide a starting point that should potentially work with the options set above. When working through your regex, Regex 101 can be an invaluable resource. Gravwell also hosts a clone at https://regex101.labs.gravwell.io/ which you can feel comfortable pasting examples from your system as we do not retain any data.

^.+:(?P<hostname>.+?): (?P<logid>\d*): (?P<date>.+?): (?P<type>.+?): (?P<message>.*)