How to Track and Report on Splunk .conf File Changes

Have you ever wanted to know what changes have been made to your Splunk .conf files? This was somewhat painful in the past. However, with version 9 Splunk itself will monitor your conf files and track changes that are made. The changes are stored in the _configtracker index in JSON format. This is what it looks like.

Source JSON

{
  "datetime": "08-01-2022 14:06:00.516 -0700",
  "log_level": "INFO ",
  "component": "ConfigChange",
  "data": {
    "path": "/opt/splunk/etc/users/splunkadmin/user-prefs/local/user-prefs.conf",
    "action": "update",
    "modtime": "Mon Aug  1 14:06:00 2022",
    "epoch_time": "1659387960",
    "new_checksum": "0x3ea7786da36c0d80",
    "old_checksum": "0xa01003ea5c398010",
    "changes": [
      {
        "stanza": "general",
        "properties": [
          {
            "name": "tz",
            "new_value": "",
            "old_value": "America/Denver"
          }
        ]
      }
    ]
  }
}

Flattening the Events

Most of the elements of the above JSON are pretty easy to comprehend. The catch comes with the changes (line 12) and properties (line 15) elements. These can have multiple values. So, the first things we need to do is flatten the JSON such that we create an event for each property modification under each change.

index="_configtracker" sourcetype="splunk_configuration_change"
| spath output=changes path=data.changes{}
| mvexpand changes
| spath input=changes output=properties path=properties{}
| mvexpand properties

App and Change Details

Once we’ve done that we can start pulling out fields. The first ones we will pull out are the app, the conf file type (default or local), the conf file, the modification date, and the action taken (update or add). For app, conf_type, and conf_file we can get those from the data.path field. We can split it by “/” for *Nix systems or “\” for Windows systems and then work from the right to the left to with the ultimate index split being the conf file, the penultimate being the conf file type, and the anti-penultimate being the app. Yes, I wrote it that way simply so I could use the words ultimate, penultimate, and anti-penultimate. For mod_time and action we will simply rename their data fields. While we are at it we will also format the mod_time field to be in YYYY-MM-DD HH:MM:SS format for easier legibility. If that’s not more legible to you then you can leave the last line out … but then we can’t be friends.

| eval path_type = if(match('data.path', "^.+/(local|default)/.+$"), "nix", "windows")
| eval app=if(path_type=="nix", mvindex(split('data.path', "/"), -3), mvindex(split('data.path', "\\"), -3))
| eval conf_type=if(path_type=="nix", mvindex(split('data.path', "/"), -2), mvindex(split('data.path', "\\"), -2))
| eval conf_file=if(path_type=="nix", mvindex(split('data.path', "/"), -1), mvindex(split('data.path', "\\"), -1))
| rename "data.modtime" as mod_time, "data.action" as action
| eval mod_time=strftime(strptime(mod_time, "%a %b %d %H:%M:%S %Y"), "%Y-%m-%d %H:%M:%S")

The Changed Values

This next part will pull out the stanza, property name, old value, and new value from the events based on how we expanded the event to flatten out the JSON.

| spath input=changes output=stanza path=stanza
| spath input=properties output=property path=name
| spath input=properties output=old path=old_value
| spath input=properties output=new path=new_value

Filling in the Blanks

Just to make the old and and new values a bit more legible if either value is blank let’s put {BLANK} in … this make me happy.

| eval old=if((old=="" OR isnull(old)), "{BLANK}", old)
| eval new=if((new=="" OR isnull(new)), "{BLANK}", new)

Formatting the Results

Finally, let’s display the fields we’ve extracted as a table and sort it by the modification time with the newest changes being shown first.

| table mod_time, app, conf_type, conf_file, stanza, action, property, old, new
| sort -mod_time, app, conf_type, conf_file, stanza

Full SPL Search

So, that’s it! You now have an SPL search that you can use to see how your conf files have changed over time. You can save this as a report, create an alert whenever a default conf file is modified, or you could use lines 1-16 as a base search in an accelerated data model. There are so many options! Hope this is helpful to you.

index="_configtracker" sourcetype="splunk_configuration_change"
| spath output=changes path=data.changes{}
| mvexpand changes
| spath input=changes output=properties path=properties{}
| mvexpand properties
| eval path_type = if(match('data.path', "^.+/(local|default)/.+$"), "nix", "windows")
| eval app=if(path_type=="nix", mvindex(split('data.path', "/"), -3), mvindex(split('data.path', "\\"), -3))
| eval conf_type=if(path_type=="nix", mvindex(split('data.path', "/"), -2), mvindex(split('data.path', "\\"), -2))
| eval conf_file=if(path_type=="nix", mvindex(split('data.path', "/"), -1), mvindex(split('data.path', "\\"), -1))
| rename "data.modtime" as mod_time, "data.action" as action
| eval mod_time=strftime(strptime(mod_time, "%a %b %d %H:%M:%S %Y"), "%Y-%m-%d %H:%M:%S")
| spath input=changes output=stanza path=stanza
| spath input=properties output=property path=name
| spath input=properties output=old path=old_value
| spath input=properties output=new path=new_value
| eval old=if((old=="" OR isnull(old)), "{BLANK}", old)
| eval new=if((new=="" OR isnull(new)), "{BLANK}", new)
| table mod_time, app, conf_type, conf_file, stanza, action, property, old, new
| sort -mod_time, app, conf_type, conf_file, stanza