Splunk Investigations

Guide for Splunk Usage

Basic Concepts

• index = bucket of logs

  • ex. wineventlog

  • can search by index to get all logs in that index

• sourcetype = log source

  • ex. XmlWinEventLog

• there are normalized fields with specific values

  • host

  • user

  • user_id/UserID

  • EventID_Description/signature/signature_description

  • action

  • signature_id/EventID/EventCode

  • user/UserName

  • tcp_flag

    • automatic lookup field used with mapping flags to codes

  • action

  • src/src_ip

  • dest/dest_ip

  • src_port and dest_port

  • ThreadID (use single quotes around the ID numbers)

• Splunk uses epoch time

  • can convert with | convert ctime(time_name)

• can link commands together with |

• uses boolean logic for fields

  • if you don't specify something like OR, it automatically assumes AND

• sourcetype vs. source

  • source is where the actual logs come from- ex. a .log file

• wildcarding before a string will run slower than wildcarding after a string

Important Commands

Working with Indices and Sourcetypes

List all indices and sourcetypes

| tstats count WHERE index=* OR sourcetype=* BY index,sourcetype | stats values(sourcetype) AS sourcetypes BY index

List all indices, sourcetypes, and sources for a host

| tstats values(source) AS sources WHERE index=* host=[hostname] BY host index sourcetype

table

Shows results of only required field values in a table

| table [value, ex. host]

dedup

Removes duplicate values in a table

| dedup [value, ex. host]

stats

Calls mathematical functions

| stats [sum(numbers), avg(numbers), count(action), etc.) BY _time span=[ex. 5min]

When combining two functions, put the "by" clause after the functions:

| stats sum(number) as sum_total avg(number) as avg_total by [ex. dest]

Getting all the values of a field and how many there are based on another field

| stats values(field1) count by [field2]

Deduplicating field1 values

| stats values(field1) dc(field1) by [field2]

Getting a count of how many times distinct values in that field appear- limits what fields appear

| stats count by [field1] [field2]

If one field doesn't exist, you won't get results at all

_time

Used after "by" to show what time an event occurred, as well as the earliest/latest time something occurred

| [] by _time / latest_time / earliest_time

Converting Properly - have to call out the times in order to convert

[search query]
| stats earliest(_time) as earliest latest(_time) as latest by [field1] [field2]
| convert ctime(latest)
| convert ctime(earliest)

Lookup Tables

List values in a lookup table

| inputlookup [lookup table]

Match field values to values in a lookup table

| lookup[lookup table] [field in lookup table] as [field in log event]

Can add stats command to show less

| stats count by [field]

Show only logs where a field in the lookup has a value

| where isnotnull(field)

Output only a certain field

| lookup [lookup table] [lookup field] as [log field] output [lookup field 2]

Can chain lookups together

[search query]
| lookup [lookup1] [lookup field] as [log field]
| where isnotnull(field2)
| lookup [lookup2] [lookup field 2] as [log field 2]

To output a lookup field that doesn't have a match in the log fields, find another field in the lookup that does match to a log field and use that with lookup as to output lookup fields from that table

Ex. from query above, you're looking for lookup field 3 but it has no log field equivalent, but lookup field 2 does- the above query will output lookup field 3 in the results along with everything else

Counting Events

Get count of events occurring matching the query

| tstats count [ex. from datamodel=Web] WHERE index=[] AND sourcetype=[] AND etc.

rename

Rename a field name as something else

| rename [default field name] AS [new name]

Data Models

Looking at data models can show you fields aligned to certain types of logs

This is field normalization in Splunk- logs from different sources that are all network logs will all map to the same field

Called the Common Information Model

However, the same field in different sources will not always mean the same thing