So I am trying to take a single monitored log, and split sourcetypes based off of the terms SCAN, RECV, SEND. I created my props and transforms first. I made sure that splunk recognized the sourcetype in props, this was successfull. I created inputs stanza for the monitored file and the only sourcetype I see is test_barracuda, which was from the props stanza, I am not getting the split transforms should be doing.
####inputs.conf####
[monitor://C:\\Users\\eap\\Desktop\\security\\Splunk\\DEVELOPMENT\\Test_Ingest\\test_raw_spam.txt]
disabled = false
sourcetype = test_barracuda
index = test
####props.conf####
[test_barracuda]
CHARSET=AUTO
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true
category=Custom
disabled=false
pulldown_type=true
TRANSFORMS-overridest = set_sourcetype
####transforms.conf####
[set_sourcetype]
REGEX = \d+\s+(SEND|SCAN|RECV)\s
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::test_$1
This is an all in one test instance, so I placed these in SPLUNK_HOME/etc/apps/search/local/
Order of implementation: props.conf --> transforms.conf --> stop splunk --> clear event data from index test --> start splunk --> inputs.conf --> restart splunk.
Splunk gets the data. In the correct index, but only in sourcetype=test_barracuda.
I check to see if the regex in transforms is correct:
####Query
index=test sourcetype="test_barracuda" | rex field=_raw "\d+\s+(?PSEND|SCAN|RECV)\s"
Query works and I get exactly as many events in the 3 correct st fields.
So I do btools on props and transforms (note I'm not seeing any errors during /debug/refresh or splunk restart)
###btools
####C:\Program Files\Splunk\bin>splunk cmd btool transforms list set_sourcetype --debug
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf [set_sourcetype]
C:\Program Files\Splunk\etc\system\default\transforms.conf CAN_OPTIMIZE = True
C:\Program Files\Splunk\etc\system\default\transforms.conf CLEAN_KEYS = True
C:\Program Files\Splunk\etc\system\default\transforms.conf DEFAULT_VALUE =
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf DEST_KEY = MetaData:Sourcetype
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf FORMAT = sourcetype::test_$1
C:\Program Files\Splunk\etc\system\default\transforms.conf KEEP_EMPTY_VALS = False
C:\Program Files\Splunk\etc\system\default\transforms.conf LOOKAHEAD = 4096
C:\Program Files\Splunk\etc\system\default\transforms.conf MV_ADD = False
C:\Program Files\Splunk\etc\apps\search\local\transforms.conf REGEX = \d+\s+(SEND|SCAN|RECV)\s
C:\Program Files\Splunk\etc\system\default\transforms.conf SOURCE_KEY = _raw
C:\Program Files\Splunk\etc\system\default\transforms.conf WRITE_META = False
C:\Program Files\Splunk\etc\system\default\transforms.conf [set_sourcetype_to_stash]
C:\Program Files\Splunk\etc\system\default\transforms.conf CAN_OPTIMIZE = True
C:\Program Files\Splunk\etc\system\default\transforms.conf CLEAN_KEYS = True
C:\Program Files\Splunk\etc\system\default\transforms.conf DEFAULT_VALUE =
C:\Program Files\Splunk\etc\system\default\transforms.conf DEST_KEY = MetaData:Sourcetype
C:\Program Files\Splunk\etc\system\default\transforms.conf FORMAT = sourcetype::stash
C:\Program Files\Splunk\etc\system\default\transforms.conf KEEP_EMPTY_VALS = False
C:\Program Files\Splunk\etc\system\default\transforms.conf LOOKAHEAD = 4096
C:\Program Files\Splunk\etc\system\default\transforms.conf MV_ADD = False
C:\Program Files\Splunk\etc\system\default\transforms.conf REGEX = .
C:\Program Files\Splunk\etc\system\default\transforms.conf SOURCE_KEY = _raw
C:\Program Files\Splunk\etc\system\default\transforms.conf WRITE_META = False
####C:\Program Files\Splunk\bin>splunk cmd btool props list test_barracuda --debug
C:\Program Files\Splunk\etc\apps\search\local\props.conf [test_barracuda]
C:\Program Files\Splunk\etc\system\default\props.conf ANNOTATE_PUNCT = True
C:\Program Files\Splunk\etc\system\default\props.conf AUTO_KV_JSON = true
C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True
C:\Program Files\Splunk\etc\apps\search\local\props.conf CHARSET = AUTO
C:\Program Files\Splunk\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml
C:\Program Files\Splunk\etc\system\default\props.conf HEADER_MODE =
C:\Program Files\Splunk\etc\system\default\props.conf LEARN_MODEL = true
C:\Program Files\Splunk\etc\system\default\props.conf LEARN_SOURCETYPE = true
C:\Program Files\Splunk\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_AGO = 2000
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_HENCE = 2
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800
C:\Program Files\Splunk\etc\system\default\props.conf MAX_EVENTS = 256
C:\Program Files\Splunk\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
C:\Program Files\Splunk\etc\system\default\props.conf MUST_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE =
C:\Program Files\Splunk\etc\apps\search\local\props.conf NO_BINARY_CHECK = true
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION = indexing
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-all = full
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-inner = inner
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-outer = outer
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard
C:\Program Files\Splunk\etc\apps\search\local\props.conf SHOULD_LINEMERGE = true
C:\Program Files\Splunk\etc\system\default\props.conf TRANSFORMS =
C:\Program Files\Splunk\etc\apps\search\local\props.conf TRANSFORMS-overridest = set_sourcetype
C:\Program Files\Splunk\etc\system\default\props.conf TRUNCATE = 10000
C:\Program Files\Splunk\etc\apps\search\local\props.conf category = Custom
C:\Program Files\Splunk\etc\system\default\props.conf detect_trailing_nulls = auto
C:\Program Files\Splunk\etc\apps\search\local\props.conf disabled = false
C:\Program Files\Splunk\etc\system\default\props.conf maxDist = 100
C:\Program Files\Splunk\etc\system\default\props.conf priority =
C:\Program Files\Splunk\etc\apps\search\local\props.conf pulldown_type = true
C:\Program Files\Splunk\etc\system\default\props.conf sourcetype =
No conflicts, but I'm not seeing anything for the expected sourcetypes test_SEND, test_RECV, or test_SCAN
Any idea where I messed up?
####events for testing####
In case you want to test, here are 3 events that match the criteria for each expected sourcetype
Feb 13 12:14:57 192.168.x.x outbound/smtp: 127.0.0.1 1487013294-09b08c0e0d22d7b0001-vy5CMk 0 0 SEND - 1 0112918C8063 250 2.6.0 <2050829162.21743.1487013293752.JavaMail@dc1prjasszap434.whc> [InternalId=91736206477550, Hostname=host.prod.outlook.com] 11518 bytes in 0.192, 58.365 KB/sec Queued mail for delivery #to#name-com.mail.protection.outlook.com[8.8.8.8]:25
Feb 13 12:14:56 192.168.x.x scan: mail2-3.place.com[8.8.8.8] 1487013294-09b08c0e0d22d7b0001-vy5CMk 1487013294 1487013296 SCAN - prvs=7217d0fa1f=services_noreply@place.com name@otherplace.com - 7 88 corporate SZ:3263 SUBJ:Message: Attempt to retrieve your User ID
Feb 13 12:14:15 192.168.x.x inbound/pass1: name.place1.com[8.8.8.8] 1487013254-09b08c0e0e22d7a0001-VY5SBA 1487013254 1487013255 RECV information=place2.com@thing.com name@thingy1.com 2 3 blacklist.org[8.8.8.8]
↧