Hi All,
So following this excellent blog post I thought I found a solution to ingesting a binary logfile with Splunk.
https://www.splunk.com/blog/2011/07/19/the-naughty-bits-how-to-splunk-binary-logfiles.html
Unfortunately nothing is making it into Splunk at all.
**inputs.conf**
[monitor://$SPLUNK_HOME/etc/apps/TA-myBinary/spool/*.evl]
disabled = 0
index = myBinary
sourcetype = myBinary:evl
followTail = 0
**props.conf**
[myBinary:evl]
NO_BINARY_CHECK = true
invalid_cause = archive
unarchive_cmd = /opt/splunk/etc/apps/TA-myBinary/bin/decode_evl.py
unarchive_sourcetype = myBinary:evl:unarchived
priority = 10
TIME_FORMAT = %Y-%m-%d %H:%M:%S
SHOULD_LINEMERGE = false
Some of those additional props entries (unarchive_sourcetype, priority) I discovered as attempts to resolve the issue from searching answers high and low.
Sadly nothing makes it into Splunk and there are no errors during processing.
The script runs just fine and will extract the data manually.
cat ../spool/20170529.evl | ./decode_evl.py > ../spool/20170529.evl.out
Initially I thought it might be my script not writing to stdout so I made sure it did that.
#!/usr/bin/python
import os, sys, json, time
from datetime import datetime
from HTMLParser import HTMLParser
import logging
import pprint
import binascii
..... decoding components......
while 1:
splunkEvent = readEvent()
if len(splunkEvent['splunkMessage']) == 0:
break
sys.stdout.write(splunkEvent['splunkMessage'] + '\n')
# Flush out any extra data
sys.stdout.flush()
sys.exit()
**splunkd.log**
06-15-2017 09:06:23.165 +1000 INFO ArchiveProcessor - Handling file=/opt/splunk/etc/apps/TA-myBinary/spool/20170529.evl
06-15-2017 09:06:23.165 +1000 INFO ArchiveProcessor - reading path=/opt/splunk/etc/apps/TA-myBinary/spool/20170529.evl (seek=0 len=59936626)
06-15-2017 09:06:23.324 +1000 INFO ArchiveProcessor - Finished processing file '/opt/splunk/etc/apps/TA-myBinary/spool/20170529.evl', removing from stats
Now that I look at the above timings. This particular file takes at least 1 minute to parse the binary file and the three log events above all happen within a second of each other.
Anyone with any thoughts on what could be wrong here?
↧