Hi,
I am trying to escape backslash character from json data. It works when I apply SEDCMD definations in props.conf soucetype - mysrc. But when I re-direct the definations to transforms.conf (custom_data_one and custom_data_two) to transform data for particular pattern & extract required data from the json event, and then apply the SEDCMD in mentioned sourcetype (mysrc_two) it doesnt works.
Please share your thoughts on this.
Data:
{"docker":{"container_id":"852241528698541tzfjztdgtzjsxf"},"kubernetes":{"container_name":"a-kterminal","namespace_name":"kterminal","pod_name":"a-kterminal-555-85chghv","pod_id":"858gh-zgzh-gjh-ghg-896545213","labels":{"application":"a-kterminal","createdBy":"k-rass-template","deployment":"a-kterminal-555","deploymentConfig":"a-kterminal","deploymentconfig":"a-kterminal"},"host":"sdeb-gv-g58","master_url":"https://kubernetes.default.hgfbsjbgsk","namespace_id":"uzsefgvshj-dsgfvjhdv-ztfvsjhybv","namespace_labels":{"app_code":"mycode","network-policy":"true","splunk":"true","splunkindex":"myindex"}},"message":"2019-11-04 14:07:12.321 TRACE 1 --- [nio-8080-exec-4] c.k.k.d.trackinglogger.TrackingLogger : {\"timeStamp\":\"2019-11-04T14:07:12.321Z\",\"country\":\"DE\",\"environment\":\"at\",\"payload\":\"/bye/0\",\"loggingVersion\":\"1.0.0\",\"sessionId\":\"uzsefgvshj-dsgfvjhdv-ztfvsjhybv\",\"terminalId\":\"ABC-12345TST0103\",\"storeId\":\"8950\",\"floor\":\"0\",\"type\":\"System\"}\n","level":"info","hostname":"abc-555-g85","pipeline_metadata":{"collector":{"ipaddr4":"123.12.00.123","ipaddr6":"abc::abc5:abc54:a12:12a","inputname":"fluent-plugin-systemd","name":"fluentd","received_at":"2019-11-04T14:07:13.101993+00:00","version":"0.12.43 1.6.0"}},"@timestamp":"2019-11-04T14:07:12.321816+00:00","viaq_index_name":"project.kterminal.uzsefgvshj-dsgfvjhdv-ztfvsjhybv","viaq_msg_id":"uzsefgvshj-dsgfvjhdv-ztfvsjhybv","forwarded_by":"splunk-connect-1-854ik","source_component":"t01"}
Data from which all backslash (\) need to be removed to view the data in proper json format:
{\"timeStamp\":\"2019-11-04T14:07:12.321Z\",\"country\":\"DE\",\"environment\":\"at\",\"payload\":\"/bye/0\",\"loggingVersion\":\"1.0.0\",\"sessionId\":\"uzsefgvshj-dsgfvjhdv-ztfvsjhybv\",\"terminalId\":\"ABC-12345TST0103\",\"storeId\":\"8950\",\"floor\":\"0\",\"type\":\"System\"}
Configurations :-
props.conf
[mysrc]
TRUNCATE = 0
CHARSET = UTF-8
KV_MODE=JSON
SHOULD_LINEMERGE=false
SEDCMD-remove_header = s/{\"docker.*\,\"message":.*\s+\:\s+//g
SEDCMD-remove_footer = s/\\n"\,\"level"\:.*//g
SEDCMD-replace_backslash = s/\\//g
[mysrc_one]
TRUNCATE = 0
CHARSET = UTF-8
KV_MODE=JSON
SHOULD_LINEMERGE=false
TRANSFORMS-kdt-one = custom_data_one
TRANSFORMS-kdt-two = custom_data_two
[mysrc_two]
TRUNCATE = 0
CHARSET = UTF-8
KV_MODE=JSON
SHOULD_LINEMERGE=false
SEDCMD-replace_backslash = s/\\//g
transforms.conf
[custom_data_one]
REGEX = "splunkindex":"myindex"
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::mysrc_two
[custom_data_two]
REGEX = ({\"docker.*"splunkindex":"myindex"}},\"message":.*\s+\:\s+)(.*)(\\n"\,\"level"\:.*)
DEST_KEY = _raw
FORMAT = $2
Thanks!
↧