We are pulling data like Red Hat logs, Apigee, Ansible etc. from AWS through fluentd plugin which is forwarding data to our Heavy Forwarder in AWS, and then from that, the HF to another HF in a DMZ to another HF outside of DMZ.
The data is passing through and getting indexed, so the firewall rules and ports are established properly. However, when trying to transform the data so that we can split it into numerous sourcetypes, it will not work. It still applies the original sourcetype applied from fluentd plugin.
In the fluentd plugin, we are defining index name, sourcetype, and the default format is JSON. We are trying to override this index and sourcetype at the destination for differentiating types of data with different sourcetypes by defining inputs.conf, props.conf, transforms.conf. It is not applying the values what we define here at the destination. It is only taking the values that the source is defining in the fluentd plugin config file.
So the question is, can we add a props and transforms config in fluentd plugin in AWS to differentiate the logs with sourcetypes? Can anyone suggest a possible solution for this kind of problem?
FLuentd plugin is ----k24d/fluent-plugin-splunkapi
We are using Splunk 6.2.2 in all Indexers, Forwarders etc
Here are the configs that we defined at the destination.
Please help us.
inputs.conf
[splunktcp://1600]
connection_host = ip
sourcetype = journald
index = aws_fluentd_index
props.conf
[source::poc.aws.system.journald]
KV_MODE = json
TIME_PREFIX=^
TIME_FORMAT=%Y-%m-%d %T %z
SHOULD_LINEMERGE=false
MAX_TIMESTAMP_LOOKAHEAD=30
NO_BINARY_CHECK = 1
pulldown_type = 1
[source::poc.aws.system.journald]
TRANSFORMS-override=override_ST_journald,override_IDX_journald
transforms.conf
[override_ST_journald]
SOURCE_KEY=_raw
REGEX=.*
FORMAT = sourcetype::journald
DEST_KEY = MetaData:Sourcetype
[override_IDX_journald]
SOURCE_KEY=_raw
REGEX=.*
FORMAT = aws_fluentd_index
DEST_KEY = _MetaData:Index
↧