I have some TSV files that I am forwarding with a Universal Forwarder.
I have props.conf configured on the UF with the following for the sourcetype:
FIELD_DELIMITER = \t
HEADER_FIELD_LINE_NUMBER = 1
That has worked great. But now I have a need to drop some events so they do not get indexed.
On the Indexer I have configured the following for the sourcetype in props.conf:
[]
TRANSFORMS-null = drop_batchrequests
...and in transforms.conf:
[drop_batchrequests]
REGEX = batchRequest
DEST_KEY = queue
FORMAT = nullQueue
At first it was not working, I was still getting events that contain batchRequest. So I temporarily removed the structured data configuration on the Universal Forwarder (shown above) and the transform worked as desired - batchRequest events were no longer indexed.... But now the tsv format and field recognition was not there...
So I tried to configure everything in one place. On the Indexer I specified the structured data config in props.conf using FIELD_DELIMITER and FIELD_NAMES (since I can't use HEADER_FIELD_LINE_NUMBER on the Inedxer.) The result of that was the batchRequests events were not indexed, but the fields (from the header row) still were not extracted.
Am I doing something wrong? Or is there some reason why these configurations (TSV/structured data field recognition and dropping certain events to the nullQueue) on the same sourcetype will not work together? I can get each to work independently - but not together.
↧