A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Use Cases
Honeycomb's create events API supports a samplerate field for each event which enables Honeycomb to automatically inflate data when queried by the sample rate to give a better estimate of the unsampled data.
I'm using a Sample transform to sample my events before sending them to Honeycomb, and I would like to have the data automatically inflated.
Attempted Solutions
Currently there is no way to supply this field when using the Honeycomb sink. The data would need to be included here: https://github.com/vectordotdev/vector/blob/master/src/sinks/honeycomb.rs#L143
Since the Sample transform adds a sample_rate property to the events it samples, my current workaround is to do the inflation manually in Honeycomb by multiplying values by the sample_rate column.
Proposal
Add a samplerate_field configuration to the Honeycomb sink which represents the name of a field on the events which represents its sampled rate. The Honeycomb sink will check each event for a valid (number) field with the name supplied in samplerate_field. If it exists, it will remove the field from the event and it will include that value as the samplerate for the event in the Honeycomb object. If it does not exist, it will leave the samplerate undefined.
Since the Sample transform adds a sample_rate property to the events it samples, it will be easy to chain this configuration with a Sample transform by setting it like so: samplerate_field: sample_rate.
Indeed, it may even be a good idea to set the default value of samplerate_field to be sample_rate so that no configuration is required when chaining the Sample and Honeycomb components together. However, this would be a potentially breaking change for anyone that's including sample_rate in their events.
References
No response
Version
No response
A note for the community
Use Cases
Honeycomb's create events API supports a
sampleratefield for each event which enables Honeycomb to automatically inflate data when queried by the sample rate to give a better estimate of the unsampled data.I'm using a Sample transform to sample my events before sending them to Honeycomb, and I would like to have the data automatically inflated.
Attempted Solutions
Currently there is no way to supply this field when using the Honeycomb sink. The data would need to be included here: https://github.com/vectordotdev/vector/blob/master/src/sinks/honeycomb.rs#L143
Since the Sample transform adds a
sample_rateproperty to the events it samples, my current workaround is to do the inflation manually in Honeycomb by multiplying values by thesample_ratecolumn.Proposal
Add a
samplerate_fieldconfiguration to the Honeycomb sink which represents the name of a field on the events which represents its sampled rate. The Honeycomb sink will check each event for a valid (number) field with the name supplied insamplerate_field. If it exists, it will remove the field from the event and it will include that value as thesampleratefor the event in the Honeycomb object. If it does not exist, it will leave thesamplerateundefined.Since the Sample transform adds a
sample_rateproperty to the events it samples, it will be easy to chain this configuration with a Sample transform by setting it like so:samplerate_field: sample_rate.Indeed, it may even be a good idea to set the default value of
samplerate_fieldto besample_rateso that no configuration is required when chaining the Sample and Honeycomb components together. However, this would be a potentially breaking change for anyone that's includingsample_ratein their events.References
No response
Version
No response