OpenTelemetry Collector - What max size value to use for the batch processor?

Given that the New Relic exporter is being deprecated, we attempted to update our collector config to remove this and use the standard OTLP exporter instead, following your example collector config. At the same time we also added the ‘logs’ data type in, previously it was just ‘trace’.

Unfortunately, when used in production the collector now gives an error unexpected HTTP status code received from server: 413 (Request Entity Too Large). It seems this can be solved by supplying some max size parameter to the batch processor in the collector config but I’m unsure what this would need to be set to. Or perhaps we’re being rate-limited because of the extra log data? Can anyone advise?

Thanks

Hello @domholmes,

Welcome back to our Community forum! Thank you for your post. You are correct in thinking that you may need to supply some limits to the batch processor in the Collector config; we also recommend enabling gzip compression for your OTLP exporter(s). A combination of both will help avoid rate limiting.

To explain the issue you are experiencing and its resolution – the maximum allowed payload size is 1MB (10^6 bytes). By default, the OpenTelemetry SDKs and Collector send one (1) data point per request. Using these defaults, it is likely your account will be rate limited. All OpenTelemetry SDKs and Collectors provide a BatchProcessor, which batches data points in memory. This batching allows requests to be sent with more than one (1) data point. Additionally, to maximize the amount of data you can send per request, we recommend enabling compression in all OTLP exporters you are using. It is best to set compression explicitly, as gzip compression.

For the batch processor config, we have been recommending setting send_batch_size to 1000, as this is roughly equivalent to what our APM agents do. It’s not guaranteed that a batch size of 1000 will be less than our 1MB ingest limit, but it makes it far more likely. For example, a single span can vary greatly in size depending on the number of attributes and attribute length - so 1000 very large spans may exceed the 1MB limit. We also wanted to point you to some internal metrics emitted by the Collector that are good to monitor.

Speaking of attribute length, I looked at your account and noted that you have some NrIntegrationError events indicating that data points have been dropped due to the length of an attribute value being over the limit. In addition to the Collector configuration settings above, I would also recommend taking a look at this troubleshooting section to resolve those errors.

Let us know if you have any questions or run into any more issues, we are happy to help!

Best,
Reese Lee

1 Like

Hi there! Thanks for the detailed response. So we’ve made the suggested changes (namely compression and batch size) and that worked. It would be good to add these into your collector config examples I think. I’ll take a look at NrIntegrationError. Thanks again.

Great feedback @domholmes - Very glad to hear this is working for you now!

Hello @domholmes,

Thank you for letting us know this worked for you! I agree that it would be great to have an example of this in our examples repo, and hope to have one soon. In the meantime, I’ve updated our documentation to make it clearer how to avoid getting rate limited.

Have a great week!

Best,
Reese Lee

2 Likes