Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Announcement: Changes to the Insights API for Custom Events

insights
events
api
custom_events
rfb

#1

Greetings

The week of November 5, 2018, we will be making some changes to the Insights API. We are upgrading the supporting platform in order to greatly improve the reliability, scalability, and performance of this API. For the most part, the migration of this API to the new platform will be totally transparent to you. However there are a few operational changes we want you to be aware of, and some existing limits that we want to remind you of.

We are also taking this opportunity to update the naming of this API. Instead of referring to this API endpoint as the “Insights API”, we will simply refer to it as the “Event API”. This is the term that we will use in this post, and we will be updating our docs and public content over the next few weeks.

What you should know…

1. Updated documentation.

We have updated the docs to reflect all of the new changes. See : “Send custom events with the Event API

2. Change in when we parse and validate content.

We are delaying content parsing until after we receive the content. We refer to this as being an asynchronous API. The role of the Event API is to accept the message payload from the HTTP response, reliably queue it, and return a successful response code as quickly as possible.

We will still synchronously validate all aspects of the POST request, including license keys, required headers, and payload size. It is just the content parsing and validation that will happen after we have returned a successful HTTP:200 OK response code. If parsing errors occur, we will create an NrIntegrationError event in your account that captures the details. Please review the documentation to see how you can query for these events, create alerts to monitor for them, and search for errors related to specific POST responses.

3. Reminder: There is a 1MB limit on the size of the POST payload.

While the 1MB payload size limit is documented, we have not always strictly enforced it. With this upgrade, this limit will be strictly enforced. HTTP POSTs with a payload larger than 1MB will be rejected with an HTTP 413 (Content too large) response code. We strongly encourage you to compress your content before POSTing it to the API.

4. Default API rate limits

In order to ensure the reliability and availability of our multi-tenant platform, it is critical that this API enforces rate limits across all accounts. We must ensure that extreme, unexpected traffic spikes do not negatively affect the platform and degrade the service for all of our customers. The new API platform will automatically enforce a maximum default rate of 100,000 requests per minute.

If your API usage exceeds 100k POSTs in a 1 minute window, we will reject subsequent API requests with a 429 response code for the remainder of the 1 minute window. At the end of the 1 minute window, we will reset the counters and allow traffic to resume until the threshold is triggered again.

These rate limits are intended to be an upper threshold that you should not hit under normal usage scenarios… If you experience an unexpected number of 429 responses, please contact Support to discuss your needs.


#2

One thing which is not clear, unless you read the documentation, the endpoints that are currently being used are not renamed?


#3

As far as I’m aware, that is correct @stefan_garnham While we’ll now be referring to the Insights API as the Event API, it is still broken down into Query or Insert, so those Endpoints can remain unchanged.

@bgoleno will be able to clarify is the end goal is for the endpoints to change as well, however for now I believe they are remaining as they are.


#4

@stefan_garnham Correct. This is migrating the back-end of the Version 1 of this API, so the hostname and URL can not change. When we release a version 2 of the API, we will have the opportunity to use a new hostname and URL, but we do not have a forecast for that yet. ( cc @RyanVeitch )


#5

When will these ^^ events start appearing in our New Relic Insights accounts? Should they be appearing already?


#6

Hi, We just finished the migration a few moments ago. Everything should be working as documented.


#7

Hi,

I’m sending the Insight custom events through the .NET library (installed .NET agent). I don’t make HTTP POST API call. Am I restricted by the 100K requests/minute mentioned above?

Thanks


#8

@huy great question and no if you’re using the agent API functions, you’re not hitting the insights-collector.newrelic.com so to the best of the knowledge I have right now. You’re not affected by these limits due to not hitting that API endpoint.


#9

Bravo on this new release!

I was having trouble with intermittent failures from my integration platform (Zapier) sending data from webhooks to New Relic Insights. The webhooks were configured on their source systems to send “everything”, and Zapier too was configured to send all of the webhook payload to New Relic Insights.

The failures noted in Zapier’s log were mysterious - why would a given webhook -> Insights function most of the time, and fail some of the time?!

As soon as the new NrIntegrationError event started populating, I was able to quickly use the data in those events to correlate with specific events occurring in my source data systems - and discover that the failing webhooks were failing because their payloads exceeded Insights’ Event API limits.

The fix was straightforward - my colleagues and I didn’t really need all the fields of data we were sending over, I’d been “sending everything” because it helped us explore the data in Insights to decide what we wanted and what wasn’t useful. So, I reconfigured the Zapier integration to filter out the fields of data we didn’t need, which resulted in all the POSTs to the Events API being within limits, which resulted in no more NrIntegrationError events.

Thanks New Relic, I continue to :heart: you, from afar. :blush: