Agent SDK, API docs or collector communication

I love the direction with the Agent SDK, but it’s limited in scope where it needs to be linked into the application using it. I’d love to be able to instrument applications in Erlang, Go, or other languages where it it’d be more idiomatic to not link to a .so for use.

Is there any effort or thoughts about opening up/documenting the API used by agents for transaction instrumentation?

Alternatively, opening up the protocol used in the agent for the newrelic-client-collector-daemon would be helpful, as non C based languages or interpreters could communicate with it idiomatically, providing the buffering, etc that I assume it provides.

Glad to hear you’re interested in the Agent SDK! @allie is the expert there, so I’ll pass your question along to her.

We don’t have any plans to make our collector protocol public at this time. We want and need to maintain the ability to change this protocol as needed without potentially breaking existing customers.

The Agent SDK is our solution to providing a layer of indirection so you can develop your own instrumentation while being insulated from protocol changes. All you need to do is get a new version of the Agent SDK.

We’ve added your request to our feature request system and are interested in hearing what others think about this suggestion. Also, it would be helpful to hear some of the roadblocks that are preventing you from using the SDK as a solution.

@allie I’m one of the contributors of Kamon (http://kamon.io), a open source tool for monitoring reactive applications were we developed a very basic integration with New Relic, but to build on @gavinr’s reasons for having an public API for reporting metrics/traces to New Relic I would like to expose a few limitations that we are facing regarding moving our integration to the Agent SDK:

  • The Agent SDK does not support transactions with segments in multiple threads: We are working hard to make monitoring applications made on top of Akka/Spray/Play! as easy as possible and we rolled our own instrumentation to follow transaction traces across threads and gather metrics without intervention from the user, it means that we have our own model for traces and it would be nice if we could just accommodate the data in compliance with a public API and post it. We are working on the JVM but I expect the same kind of problems to occur with Go and Erlang.
  • The Agent SDK does not support reporting metrics: As I said, we already have all the transaction metrics and we already have them in a histogram form. If we were able to post the latencies and counts for HttpDispatcher and some other useful metrics using a public API our integration could provide much more value to our current users, which are also New Relic users. We also are about to publish a system and JVM metrics module that could make our users run their apps just with our instrumentation and without the New Relic agent but without a way to report metrics it wont go anywhere and might drive our users to other supported metric backends.

My feeling is that the Agent SDK is really focused on developing agents for languages/frameworks that follow the typical blocking, one thread per transaction approach, which I guess is fine for most of the users out there but that renders it useless for reactive applications, regardless of it being made in the JVM, Go, Erlang, etc.

Also, in my opinion the Agent SDK (or the public API) isn’t something meant to be used by everyone, it is more like a tool for certain group of library maintainers and enthusiasts who have relevant data about their apps and want to leverage New Relic’s awesome infrastructure for displaying, storing, alerting and sharing that data. Being it something targeted to a very specific audience and having deprecation notices, certain quality requirements and a complete documentation about the collectors API or some public API, I think that a great door will be open for innovation from people in the community.

In case there is no chance at all of having access to an API, I would like to know if reporting metrics is something being planned for the Agent SDK.

@ivantopo This is great feedback for our team. You mentioned that you’d like to be able to report metrics, so I have good news for you. The next Agent SDK release is adding this feature. On top of being able to instrument your code using our Transaction API, which creates tracers that automatically report metrics for you, you’ll be able to report custom metrics like you can in our language agents. Since reporting a custom metric doesn’t require any instrumentation, you don’t have to worry about Transaction/Segment thread-safety issues. You can make the call to report a metric anywhere in your code.

Regarding instrumentation, on our list of things to do is resolve the thread-safety issue of creating a transaction trace with multi-threaded segments. Actually, we’ve been preparing to support multiple threading models since the early days of the Agent SDK, which is why we return unique transaction and segment ids when beginning a trace. Adding support for multi-threaded segments in a single transaction won’t cause our API to change.

I think that addresses the two roadblocks you mentioned. Let me know if there’s anything I missed.

Hello @allie, a long time has passed since my initial post and now, I found myself searching through the forums again for news with regards to a proper way of integrating with New Relic.

After reading your reply again, I think that you are talking about a code API, used by calling some NR-provided methods to instrument the application code, but I’m talking about an HTTP API, where people could just POST the metrics and traces information directly, without any New Relic provided code or libraries.

Going back to our use case (and I guess @gavinr as well), we, the Kamon team, do have a lot of instrumentation code for several frameworks and we do have our own code API to gather transaction information and metrics which is then reported with the very same format to all the backend modules we support and then adjusted to the needs of each backend: UDP for StatsD and Datadog, Json for our current New Relic module and a few more integrations will come soon. So, obviously we cannot spread New Relic specific code through our codebase but rather find a way to push data to your servers, and that currently translates to talking directly to your collectors.

The Agent SDK solves the issues you mentioned only when you use it as the base for all instrumentation in a given application and that is not the case for us. Also, I did review the new Async Transaction API introduced in the New Relic Agent but, same as with the Agent SDK, it requires us to spread New Relic specific code through our codebase and we can’t do that.

You mentioned that:

I have been following the collectors API for at least two years now and I can say that nothing that was there when I started working with it has ever changed, although some new things have been added. So, at least two years of no breaking changes wouldn’t make it an stable HTTP API for the community to use?.

Finally, the decision of not making your collector protocol public is still a final decision? has the public HTTP API idea found any sponsors on your side?

Yes, it is our final decision. The Agent SDK is the official way to push metrics to New Relic. Your use case is interesting, and we’ll take it into account in future updates to the Agent SDK.

@ivantopo I still think the Agent SDK works very nicely for your situation, and it’s a plus that you’ve already done so much of the instrumentation work. Now all you need to do is wrap the Agent SDK API inside the code you’re already using to instrument your frameworks. This would make it so you only have to write each Agent SDK function once in a single location, which would make it easy to update or remove in the future. You could also add an option in your configuration to call an Agent SDK function that turns New Relic instrumentation off (each Agent SDK function returns immediately). Once you have this bit of work done, you’ll get New Relic support automatically for any additional code you instrument in the future.

Another way you could get New Relic without having to make New Relic API calls in your instrumented code is by using the Agent SDK’s custom metric API that I mentioned earlier. It allows you to send existing metrics to be collected and sent every minute to New Relic. This sounds closer to what you want, but I recommend the first option; you get more New Relic features, and it requires little extra work.

Hope this helps!

Allie

@allie Sorry for reviving such an old discussion, but it seemed better to add my question here instead of starting a new thread, so it all stays together.

I’m working on creating a custom agent in Elixir, and I’ve come across someone else attempt that I was hoping to build upon. It currently reports transactions and I want to expand that functionality to include errors.

I’ve looked through the code and found it sending transaction reports to collector.newrelic.com. Is this the ‘collector protocol’ that you mentioned earlier that isn’t going to me made public?

If this is the wrong endpoint to use, what’s the alternative? Is it platform-api.newrelic.com, or something else I’ve missed.

@chris.wilson at the moment, passing errors to our service with the Agent SDK just isn’t supported. There are no separate endpoints to use for this.

The other endpoint you call out in particular, platform-api.newrelic.com, is explicitly for Plugin metrics, so I don’t recommend utilizing it for any work in an “agent” setting. For the most part it’s not geared toward the kind of transaction-based data you would normally be collecting throughout a web application’s runtime.

I am going to file this request with our product management team though to keep them aware of this need with the Agent SDK. Should we implement such a feature, we’ll try to send you a notification on it.

In the meantime, one possible workaround for tracking error information would be to send the data to Insights using our insertion API. It won’t influence the Apdex value in the same way as our other APM agents, but it can at least provide a method of viewing the errors within New Relic’s services.