If you were not directed to this post from the ‘Get Certified, Get Socks’ post, I recommend you take a look at that for some more context before continuing on this thread.
The Fundamentals Certification is difficult, there are a total of over 180 possible questions covering every area of New Relic, you get a random selection of 20 questions for each test you take. It’s impossible to know what will come up in your test, but it’s good to be as prepared as possible. To that end, we are posting weekly Tips of the week, that are geared towards a specific sub-section of the Fundamentals Certification.
The Tip of the Week this week will be aimed at Deployment Markers .
If your production workflow is fast, with constant updates and constant deployments, there is a chance the some day one of those deployments won’t go to plan. It’s ok, it happens to everyone in the software world. It’s hard to put enterprise scale software under an enterprise scale test in a staging environment, so it’s inevitable that some day, it won’t go completely to plan. How do you monitor that?
Deployment markers are perfect in this scenario, when used effectively. The attributes provided in a deployment marker can show you who made changes, what changes were made, and when they were deployed.
Deployment markers are important even outside of the realm of ‘things can go wrong’ mindset. Ideally, every deployment made to your application will come with a positive impact. From an APM perspective, you would hope that the deployment has a positive performance impact. Deployment markers can help you confirm this impact.
What do they look like?
Below is an example from the APM overview page, You’ll see that our response time has a drastic increase immediately following an app deployment. Conveniently, the APM Overview charts have a vertical bar superimposed over them to highlight deployment events. In our example we see that the impact of the deployment went so far as to trigger an alert violation.
As above, we see that our initial deployment was bad - we failed to optimise DB query performance, and it near quadrupled our response time. Obviously this needs to be rolled back.
Thankfully this application has a response time alert condition targeting it, and so we are notified quickly of this poor performance.
We can drill down further into the specific deployment by either clicking on the vertical bar in the chart, or, going to the deployments tab in the left side of the UI.
From there we are looking at the 90 minute period before and after the deployment, watching the performance change over a number of metrics;
So what we see here is, who made the change, what notes did they leave with the change, and what was impacted most by that change.
We can see that our Apdex dropped in half, CPU utilisation increased, the percentage of time taken in databases increased hugely too.
Clearly our DB optimisation has not gone to plan.
How do I use them?
Now that we can see how bad our deployment was, we can make the call to either put together a quick fix to patch the issue, or do a full rollback to re-evaluate the way we are doing our query optimisations.
Really though - no matter how useful deployment markers are, the key to getting them right is:
- Watch them!
- When you make a deployment be sure you are watching the NR UI to see what happens.
- Effective Alerting!
- Make sure that you have the right kind of alert conditions targeting your applications before making your deployments, so that if you are not watching the NR UI, you are at least notified when the deployment doesn’t go to plan.
Once we have rolled back our deployment, we can get into a deep dive and figure out what exactly went wrong. we do this by narrowing in on the time period relevant for the poor performance, and looking into transaction traces.
There I see that MySQL is taking up nearly 40% of my transaction time, and database calls are happening on average ~1k times per transaction.
With the right configuration (DB queries set to
OFF) - we can click through to the Database Queries tab and see what queries exactly are being called so much.
How do I post Deployment Markers to New Relic?
Most Agents have methods to push deployment info - see these docs for specifics in your environment: https://docs.newrelic.com/docs/apm/new-relic-apm/maintenance/record-monitor-deployments#agent
Alternatively, depending on your CI tooling, you can add the Deployments API call to a post-deploy script using the Deployments API.
Another option is to manually use the Deployments API, as documented here: https://docs.newrelic.com/docs/apm/new-relic-apm/maintenance/record-monitor-deployments#api
It’s time to get certified! And Earn some socks by completing the certification, and sharing your results, before Feb 14th!