Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

How does 'App instance busy' handle thread-based application servers (e.g. Puma)


#1

We are currently using Unicorn to serve a Rails 5.1/Ruby 2.4.4 application. We monitor NewRelic’s capacity data and use this to trigger autoscaling (scale-out and scale-in) of our containers.

I am in the process of testing a migration to Puma. I have changed our Staging environment from a 6 process Unicorn server to a Puma cluster of 4 workers, each with 4 threads. Load testing has shown this new setup can handle more than double the load.

I am now testing if the NewRelic capacity data is still accurate enough to be used for our autoscaling.

I ran a test with 6 concurrent clients randomly requesting our site’s most common pages (typically requests times for the pages are between 0.5 to 2 seconds).

siege --delay=0.1 --internet --file=staging-urls.txt --verbose --reps=200 --concurrent=6 --no-parser

The test took about 5 minutes to complete and requests times at the beginning were taking the same time as the end. So this suggests we had no backlog and the 4 workers (4 thread) Puma cluster could handle this traffic load.

Although New Relic correctly reported we were running 4 app instances, it deemed our ‘App instance busy’ rate at 150% and suggested we should be running 6 instances to handle the load.

This seems incorrect and would cause our autoscaling to over provision. How exactly is ‘App instance busy’ calculated? Is it aware of Puma threads?

Puma can report how many threads are idle for each worker (see pool_capacity in puma’s stats output below). If possible that too might be a nice number to report in your capacity screen.

 {
  "workers": 2,
  "phase": 0,
  "booted_workers": 2,
  "old_workers": 0,
  "worker_status": [
    {
      "pid": 1301,
      "index": 0,
      "phase": 0,
      "booted": true,
      "last_checkin": "2018-10-19T05:10:58Z",
      "last_status": {
        "backlog": 0,
        "running": 4,
        "pool_capacity": 4,
        "max_threads": 4
      }
    },
    {
      "pid": 1315,
      "index": 1,
      "phase": 0,
      "booted": true,
      "last_checkin": "2018-10-19T05:10:58Z",
      "last_status": {
        "backlog": 0,
        "running": 4,
        "pool_capacity": 3,
        "max_threads": 4
      }
    }
  ]
}

#2

Hi @guy.c, thanks for writing in! :wave:

You definitely raise a good point here. My first question is what New Relic Ruby Agent version are you currently running?

Metrics around capacity analysis did not previously account for multi-threaded dispatchers, such as Puma, and consequently could result in capacities of over 100% being recorded.

The good news is that our most recent agent release has resolved this issue!

If this is the case, upgrading your agent should give you the solution you need to see accurate analysis as you migrate to Puma. Please let me know if this resolves the issue for you.

Otherwise, please let me know if you are experiencing this issue on our current Ruby Agent version and we can troubleshoot together from there.

Cheers!