To follow up on this in case any future engineers read this and see a similar issue, I’ve found the root cause and will hope to communicate what it was. It was a very specific issue that likely isn’t your issue, but the way to debug might also help you debug your issue.
Just like other google results and posts, this was in fact a middleware issue that is outside normal NewRelic instrumentation and thus shows up as time spent in ActionDispatch::Routing::RouteSet#call. Since AppName::Application.routes is the last middleware to be loaded, any time spent outside of normal instrumentation will show up as ActionDispatch::Routing::RouteSet#call.
The above being said, we also utilize lograge to normalize logging in to a more digestable single line format. A feature of lograge is the ability to configure a custom block to augment log lines with extra contextual data like user/session data. As part of this custom data block we wanted to encrypt certain things before they made it in to the logs, and they way we did that was to use ActiveSupport::KeyGenerator and ActiveSupport::MessageEncryptor. Our mistake was to use the request id as the salt. Creating the key and message encryptor object is CPU intensive, and since it was using request id as the salt, it wasn’t memoized and regenerated each time it was called, on every request that needed encryption.
The way I was able to verify this was to use NewRelic’s great custom method instrumentation to explicitly instrument the custom block code which once deployed, was very telling and showed a majority of time spent in a single method that generated the ActiveSupport::MessageEncryptor object.
I relied on Benchmark.measure as well to find out that instantiating 10 ActiveSupport::MessageEncryptor instances took about 10 seconds.
If anyone sees similar timings in ActionDispatch::Routing::RouteSet#call I encourage you to use a combination of newrelic custom method instrumentation and take a hard look at all your middleware as some small piece of code that might go unnoticed could be having a significant impact on your application.