Ruby Troubleshooting Framework Configuration

Basic Configuration Troubleshooting

  1. Check if data is currently reporting and work through all steps in the Install Troubleshooting Framework if it is not.

  2. Confirm the environment is correct, i.e. test or development

  3. Run NrDiag to collect data then validate:

    1. Confirm Ruby version meets requirements.
    2. Verify OS meets requirements.
    3. Verify existence of newrelic_rpm in Gemfile/Gemfile.lock
    4. Verify newrelic.yml file is present as described in our Ruby Agent Install Documentation.
    5. Ensure configuration is valid with a license key and app name.
    6. Check for the existence of ruby agent logs in the default path.
    7. Check that agent can connect to the New Relic collector.
    8. Verify agent isn’t an EOL version.
    9. Check for a “reporting to” line in the logs.
    10. Check that High Security Mode settings for the application and account match.
  4. Check if any errors are raised in the console at application start, errors indicating a configuration issue would be:

    1. Error in loading newrelic.yml
    2. license_key was not set in newrelic.yml
  5. Next, collect debug level logs and check for error messages and configuration settings reported at initialization

    1. If no log file, check:

      1. log_file_name
      2. log_file_path
  6. Confirm that the desired settings are correct in newrelic.yml the configuration file

    1. Make sure the YAML file is valid. You can do this using an online YAML validator (remove sensitive data first). If incorrectly formatted, the agent should throw an error Unable to parse configuration file error on startup

    2. When in doubt, download a clean configuration:

      1. You can download a clean configuration file from your Account Settings page
      2. You can also generate a new file manually using the command newrelic install --license_key=“YOUR_KEY” “My application”
  7. Check configuration file location. For Ruby, the primary (default) method to configure the Ruby agent is via the configuration file (newrelic.yml) in the config subdirectory. When in doubt, check agent logs as any level of agent logs will show if the configuration file is not accessible to the agent

  8. Check alternate configuration methods. For Ruby, the precedence is: Environment variables > server-side config > Config file > Default

    1. Check Server Side configuration in the New Relic One UI, Go to rpm.newrelic.com/apm OR one.newrelic.com > APM > (select an app) > Settings > Application > Server-side agent configuration.

    2. Ruby has many places in which environment variables can be set. Have the customer run the command env | grep NEW_RELIC_ or printenv, or ruby -e 'p ENV['NEW_RELIC_<specific var name>']' from the command line within their application directory or in Ruby console p ENV['NEW_RELIC_<specific var name>'] or p ENV.keys

    3. For non-Rails apps, the Ruby agent looks for the following environment variables, in this order, to determine the application environment so if there is a configuration issue, be sure to check all:

      1. NEW_RELIC_ENV
      2. RUBY_ENV
      3. RAILS_ENV
      4. APP_ENV
      5. RACK_ENV
    4. Confirm variables are correctly set, name should be the configuration variable name with all . replaced with _

    5. If using a cloud provider, container or container manager, check that specific technology for alternate methods to set environment variables

  9. For Ruby agent version v5.5.0 or less, there is the option to run in developer_mode which can have alternate values for environment variables or settings in the configuration file. Confirm which mode they are currently working in

  10. If you cannot find the source of a configuration setting, check default settings to see if the Ruby agent is reverting to this value. This would indicate that your configuration is not being read correctly

  11. Looking for a configuration option but not sure the name? Run the command rake newrelic:config:docs for the most common options

Controlling Agent Start Up and Shut Down

  1. Why isn’t the agent starting? The Ruby Agent uses autostart logic attempt to avoid starting at “inappropriate” times

    1. Inactive sessions: for example in an IRB session, Rails console, or for Heroku, logs typically go to STDOUT so agent logs can spam the console during interactive sessions.
    2. The agent detects it is in one of Rails’ built-in rake tasks; for example, assets:precompile or db:migrate.
  2. To override the agent’s auto-start logic, the easiest mechanism is to set a NEW_RELIC_AGENT_ENABLED=true environment variable

    1. for example: NEW_RELIC_AGENT_ENABLED=true rake assets:precompile
  3. For very short lived processes, you will need to configure the agent to connects to the collector as soon as possible, and all data is sent to New Relic before shutting down

    1. Use sync_startup to force a synchronous connection to the New Relic collector during application startup to ensure the New Relic agent has time to report.
    2. Use send_data_on_exit to enable the exit handler that sends data to the New Relic collector before shutting down.
  4. Customize the auto-start configuration variables

    1. Constants

      1. Ruby v6.8.0 and higher:

      2. autostart.denylisted_constants: Specify a list of constants that should prevent the agent from starting automatically. Separate individual constants with a comma

      3. For example, Rails::Console,UninstrumentedBackgroundJob.

    2. Ruby v6.7.0 and lower use the same value format as above with the variable autostart.blacklisted_constants instead

    3. Rake tasks: the autostart logic does not disable the Ruby agent in all rake tasks, because tasks like resque:work generally are monitored. For Rake tasks you do not want the agent to monitor, use:

    4. Ruby v6.8.0 and higher:

      1. autostart.denylisted_rake_tasks: Defines a comma-delimited list of Rake tasks that the agent should not instrument.

      2. For example, assets:precompile,db:migrate.

    5. Ruby v6.7.0 and lower use the same value format as above with the variable autostart.blacklisted_rake_tasks instead

    6. Executables:

      1. Ruby v6.8.0 and higher:

      2. autostart.denylisted_executables: Defines a comma-delimited list of executables that the agent should not instrument.

        1. For example, rake,my_ruby_script.rb.
    7. Ruby v6.7.0 and lower use the same value format as above with the variable autostart.blacklisted_executables instead

Missing Transactions

  1. First follow the Install Troubleshooting guide to confirm your application meets the requirements and is fully instrumented

  2. You can define transactions you want the agent to ignore, by specifying a list of patterns matching the URI you want to ignore using rules.ignore_url_regexes. Regex mistakes are common, so if a ignore rule is in place and transactions not intended to be captured by that rule are also missing:

    1. Ensure that the regex is a valid pattern and rules are in a comma separated array. Online regex checkers are available which you can use to test if the missing transaction is mistakenly being matched by another ignore rule. Rules are given precedence based on order.

Browser Configuration

  1. Set browser_monitoring.auto_instrument to true to auto-injection the JavaScript header for page load timing. Automatic instrumentation works with Rack, and requires Rails 2.3 or higher.

    1. If the Browser agent is not reporting a controller or action, confirm you have not added the call newrelic_ignore_enduser anywhere in the application
    2. In a Rails application, newrelic_ignore_enduser supports the same :only and :except options as newrelic_ignore. In a Sinatra application, it will accept the same Sinatra-style route for targeting specific transactions.
  2. See Browser Monitoring document for more information

Attributes

  1. See our document on Ruby agent attributes for a full guide. Note these attribute settings apply to version v3.12.0.288 of the Ruby agent which was released in 2015 therefore any versions earlier are outside of support according to the New Relic agent and plugin end-of-life policy

  2. Default Configuration:

    1. Attributes: Enabled

      1. attributes.enabled
      2. attributes.exclude to exclude attributes
    2. Transaction traces: Enabled

      1. transaction_tracer.attributes.enabled
    3. Error collector (traced errors): Enabled

      1. error_collector.attributes.enabled
    4. Transaction events: Enabled

      1. transaction_events.attributes.enabled
      2. Caution: When Distributed Tracing and/or Infinite Tracing are enabled, information from Transaction Events is applied to the root Span Event of the transaction. Consider applying any attribute settings for Transaction Events to Span Events and/or apply them as Global Attribute settings.
    5. Span events: Enabled

      1. span_events.attributes.enabled
    6. Page views (browser monitoring): Disabled

      1. browser_monitoring.attributes.enabled
    7. Custom attributes: Enabled

      1. Custom_attributes.enabled
  3. Full list of attributes that can be automatically captured by the Ruby agent

  4. Guide to Enabling and Disabling attributes

    1. Root level takes precedence for enabled so the attributes.enabled field trumps all other settings. When false, no attributes will be reported to New Relic.

    2. Destination enabled takes precedence over include and exclude

      1. Attribute is included if the destination is enabled. If a destination is enabled, all user attributes are sent to that destination by default.
      2. Include or exclude affects the specific destination. If the attribute include or exclude is specified on a destination, then it only impacts that destination.
      3. Note: All user attributes default to true. However, by default, request parameters are disabled for all destinations.
    3. Exclude always supersedes include If the same key is listed in the include and exclude lists, then attributes with the specified key will be excluded.

    4. Formatting:

      1. Keys are case sensitive.
      2. Use * for wildcards. You can use an asterisk * at the end of a key as a wildcard. This will match a set of attributes with the same prefix.
      3. Most specific setting for a key takes priority. Order is irrelevant here, if multiple include or exclude attributes affect the same key, the most specific setting will have priority.
  5. Check if using deprecated attribute properties, these have been replaced with equivalent attributes enabled. See this document for the Deprecated Properties section in this document

    1. capture_params
    2. Resque.capture_params
    3. Sidekiq.capture_params
    4. transaction_tracer.capture_attributes
    5. error_collector.capture_attributes
    6. browser_monitoring.capture_attributes
    7. analytics_events.capture_attributes

Missing Thread Profiler Data

  1. Confirm thread_profiler.enabled is set to true. After, they will need to start the profiler in the UI. See our full guide here

Cross Application and Distributed Tracing

  1. Cross Application Tracing missing data

    1. Confirm cross_application_tracer.enabled is set to true and distributed_tracing.enabled is set to false
  2. Missing Distributed Tracing data

    1. Confirm distributed_tracing.enabled is set to true
    2. Confirm the account is not using the FedRAMP-compliant endpoints. Distributed Tracing is not currently supported for FedRAMP accounts
    3. If blank is exclude_newrelic_header is set to true, confirm all APM and Browser agents in the trace on are an agent that supports W3C Trace Context Headers. For Ruby, this is Ruby Agent v6.9.0.363
  3. Missing detail in External Services

    1. If distributed_tracing.enabled is set to true, this is expected behavior. Read more about this in the transition guide

Transaction Trace Configuration

  1. Transaction Trace reporting is enabled by default

  2. If Transaction Traces or explain plans are not reporting:

    1. Confirm transaction_tracer.enabled is set to true

    2. For explain plans, confirm transaction_tracer.explain_enabled is set to true

    3. Confirm that the transaction_tracer.explain_threshold and transaction_tracer.stack_trace_threshold is set low enough to capture queries. Use the below queries to help determine what to set this value at. Be aware these queries return milliseconds and transaction_tracer.explain_threshold is set in seconds

      1. SELECT average(duration) FROM Transaction WHERE (appId=) SINCE 1 month ago TIMESERIES MAX
      2. SELECT percentile(duration, 50, 75, 95) FROM Transaction WHERE (appId=) SINCE 1 month ago TIMESERIES MAX
  3. If traces are incomplete for long running transactions, it may be necessary to raise the transaction_tracer.limit_segments from the default 4000

Error Configuration

  1. Error reporting is enabled by default

  2. If errors or stack traces are not reporting:

    1. Confirm error_collector.enabled is set to true
    2. Confirm the error is not set as ignored with in the configuration file or environment variable using error_collector.ignore_errors or in Server SIde Configuration
    3. If TransactionError event data are not reporting, confirm error_collector.capture_events is set to true
  3. If you are missing TransactionError event data, you can raise the error_collector.max_event_samples_stored from the default 100 per minute long harvest cycle although this may increase agent footprint.

  4. If back traces for errors are truncated, it may be necessary to raise the error_collector.max_backtrace_frames from the default 50 frames

Database Reporting Configuration

  1. Datastore instance and name reporting is enabled by default

  2. If Slow queries or explain plans are not reporting:

    1. Confirm slow_sql.enabled is set to true

    2. For explain plans, confirm slow_sql.explain_enabled is set to true

    3. Confirm that the slow_sql.explain_threshold is set low enough to capture queries. Use the below queries to help determine what to set this value at. Be aware these queries return milliseconds and slow_sql.explain_threshold is set in seconds

      1. SELECT average(apm.service.datastore.operation.duration) FROM Metric WHERE (entity.guid = ‘’) FACET datastoreType, operation SINCE 1 month ago TIMESERIES MAX
      2. SELECT max(apm.service.datastore.operation.duration) FROM Metric WHERE (entity.guid = ‘’) FACET datastoreType, operation SINCE 1 month ago TIMESERIES MAX

Developer Mode

  1. Developer mode is removed and no longer supported as of Ruby agent version 4.1.0 which was released April 2017 and therefore any versions earlier are outside of support according to the New Relic agent and plugin end-of-life policy
  2. See our full documentation here

Gem Specific Configuration

  1. If all configuration looks correct, but there is still missing/inaccurate data, check for Incompatible gems which may be impacting instrumentation beyond the Incompatible Gem itself
  2. Redis Instrumentation Guide
  3. Mongo Instrumentation Guide
  4. Rack Middleware Guide
  5. Sequel Instrumentation Guide
  6. Sinatra Instrumentation Guide