OpenTelemetry Troubleshooting Framework - Enable Logging

Enable Logging

Below are instructions for enabling debug logging for some of the available language SDKs. Please keep the following in mind as you troubleshoot:

  • Enable debug logging only when necessary, as debug mode requires more resources.
  • Printing spans to the console should only be used for debugging and not for production

JavaScript

Here is an example of how to enable logging if you are using OpenTelemetry-JavaScript. In your tracing.js file, add the following (add the logger after the SDK):

const api = require('@opentelemetry/api');
...

api.diag.setLogger(
    new api.DiagConsoleLogger(),
    api.DiagLogLevel.WARN
);

The log level can be adjusted per this doc based on the logging level you want. You should now see logs being output to the console that can indicate any issues.

Pro tip: Use the OTEL_LOG_LEVEL environment variable to set DiagLogLevel to easily turn it off and on.

Dotnet

This SDK ships a self-diagnostics feature, which, when enabled, will listen to internal logs generated by all OpenTelemetry components (i.e EventSources whose name starts with “OpenTelemetry-”) and writes them to a log file. See the repo for other options on how to view logs.

The self-diagnostics feature can be enabled/changed/disabled while the process is running, which means you do not have to restart the process. To enable self-diagnostics, go to the current working directory of your process and create a configuration file named OTEL_DIAGNOSTICS.json with the following content:

{
    "LogDirectory": ".",
    "FileSize": 1024,
    "LogLevel": "Error"
}

To disable self-diagnostics, simply remove the above file. Find more information about the parameters, including other log levels, here. Here is a handy tip directly from the repo:

Tip: In most cases, you could just drop the file along your application. On Windows, you can use Process Explorer, double click on the process to pop up Properties dialog and find “Current directory” in “Image” tab. Internally, it looks for the configuration file located in GetCurrentDirectory, and then AppContext.BaseDirectory. You can also find the exact directory by calling these methods from your code.

Go

Use a console exporter and a logging package such as stdr, which will allow you to adjust the logging level and see SDK status messages and spans in the console. Printing to the console should only be used for debugging purposes.

First, import stdouttrace and stdr. Next, temporarily replace the OTLP Exporter and TracerProvider. Finally, adjust the logging level. To see more verbose information, set SetVerbosity to a high number; to see less verbose information, set it to a lower value. See the repo for a more detailed example.

import (
  // ...
  "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
  "github.com/go-logr/stdr"
  // ...
)

func initTracer() {
	var err error
	exp, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
	if err != nil {
		log.Panicf("failed to initialize stdouttrace exporter %v\n", err)
		return
	}

	ssp := sdktrace.NewSimpleSpanProcessor(exp)
	tp = sdktrace.NewTracerProvider(
		sdktrace.WithSpanProcessor(ssp),
	)
	otel.SetTracerProvider(tp)
}

func main() {
	// Set logging level to info to see SDK status messages
	stdr.SetVerbosity(5)

	// initialize trace provider.
	initTracer()
...
}

Once the debugging information has been found and the issue resolved, remove the console exporter and return to using your exporter(s) of choice.

Python

Enable debug logging by importing the logging module and configuring the logging level to DEBUG:

import logging

logging.basicConfig(level=logging.DEBUG)

Errors due to misconfigured exporter endpoints may not be printed to the console; you may just see messages like: Waiting 32s before retrying export of span. If you have verified that the endpoint(s) is/are correct and you are still unable to see your spans in NR1, confirm that your app is being instrumented with the data you expect by using a console exporter, ConsoleSpanExporter, to print your spans to the console.

Import the ConsoleSpanExporter and temporarily replace the OTLPSpanExporter:

from opentelemetry.sdk.trace.export import (SimpleSpanProcessor, ConsoleSpanExporter)

trace.get_tracer_provider().add_span_processor(
    SimpleSpanProcessor(ConsoleSpanExporter())
)

If you are using auto-instrumentation for a Flask app, you can enable debug mode using the example below. However, because this mode enables a reloader, it can break instrumentation from happening. Set the use_reloader option to False to run instrumentation while debug mode is enabled.

if __name__ == "__main__":
    app.run(port=8082, debug=True, use_reloader=False)
1 Like