[Python] Enabling Python Agent causes crash in inspect.getfullargspec for builtin

We have a Flask (1.1.1) server with Flask-SocketIO (4.2.1) running in a Python 3.6.11 docker container. It is started directly, via a main.py:
python3 -u main.py. It works great. When I change the command line to NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program python3 -u main.py, with the settings file having monitor_mode = false, it still works great. When I enable monitor_mode, it crashes at startup while initializing the injection kernel, which internally is using the inspect.getfullargspec function. The mysterious part is that it is failing for a built-in method.

The full error is below:

Traceback (most recent call last):
galileo-server_1         |   File "/usr/local/lib/python3.6/inspect.py", line 1126, in getfullargspec
galileo-server_1         |     sigcls=Signature)
galileo-server_1         |   File "/usr/local/lib/python3.6/inspect.py", line 2273, in _signature_from_callable
galileo-server_1         |     skip_bound_arg=skip_bound_arg)
galileo-server_1         |   File "/usr/local/lib/python3.6/inspect.py", line 2097, in _signature_from_builtin
galileo-server_1         |     raise ValueError("no signature found for builtin {!r}".format(func))
galileo-server_1         | ValueError: no signature found for builtin <FunctionWrapper at 0x7f0e91684388 for functools.partial at 0x7f0e916e4408>
galileo-server_1         |
galileo-server_1         | The above exception was the direct cause of the following exception: 
galileo-server_1         | 
galileo-server_1         | Traceback (most recent call last):
galileo-server_1         |   File "main.py", line 7, in <module>
galileo-server_1         |     app = create_app()
galileo-server_1         |   File "/app/app/__init__.py", line 208, in create_app
galileo-server_1         |     _configure_dependency_injection(app, None)
galileo-server_1         |   File "/app/app/__init__.py", line 104, in _configure_dependency_injection
galileo-server_1         |     modules=modules,
galileo-server_1         |   File "/usr/local/lib/python3.6/site-packages/flask_injector.py", line 317, in __init__
galileo-server_1         |     process_dict(container, injector)
galileo-server_1         |   File "/usr/local/lib/python3.6/site-packages/flask_injector.py", line 344, in process_dict
galileo-server_1         |     d[key] = wrap_fun(value, injector)
galileo-server_1         |   File "/usr/local/lib/python3.6/site-packages/flask_injector.py", line 78, in wrap_fun
galileo-server_1         |     return wrap_fun(inject(fun), injector)
galileo-server_1         |   File "/usr/local/lib/python3.6/site-packages/injector/__init__.py", line 1354, in inject
galileo-server_1         |     bindings = _infer_injected_bindings(function, only_explicit_bindings=False)
galileo-server_1         |   File "/usr/local/lib/python3.6/site-packages/injector/__init__.py", line 1159, in _infer_injected_bindings
galileo-server_1         |     spec = inspect.getfullargspec(callable)
galileo-server_1         |   File "/usr/local/lib/python3.6/inspect.py", line 1132, in getfullargspec
galileo-server_1         |     raise TypeError('unsupported callable') from ex
galileo-server_1         | TypeError: unsupported callable

The only difference is enabling monitor_mode, which of course is the point. Is there something that the newrelic agent is doing to the execution environment that I should be aware of?

Hiya @charlie15!

I’m unable to replicate this using the following (mind ya, it’s a super simple script):

I’ll follow up on this with the engineers for their insight. But please let me know your thoughts on the above.


I will have a look at what you came up with. After looking at this for a few days, I suspect that the problem might be in multiple monkey patching of the core libraries. We are using Eventlet, which patches many core libraries to enable cooperative yielding for multitasking, and I suspect it may have a part to play. I’ll mess with your script and try and post a recreation.


1 Like

Sounds good! Also, what version of the agent is installed?

We’re using the latest version, looks like it’s not locked in our requirements.txt.

I cloned your repo and have been working with it for a while, and all of my expected recreations have nothing to do with NewRelic

`import inspect
import functools


Will crash every time, no matter how much newrelic is involved. What I can’t figure out is why enabling newrelic in my code changes the code such that it ever makes this 100% deadly call. I will continue working with the recreate, if only for my own purposes, but at this point in time I’m pretty sure it’s not you, it’s me.

Thanks for taking the time to look at this!

1 Like

Hello Charlie,

Just checking in on how you are doing, and do let us know how it goes!


I suspect that the problem might be in multiple monkey patching of the core libraries.

Still having the problem, never did get to the bottom of it, and even more frustrating, I have never been able to recreate it from scratch. I’m currently re-looking at this because we really need to up our monitoring game. Multiple monkey patching does not seem to be the issue, as I tried that in a fresh project recreate and it had no effect. Removing the only manual monkey patch in the system (from eventlet) does nothing (and did not affect my recreate attempt. At this point in time, I think some random module we’re importing is doing some kind of patching just by importing it; I’m currently stripping out code to see if that is telling.

I finally figured this out! I have created a recreate here: https://github.com/csibbach/NewRelicRecreate.

We were doing some interesting meta programming with a function decorator_shoehorn, which uses functools.partial. That’s where it partial was entering the injection stack. The odd thing is that enabling NewRelic causes this to break; it otherwise works fine. If you stick in a newrelic key into newrelic.ini and do docker-compose up, you can see the broken behavior. If you comment out the first two lines (import newrelic and monkey patch) it works perfectly fine.

I’ve removed usage of decorator_shoehorn in my home codebase and now I am getting the stats in NewRelic that I would expect. I’ll leave this around just in case it helps anybody in the distant future.

1 Like