Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Feature Idea: Pika instrmentation leaks memory by holding references to all connections ever created

feature-idea
rfb

#1

Hello,

It appears that pika instrumentation done by the Python agent leaks memory. If you grep through the code you can see that every time a BlockingChannel instance is created, it’s bound method is added to _no_trace_methods which is a module level set. But it’s never removed from the set.

grep -C 4 -r _no_trace_methods newrelic/
--
newrelic/hooks/messagebroker_pika.py-from newrelic.common.object_wrapper import (wrap_function_wrapper, wrap_object,
newrelic/hooks/messagebroker_pika.py-        FunctionWrapper)
newrelic/hooks/messagebroker_pika.py-
newrelic/hooks/messagebroker_pika.py-
newrelic/hooks/messagebroker_pika.py:_no_trace_methods = set()                                                        # XXXX
newrelic/hooks/messagebroker_pika.py-_START_KEY = '_nr_start_time'
newrelic/hooks/messagebroker_pika.py-KWARGS_ERROR = 'Supportability/hooks/pika/kwargs_error'
newrelic/hooks/messagebroker_pika.py-
newrelic/hooks/messagebroker_pika.py-
--
newrelic/hooks/messagebroker_pika.py-def _nr_wrap_BlockingChannel___init__(wrapped, instance, args, kwargs):
newrelic/hooks/messagebroker_pika.py-    ret = wrapped(*args, **kwargs)
newrelic/hooks/messagebroker_pika.py-    # Add the bound method to the set of methods not to trace.
newrelic/hooks/messagebroker_pika.py-    if hasattr(instance, '_on_consumer_message_delivery'):
newrelic/hooks/messagebroker_pika.py:        _no_trace_methods.add(instance._on_consumer_message_delivery)            # XXXX
newrelic/hooks/messagebroker_pika.py-    return ret
newrelic/hooks/messagebroker_pika.py-
newrelic/hooks/messagebroker_pika.py-
newrelic/hooks/messagebroker_pika.py-def _bind_params_BlockingChannel_basic_consume(consumer_callback, queue, *args,
--
newrelic/hooks/messagebroker_pika.py-def _wrap_Channel_consume_callback(module, obj, bind_params,
newrelic/hooks/messagebroker_pika.py-        callback_referrer):
newrelic/hooks/messagebroker_pika.py-    def _nr_wrapper_Channel_consume_(wrapped, instance, args, kwargs):
newrelic/hooks/messagebroker_pika.py-        callback, queue = bind_params(*args, **kwargs)
newrelic/hooks/messagebroker_pika.py:        if callback in _no_trace_methods:                                        # XXXX
newrelic/hooks/messagebroker_pika.py-            # This is an internal callback that should not be wrapped.
newrelic/hooks/messagebroker_pika.py-            return wrapped(*args, **kwargs)
newrelic/hooks/messagebroker_pika.py-        name = callable_name(callback)
newrelic/hooks/messagebroker_pika.py- 

Here’s a small script that demonstrates the problem:

#!/usr/bin/env python

import gc
import os
import time

from pika import ConnectionParameters, PlainCredentials
from pika.adapters import BlockingConnection
from pika.adapters.blocking_connection import BlockingChannel


def print_my_mem():
    for line in open('/proc/{}/status'.format(os.getpid())):
        if line.startswith('VmRSS'):
            print(line.strip())


def print_chans():
    num = sum(1 for o in gc.get_objects()
              if isinstance(o, BlockingChannel))
    print('BlockingChannel #: {}'.format(num))


if __name__ == '__main__':
    creds = PlainCredentials('guest', 'guest')
    params = ConnectionParameters(credentials=creds,
                                  host='rabbitmq')
    print_chans()
    print_my_mem()
    i = 0
    while True:
        conn = BlockingConnection(params)
        chan = conn.channel()
        chan.close()
        conn.close()
        del chan
        del conn
        while gc.collect():
            pass
        i += 1
        if not i % 100:
            print_chans()
            print_my_mem()

Running it:

NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program ./nrml.py 
BlockingChannel #: 0
VmRSS:	   30136 kB
BlockingChannel #: 100
VmRSS:	   31736 kB
BlockingChannel #: 200
VmRSS:	   33216 kB
BlockingChannel #: 300
VmRSS:	   34572 kB
... etc

The # of channels should remain at 0 and VmRSS should not grow.

This could be solved with WeakMethod although it’s 3.4+ or a custom object that emulates hash and == on bound methods, though I’m not 100% sure how to implement == and how it would affect performance. In any case, I don’t have enough knowledge of the Agent internals to know what would be the best option.

Python 3.6.1
Python agent v. 2.96.0.81


New Relic edit

  • I want this, too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.


#2

@ifilatov Thanks so much for reporting this issue and for your reproduction of the problem! I’ve brought this to the attention of our Python developers, who are looking into this, and I’ll make sure to update this post when we have more information.


#3

@ifilatov I wanted to let you know that we’ve released a fix for this with the latest version of the Python agent. Check out the release notes here: Python Agent 2.100.0.84.

I also wanted to say that your report of this was AWESOME and our devs were super impressed!!! Thanks for being so thorough and including a repo case and test!


#4

Glad I could help and thanks for the fix!


#5

Agent v2.98 adds instrumentation for handle_exception in Django Rest Framework views. This causes problems in my case because FunctionWrapper is not picklable (I went from 2.96 to 2.100). I’ve found a workaround but you might want to look into it - whether in general FunctionWrapper should be picklable or not.


#6

Awesome addition, @ifilatov! Thanks for jumping in and helping the community! :blush:


#7

@ifilatov You’re right that FunctionWrapper is not currently picklable. I’ve submitted a feature request on your behalf. It’s sounds like you already have a workaround, but whatever the object is that’s been wrapped, you could get the unwrapped (picklable) object (let’s say foo) like this: foo.__wrapped__.


#8

Yes, thanks, I found the __wrapped__ attr. Since I need to pickle an instance of DRF view, as workaround I just remove handle_exception from the instance, unshadowing the one on the class.