Pem_image:0.9.7

Are there any known issues with the vizier-pem pods that would cause the memory working set initialization to grow to > 95%? This happens quite frequently on our K8. Everytime I look at the NewRelic Kubernetes dashboard at least one of these vizier-pem pods is alerting.

Is there a configuration I need to change is there something I can look at to help diagnose what is going on?

Thanks!

Hi @aolds,

Thank you for reaching out to the Explorers Hub :slightly_smiling_face:

Vizier-pem pods can utilize a lot of memory and can quickly reach their default limit of 2Gi, which is a possible cause for the alerting. If the limit is reached the pod will terminate.

According to Pixie documentation:

Pixie stores the data it collects in-memory on the nodes in your cluster; no data is sent to a centralized backend outside of the cluster. This is true for both self-hosted Pixie and Pixie Community Cloud. Pixie has a 2GiB memory requirement per node. After installing Pixie, it is normal to see a temporary increase in memory usage of the vizier-pem pods as they begin to fill their data tables.

If you are using helm I suggest you try increasing the default limit:

helm upgrade -n <namespace> --reuse-values <release-name> newrelic/nri-bundle --set pixie-chart.pemMemoryLimit=<int>Gi

Let me know if this helps :slightly_smiling_face:

1 Like

I am not using helm, how else can I do this?