Error Installing Pixie with Helm

Hi

I am trying to test out Pixie in our EKS dev environment. We currently have the New relic nri-bundle installed using helm, so now I am basically trying to add the extra settings to enable pixie and upgrade.

Our commands look like this:

helm repo add newrelic https://helm-charts.newrelic.com
helm upgrade newrelic-bundle newrelic/nri-bundle -f newrelic-k8s-integration/qa/values.yml --namespace newrelic

and values .yaml looks like this (with license keys etc revoked):

global:
  licenseKey: xxxxxxxxxxxx
  cluster: eks_eksqa
newrelic-infrastructure:
  privileged: false
ksm:
  enabled: true
kubeEvents:
  enabled: true
newrelic-pixie:
  enabled: true
  apiKey: xxxxxxxxxx
pixie-chart:
  enabled: true
  deployKey: xxxxxxxxxxxxxxxxxxxxxxx
  clusterName: eks_eksqa

Everything above “newrelic-pixie” was already there, everything from that point onwards I have just added.

Trying to run this we are seeing the following error:

$ helm version
version.BuildInfo{Version:"v3.3.0", GitCommit:"8a4aeec08d67a7b84472007529e8097ec3742105", GitTreeState:"dirty", GoVersion:"go1.14.7"}

$ helm repo add newrelic [https://helm-charts.newrelic.com](https://helm-charts.newrelic.com/)
"newrelic" has been added to your repositories

$ helm upgrade newrelic-bundle newrelic/nri-bundle -f newrelic-k8s-integration/qa/values.yml --namespace newrelic
Error: UPGRADE FAILED: unable to recognize "": no matches for kind "NatsCluster" in version "nats.io/v1alpha2"

Our Kubernetes Version is: v1.19.6-eks-49a6c0
Our existing newrelic pods are running this version: newrelic/infrastructure-k8s:1.26.4-unprivileged

Any ideas where I might be going wrong here?

1 Like

Facing the same issue while trying to deploy pixie in a local k8s cluster. Any pointers appreciated.

1 Like

Hi @MartinW and @umesh.sehgal, thanks for reporting this to us! We were curious if the CRD is getting created, could you please run the following and post the output?

kubectl get crd | grep nats

If there are Nats CRDs listed, could you also please run the following and post the output?

kubectl describe crd <name>

Thanks!

Hi @daffinito

We have no CRDs called Nats.
(It’s been a few weeks since I attempted the install now so I’m not sure if that will make any difference).
Is this something that needs to be installed first or should the Pixie helm install take care of it?

Hi @MartinW - Thank you for confirming you don’t have any Nats CRDs installed

I’ve done some further testing and it looks like this occurs when trying to update an nri-bundle installation that was installed before the Nats CRDs were needed. It looks like this is due to a helm caveat mentioned here:

These CRDs are not templated, but will be installed by default when running a helm install for the chart. If the CRD already exists, it will be skipped with a warning. If you wish to skip the CRD installation step, you can pass the --skip-crds flag.

Some caveats (and explanations)
There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

To workaround this, we can reinstall the chart. Since you already have your values file, you should only need to uninstall the current chart the run a helm upgrade --install:

helm uninstall newrelic-bundle -n newrelic

helm upgrade --install newrelic-bundle newrelic/nri-bundle -f newrelic-k8s-integration/qa/values.yml -n newrelic

For anyone viewing this that doesn’t have their current values, here are complete steps to reinstall including gathering the current values used by the release:

  1. Get the release name and namespace for the nri-bundle installation
helm list -A
  1. Save current values to a file
helm get values RELEASE_NAME -n NAMESPACE -o yaml > current-values.yaml
  1. Uninstall the current chart
helm uninstall RELEASE_NAME -n NAMESPACE
  1. Re-install with the values file created on step 2, adding the options to enable pixie (either add the values to current-values.yaml such as @MartinW did above, or use the --set options):
  • If you are adding the pixie settings to current-values.yaml:
helm upgrade --install RELEASE_NAME newrelic/nri-bundle -f /path/to/current-values.yaml -n NAMESPACE
  • If you want to use --set instead:
helm upgrade --install RELEASE_NAME newrelic/nri-bundle -n NAMESPACE -f /path/to/current-values.yaml --set newrelic-pixie.enabled=true \
 --set newrelic-pixie.apiKey=PIXIE_API_KEY \
 --set pixie-chart.enabled=true \
 --set pixie-chart.deployKey=PIXIE_DEPLOY_KEY \
 --set pixie-chart.clusterName=CLUSTER_NAME

Hope this helps! Let me know how it goes or if you have any questions.

2 Likes

Thanks @daffinito :beers:
Other projects have taken priority at the moment but I’ll give this a try soon and let you know if it works.

2 Likes

Thanks @MartinW, let us know how that works out for you!

I was running into the same issue trying to use the nri-bundle chart to add Pixie to my eks 1.19 cluster. Upon trying the the chart upgrade I was getting the same CRD Nats error.

Upon following @daffinito’s suggestion of uninstalling and re-installing I am actually seeing the exact same error.

Something maybe to note: I using helm secrets.

The error I am getting seems to directly mentioning a failure to run crd-install

Release "nri-bundle" does not exist. Installing it now.
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "NatsCluster" in version "nats.io/v1alpha2"

From poking around it seems that Helm 3 removed the crd-install hook that I suspect this chart maybe using? A little odd though considering how new this chart is.

Hello I still have the same problem and not able to install:

root@k8-master01-16:/home/kubeadm# kubectl apply -f https://raw.githubusercontent.com/pixie-labs/pixie/main/k8s/vizier_deps/base/nats/nats_crd.yaml && \

helm repo add newrelic https://helm-charts.newrelic.com &&
kubectl create namespace newrelic ; helm upgrade --install newrelic-bundle newrelic/nri-bundle
–set global.licenseKey=XXXXXX
–set global.cluster=ARI-PRD
–namespace=newrelic
–set newrelic-infrastructure.privileged=true
–set ksm.enabled=true
–set prometheus.enabled=true
–set kubeEvents.enabled=true
–set logging.enabled=true
–set newrelic-pixie.enabled=true
–set newrelic-pixie.apiKey=XXXXX
–set pixie-chart.enabled=true
–set pixie-chart.deployKey=XXXXXX
–set pixie-chart.clusterName=ARI-PRD
customresourcedefinition.apiextensions.k8s.io/natsclusters.nats.io unchanged
customresourcedefinition.apiextensions.k8s.io/natsserviceroles.nats.io unchanged
“newrelic” has been added to your repositories
Error from server (AlreadyExists): namespaces “newrelic” already exists
Error: UPGRADE FAILED: “newrelic-bundle” has no deployed releases
root@k8-master01-16:/home/kubeadm# helm uninstall newrelic-bundle -n newrelic
release “newrelic-bundle” uninstalled
root@k8-master01-16:/home/kubeadm# helm upgrade --install newrelic-bundle newrelic/nri-bundle -f newrelic-k8s-integration/qa/values.yml -n newrelic
Release “newrelic-bundle” does not exist. Installing it now.
Error: open newrelic-k8s-integration/qa/values.yml: no such file or directory
root@k8-master01-16:/home/kubeadm# helm upgrade --install RELEASE_NAME newrelic/nri-bundle -n NAMESPACE -f /path/to/current-values.yaml --set newrelic-pixie.enabled=true
–set newrelic-pixie.apiKey=PIXIE_API_KEY
–set pixie-chart.enabled=true
–set pixie-chart.deployKey=PIXIE_DEPLOY_KEY
–set pixie-chart.clusterName=CLUSTER_NAME
Release “RELEASE_NAME” does not exist. Installing it now.
Error: open /path/to/current-values.yaml: no such file or directory
root@k8-master01-16:/home/kubeadm#

@rsabas, it looks like your helm install failed simply because you did not update the provided config paths?

I have the same issue:

Release “newrelic-bundle” does not exist. Installing it now.
manifest_sorter.go:192: info: skipping unknown hook: “crd-install”
Error: unable to build kubernetes objects from release manifest: unable to recognize “”: no matches for kind “NatsCluster” in version “nats.io/v1alpha2

helm version
version.BuildInfo{Version:“v3.6.3”, GitCommit:“d506314abfb5d21419df8c7e7e68012379db2354”, GitTreeState:“dirty”, GoVersion:“go1.16.5”}

kubectl get crd | grep nats

natsclusters.nats.io 2021-08-05T07:54:37Z
natsserviceroles.nats.io 2021-08-05T07:54:37Z

k get all --namespace=newrelic
No resources found in newrelic namespace.

k version
Client Version: version.Info{Major:“1”, Minor:“20+”, GitVersion:“v1.20.4-dirty”, GitCommit:“e87da0bd6e03ec3fea7933c4b5263d151aafd07c”, GitTreeState:“dirty”, BuildDate:“2021-03-15T09:58:13Z”, GoVersion:“go1.16.2”, Compiler:“gc”, Platform:“darwin/amd64”}
Server Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.2”, GitCommit:“faecb196815e248d3ecfb03c680a4507229c2a56”, GitTreeState:“clean”, BuildDate:“2021-01-13T13:20:00Z”, GoVersion:“go1.15.5”, Compiler:“gc”, Platform:“linux/amd64”}

Can NR provide the crd’s that need to be upgraded for this process in the chart instructions much how the prometheus ones are laid out?

example: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#from-17x-to-18x