Link your applications to Kubernetes

You can surface Kubernetes metadata and link it to your APM agents as distributed traces to explore performance issues and troubleshoot transaction errors. For more information, see this New Relic blog post.

You can quickly start monitoring Kubernetes clusters using Auto-telemetry with Pixie, which is currently a beta release. This Pixie integration into New Relic does not require a language agent. Learn more about Auto-telemetry with Pixie here.

Compatibility and requirements

Before linking Kubernetes metadata to your APM agents, make sure you meet the following requirements:

• Kubernetes requirements
• Network requirements
• APM agent compatibility
• OpenShift requirements

Kubernetes requirements

To link your applications and Kubernetes, your cluster must have the MutatingAdmissionWebhook controller enabled, which requires Kubernetes 1.9 or higher.

To verify that your cluster is compatible, run the following command:

kubectl api-versions | grep admissionregistration.k8s.io/v1beta1 
admissionregistration.k8s.io/v1beta1

If you see a different result, follow the Kubernetes documentation to enable admission control in your cluster.

Network requirements

For Kubernetes to speak to our MutatingAdmissionWebhook, the master node (or the API server container, depending on how the cluster is set up) should be allowed egress for HTTPS traffic on port 443 to pods in all of the other nodes in the cluster.

This might require specific configuration depending on how the infrastructure is set up (on-premises, AWS, Google Cloud, etc).

APM agent compatibility

The following New Relic agents collect Kubernetes metadata:

• Go 2.3.0 or higher
• Java 4.10.0 or higher
• Node.js 5.3.0 or higher
• Python 4.14.0 or higher
• Ruby 6.1.0 or higher
• .NET 8.17.438 or higher

Openshift requirements

To link Openshift and Kubernetes you must enable mutating admission webhooks, which requires Openshift 3.9 or higher.

1. During the process, install a resource that requires admin permissions to the cluster. Run this to log in as admin:

   oc login -u system:admin

   コピー

2. Check that webhooks are correctly configured. If they are not, update the master-config.yaml file.

   admissionConfig:
              pluginConfig:
                MutatingAdmissionWebhook:
                  configuration:
                    apiVersion: apiserver.config.k8s.io/v1alpha1
                    kubeConfigFile: /dev/null
                    kind: WebhookAdmission
                ValidatingAdmissionWebhook:
                  configuration:
                    apiVersion: apiserver.config.k8s.io/v1alpha1
                    kubeConfigFile: /dev/null
                    kind: WebhookAdmission
                    location: ""

   コピー

   重要

   Add kubeConfigFile: /dev/null to address some issues in Openshift.

3. Enable certificate signing by editing the YAML file and updating your configuration:

   kubernetesMasterConfig:
      controllerArguments:
         cluster-signing-cert-file:
         - "/etc/origin/master/ca.crt"
         cluster-signing-key-file:
         - "/etc/origin/master/ca.key"

   コピー

4. Restart the Openshift services in the master node.

Configure the injection of metadata

By default, all the pods you create that include APM agents have the correct environment variables set and the metadata injection applies to the entire cluster. To check that the environment variables have been set, any container that is running must be stopped, and a new instance started (see Validate the injection of metadata).

This default configuration also uses the Kubernetes certificates API to automatically manage the certificates required for the injection. If needed, you can limit the injection of metadata to specific namespaces in your cluster or self-manage your certificates.

Default configuration

To proceed with the default injection of metadata, follow these steps:

1. Download the YAML file:

   curl -O http://download.newrelic.com/infrastructure_agent/integrations/kubernetes/k8s-metadata-injection-latest.yaml

   コピー

2. Replace YOUR_CLUSTER_NAME with the name of your cluster in the YAML file.

3. Apply the YAML file to your Kubernetes cluster:

   kubectl apply -f k8s-metadata-injection-latest.yaml

   コピー

Custom configuration

You can limit the injection of metadata only to specific namespaces by using labels.

To enable this feature, edit your YAML file by finding and uncommenting the following lines:

# namespaceSelector:
  #   matchLabels:
  #     newrelic-metadata-injection: enabled

With this option, injection is only applied to those namespaces that have the newrelic-metadata-injection label set to enabled:

kubectl label namespace YOUR_NAMESPACE newrelic-metadata-injection=enabled

Manage custom certificates

To use custom certificates you need a specific YAML file:

1. Download the YAML file without automatic certificate management:

   curl -O http://download.newrelic.com/infrastructure_agent/integrations/kubernetes/k8s-metadata-injection-custom-certs-latest.yaml

   コピー

2. Replace YOUR_CLUSTER_NAME with the name of your cluster in the YAML file.

3. Apply the YAML file to your Kubernetes cluster:

   kubectl apply -f k8s-metadata-injection-custom-certs-latest.yaml

   コピー

Once you have the correct YAML file, you can proceed with the custom certificate management option.

You need your certificate, server key, and Certification Authority (CA) bundle encoded in PEM format.

• If you have them in the standard certificate format (X.509), install openssl, and run the following:

  openssl x509 -in CERTIFICATE_FILENAME -outform PEM -out CERTIFICATE_FILENAME.pem 
  openssl x509 -in SERVER_KEY_FILENAME -outform PEM -out SERVER_KEY_FILENAME.pem 
  openssl x509 -in CA_BUNDLE_FILENAME -outform PEM -out BUNDLE_FILENAME.pem

  コピー

• If your certificate/key pair are in another format, see the Digicert knowledgebase for more help.

Create the TLS secret with the signed certificate/key pair, and patch the mutating webhook configuration with the CA using the following commands:

kubectl create secret tls newrelic-metadata-injection-secret \
--key=PEM_ENCODED_SERVER_KEY \
--cert=PEM_ENCODED_CERTIFICATE \
--dry-run -o yaml |
kubectl -n default apply -f -

caBundle=$(cat PEM_ENCODED_CA_BUNDLE | base64 | td -d '\n')
kubectl patch mutatingwebhookconfiguration newrelic-metadata-injection-cfg --type='json' -p "[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${caBundle}'}]"

Validate the injection of metadata

In order to validate that the webhook (responsible for injecting the metadata) was installed correctly, deploy a new pod and check for the New Relic environment variables.

1. Create a dummy pod containing Busybox by running:

   kubectl create -f https://git.io/vPieo

   コピー

2. Check if New Relic environment variables were injected:

   kubectl exec busybox0 -- env | grep NEW_RELIC_METADATA_KUBERNETES
   
   NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME=fsi
   NEW_RELIC_METADATA_KUBERNETES_NODE_NAME=nodea
   NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME=default
   NEW_RELIC_METADATA_KUBERNETES_POD_NAME=busybox0
   NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME=busybox

   コピー

Disable the injection of metadata

To disable/uninstall the injection of metadata, use the following commands:

1. Delete the Kubernetes objects using the yaml file:

   kubectl delete -f k8s-metadata-injection-latest.yaml

   コピー

2. Delete the TLS secret containing the certificate/key pair:

   kubectl delete secret/newrelic-metadata-injection-secret

   コピー

Troubleshooting

Follow these troubleshooting tips as needed.

TLS handshake error from 10.131.0.29:37428: remote error: tls: unknown certificate authority
TLS handshake error from 10.129.0.1:49314: remote error: tls: bad certificate