Set up the trace observer

If you're following the Enable distributed tracing procedures and want to set up Infinite Tracing, you'll need to set up a trace observer.

A trace observer is a cloud-based New Relic tool that decides what trace data to keep and send to New Relic. The trace observer lives in New Relic Edge, which is an AWS-based service that provides you with a low-latency and low-cost way to send your telemetry data to New Relic. The trace observer sends data via our Trace API, which is the entry point for all distributed trace data we ingest.

Set up the trace observer

Before setting up a trace observer, understand these points:

• With the exception of the Trace API, these instructions are not standalone; they're part of larger enable procedures. If you're still figuring out what you need, see Enable distributed tracing.
• To avoid configuration conflict issues, you should ideally enable Infinite Tracing for all associated services. If some services in a trace have our standard distributed tracing enabled, you should upgrade those to Infinite Tracing.

To set up a trace observer:

1. Go to one.newrelic.com, and click Apps. Under Your apps, click New Relic Edge.

2. Select an account in the upper-left dropdown. If you have access to multiple accounts, make sure you're in the account where you want Infinite Tracing enabled.

3. If no trace observers are already present, click New trace observer to add one, fill out the information, and click Create.

   Important

   Note: If you select a trace observer in an EU region, you’ll still need a US-based New Relic account because data is reported to US data centers.

4. Under the Endpoints dropdown:

   • Copy the For other integrations endpoint value and have it ready: this will be referred to in later instructions as YOUR_TRACE_OBSERVER_URL.
   • If you're enabling a language agent, also copy the For language agents value and have it ready: this will be referred to as YOUR_TRACE_OBSERVER_HOST.

5. (Optional but recommended) To verify things are working, we recommend sending a sample trace payload. If you're using our Trace API: this step is especially recommended to learn how the API works.

   Send sample payload

   Important

   If you're using Zipkin-format data, see Send Zipkin payload.

   This test sends a sample trace payload with one trace and two spans from a service named Test Service A.

   To send this sample request:

   1. Get the license key for the account you want to report data to and have it ready.

   2. Copy the following curl request into a text editor:

      curl -i -H "Content-Type: application/json" \
        -H "Api-Key: $YOUR_LICENSE_KEY" \
        -H 'Data-Format: newrelic' \
        -H 'Data-Format-Version: 1' \
        -X POST \
        -d '[
            {
                "common": {
                "attributes": {
                    "environment": "staging"
                }
                },
                "spans": [
                {
                    "trace.id": "123456",
                    "id": "ABC",
                    "attributes": {
                        "duration.ms": 12.53,
                        "host": "host123.example.com",
                        "name": "/home",
                        "service.name": "Test Service A"
                    }
                },
                {
                    "trace.id": "123456",
                    "id": "DEF",
                    "attributes": {
                        "duration.ms": 2.97,
                        "host": "host456.example.com",
                        "error.message": "Invalid credentials",
                        "name": "/auth",
                        "parent.id": "ABC",
                        "service.name": "Test Service B"
                    }
                }
                ]
            }
            ]' \
      '$YOUR_TRACE_OBSERVER_URL'

      Copy

   3. Insert your own values into the curl request:

      Value	Description
      $YOUR_LICENSE_KEY	Replace this with your license key.
      $YOUR_TRACE_OBSERVER_URL	Replace this with the For other integrations endpoint value you copied in a previous step.

   4. Copy the curl request into a terminal and execute it.

   5. The test should return HTTP/1.1 202 Accepted, indicating success. If it does not, check the following common issues:

      • Confirm that you used the For other integrations endpoint value.
      • Confirm you're using single quotes around YOUR_TRACE_OBSERVER_URL.
      • Check that you're using the correct API key.

   6. If your test returned HTTP/1.1 202 Accepted, go to the New Relic UI to see a query of the sample payload data using the span attribute service.name = Test Service A (here's a link for that query). Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace.

      Tip

      Traces may take up to one minute to show up in the UI.

6. (Optional) There are several ways to configure Infinite Tracing. This configuration can wait until after you've completed the enable procedures.

7. This procedure is complete. Next, return to finish any remaining instructions for the tracing tool you started enabling:

   • Language agents
   • Third-party telemetry integrations (OpenTelemetry, OpenCensus, others)
   • Trace API: once the trace observer is set up, you're finished and can start instrumenting your application.

Trace observer endpoints

In the trace observer UI, there's an Endpoints dropdown. When setting up the trace observer, we have you copy these values for use at various points of our tracing tool setup instructions. There are two values:

• For language agents: This value is referenced in our code examples as YOUR_TRACE_OBSERVER_HOST. This is used for configuring our language agents to send data to the trace observer.
• For other integrations: This value is referenced in our code examples as YOUR_TRACE_OBSERVER_URL. This is used for configuring our telemetry integrations and for sending data via the Trace API (including sending sample payloads).