As part of Applied Intelligence, Incident Intelligence helps you correlate your incidents and reduce noise in your environment. It gives you an overview of all your incidents, their sources, and related events.
Set up Incident Intelligence
Before setting up Incident Intelligence, note that the count of incident events is a billing factor.
To enable Incident Intelligence, follow these steps. Afterwards, issues should start to appear in your issue feed.
To set up an environment in Incident Intelligence, you need an administrator to select a New Relic account for it. This account should be the one your team is using.
Who sets the environment? Only administrators, and only for accounts where they have admin privileges.
Can administrators set more than one environment? They can set one environment per parent account and its child accounts. More than one can be set if an administrator has privileges for more than one parent account.
Tip
Incident Intelligence is a cross-account product. This means you can send in data from any New Relic account or external source to correlate events.
2. Configure sources
After setting up your environment, determine your incident sources. These are your data inputs.
You can get data from any of the following sources:
By integrating Incident Intelligence with your alerts violations, you can get context and correlations from what you're monitoring.
On the left under Incident Intelligence, click Sources and then click Alerts.
Select the policies you want to connect to Applied Intelligence, and click Connect.
You can add additional alerts policies or remove policies you've already connected in Sources > New Relic Alerts.
Tip
Adding alerts as a source will not affect your current configuration or notifications.
By integrating Incident Intelligence with your Algorithmia machine-learning models, you can monitor your machine learning model performance. To configure Algorithmia for Incident Intelligence, see our integration docs.
By integrating Incident Intelligence with your Aporia machine-learning models, you can monitor your machine learning model performance. To configure our Aporia integration, see our docs.
By integrating Incident Intelligence with your Superwise machine-learning models, you can monitor your machine learning model performance. To configure our Superwise integration, see our docs.
By integrating Incident Intelligence with your New Relic Proactive Detection anomalies, you can get context and correlations from what you're monitoring.
To get data from New Relic Proactive Detection anomalies:
Set CloudWatch to forward all Alarms state changes to that topic:
In the Amazon CloudWatch UI, click Events > Event Pattern.
Select Service Name > CloudWatch.
Select Event Type > CloudWatch Alarm State Change.
Select Targets > SNS Topic, and select your new Amazon SNS topic.
Create a new subscription:
In the Amazon AWS UI, click Create a Subscription.
Select your new Amazon SNS topic.
Select Protocol > choose HTTPS.
In Endpoint, paste the URL you previously copied from the Applied Intelligence Sources.
You can integrate Incident Intelligence with Grafana's notifications for insight into events across your applications and environment. Grafana's webhook notification is a simple way to send information over HTTP to a custom endpoint.
To integrate Grafana as a new webhook:
Log into your Grafana portal using Admin permissions, and choose Alerting.
On the Grafana Notification Channels page, click New Channel > Webhook.
On the left under Incident Intelligence, click Sources, and then click Grafana.
Copy the URL, and paste it into your new Grafana webhook.
EOL NOTICE
We're discontinuing support for several capabilities, including suggested responders for PagerDuty sources in October 2021. For more details, including how you can easily prepare for this transition, see our Explorers Hub post.
You can integrate Incident Intelligence directly with your PagerDuty services to ingest, process, and enhance all of your PagerDuty incidents. Connecting PagerDuty services to Applied Intelligence will not affect your current services or notifications.
The key should be either a personal or general access key with write access. If it's created by a user, the user should be an admin.
Select the PagerDuty services you want to connect to Applied Intelligence, and click Connect.
You can add additional services or remove services you've already connected in Sources > PagerDuty.
By integrating Incident Intelligence with Prometheus Alertmanager, you can receive and correlate your Prometheus alerts with events from other sources.
To integrate Prometheus Alertmanager:
Set up your Alertmanager configuration file by running:
On the left under Incident Intelligence, click Sources and then click Prometheus Alertmanager.
Copy the Prometheus Alertmanager URL, and paste it into the <webhook_config>/url section of your Alertmanager config file.
Reload the Prometheus Alertmanager configuration with one of the two methods:
Send a SIGHUP to the process.
Send an HTTP POST request to the /-/reload endpoint.
Incident Intelligence supports a dedicated REST API interface that lets you integrate with additional systems. The interface allows instrumentation of your code or other monitoring solutions to report any kind of metric or event.
A metric can be a raw data point such as CPU, memory, disk utilization, or business KPI.
An event can be a monitoring alert, deployment event, incident, exceptions or any other change in state that you want to describe.
You can also send any type of data to Incident Intelligence straight from your own systems or applications. The REST API supports secure token-based authentication and accepts JSON content as input.
To enrich alerts data with your Splunk metadata, use Splunk tokens. This helps you leverage your search data, which includes metadata and values from the first row of search results.
If you want to...
Do this...
Access search data
Use the format $<fieldname>$. For example, use $app$ for the app context for the search.
Access field values
To access field values from the first result row that a search returns, use the format $result.<fieldname>$. For example, use $result.host$ for the host value and $result.sourcetype$ for the source type.
Use variables
You can leverage any of the Selected fields in the Splunk search and add any unique fields to the Selected fields to make the data available as a variable.
The following fields will automatically provide hints to the correlation engine:
app: parsed as APPLICATION_NAME
application:parsed as APPLICATION_NAME
application_name: parsed as APPLICATION_NAME
cluster: parsed as CLUSTER_NAME
computer: parsed as HOST_NAME
Dc: parsed as DATACENTER_NAME
datacenter: parsed as DATACENTER_NAME
host: parsed as HOST_NAME
host_name: parsed as HOST_NAME
hostname: parsed as HOST_NAME
transaction: parsed as EVENT_ID
Transaction_id: parsed as EVENT_ID
user: parsed as USER_NAME
3. Configure destinations (ServiceNow and others)
Now that you've set up your sources, you can configure your destinations. These are the data outputs where you view your incidents.
Configure ServiceNow (example)
Using ServiceNow as a destination enables you to push valuable violation data into new ServiceNow incident tickets.
To configure Incident Intelligence to send data to ServiceNow:
Go to one.newrelic.com, click Alerts & AI, in the left nav under Incident Intelligence click Destinations, then click ServiceNow.
Required: Enter a channel name. This is used internally in Applied Intelligence to identify the destination (for example, in Pathways).
Required: Enter your ServiceNow credentials:
Team domain (This must be unique. No two destinations can have the same domain).
Username
Password
Follow the two-way integration on screen instructions:
We're discontinuing support for several capabilities, including suggested responders for PagerDuty sources in October 2021. For more details, including how you can easily prepare for this transition, see our Explorers Hub post.
Recommended: Create a new PagerDuty service to use as a destination. Because PagerDuty services can also be used as sources, this can help you distinguish your data input from your output.
To create a PagerDuty destination:
Go to one.newrelic.com, click Alerts & AI, in the left nav under Incident Intelligence click Destinations, then click PagerDuty.
The key should be either a personal or general access key with write access. If it's created by a user, the user should be an admin. If you've configured a PagerDuty source with an API key, you can use the same key.
Select the PagerDuty services you want to connect to Applied Intelligence, and click Connect.
When you're ready, you can add policies for one or more PagerDuty destinations. You can also transfer the policies over from your existing services or leave them as sources if needed.
From the Destinations > PagerDuty page, you can also:
Review the permissions for your services. Click Authorize when you're done.
Add or delete existing services from the PagerDuty destination.
Edit permissions for any service.
To configure your PagerDuty destinations, use the following settings:
Configuration setting
Description
Trigger new incidents
Required. Trigger correlated parent incidents so you can identify issues faster.
Edit incident titles
Required. Alter your incident titles to help you orient and understand issues.
Add new integrations
Required. Add integrations to enable incident creation for selected services.
Add webhook extensions
Add webhook extensions to sync user actions in PagerDuty to New Relic. This lets you update the correlated issue state.
Auto-resolve correlated incidents
When enabled, this will resolve and automatically close correlated parent/child incidents.
Select a user to take actions in PagerDuty
You need to select a user before you can enable deep integration with PagerDuty. Once you do, the user can:
Add notes to incidents (required): Incident notes are used to enrich incidents with context.
Acknowledge triggered incidents: When enabled, Applied Intelligence will acknowledge and correlate newly triggered incidents in PagerDuty before you're notified.
Use the original escalation policy: When enabled, the escalation policy of the source service will be applied to each incident.
Incident Intelligence will send the event body in JSON format by HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx).
To configure Incident Intelligence to send data via webhook:
Go to one.newrelic.com, click Alerts & AI, in the left nav under Incident Intelligence click Destinations, then click Webhook.
Required: Configure the unique webhook key, used in Applied Intelligence to refer to this webhook configuration and its specific settings.
Required: Configure the destination endpoint where the webhook payload will be sent.
Optional steps:
Configure custom headers, which are key:value pairs of headers to be sent with the request. Example: "Authentication" "Bearer" <bearer token>
Configure a custom payload template that can be used to map New Relic fields to match the destination tool's expected name and format.
Configure priority mapping (critical, high, medium, or low), used to map New Relic's priorities to the priorities expected at the destination.
Tip
There’s a retry mechanism that is triggered a few times with exponential backoff for a couple of minutes once an error occurs. If we reach the retry limit, the Webhook will get auto-disabled.
