If you encounter problems while setting up private locations for New Relic's synthetic monitoring, try these troubleshooting tips to identify common minion problems.
Job demand and consumption
Identify your current throughput and assess whether your CPM is able to keep up with the job load.
Jobs demand throughput per minute:
SELECT rate(uniqueCount(jobId), 1 minute) FROM SyntheticRequest WHERE type != 'SIMPLE' FACET location SINCE 2 days agoJobs per minute over time:
SELECT rate(uniqueCount(jobId), 1 minute) FROM SyntheticRequest WHERE type != 'SIMPLE' FACET location SINCE 2 weeks ago TIMESERIESIdentify queue growth:
SELECT derivative(checksPending, 1 minute) as 'queue growth rate (per minute)' FROM SyntheticsPrivateLocationStatus where name = 'YOUR_PRIVATE_LOCATION' SINCE 2 days ago TIMESERIESConsumption rate:
SELECT rate(uniqueCount(jobId), 1 minute) FROM SyntheticRequest WHERE type != 'SIMPLE' FACET location SINCE 2 weeks ago TIMESERIESClean up failing monitors that are consuming resources:
SELECT 100*latest(minionJobsFailed)/latest(minionJobsReceived) as 'job failure rate (%)' from SyntheticsPrivateMinion FACET minionLocation SINCE 2 weeks ago TIMESERIES MAX
Memory usage troubleshooting
Recommended memory allocation per CPM
Docker: 2.5 Gi per cpu core for Docker CPM
Kubernetes CPM:
- 3 Gi per CPM heavy worker
- 3 Gi per healthcheck pod
- 1.6 Gi for minion pod
Target 60% max memory utilization
Helpful queries to identify current usage:
Memory usage per location:
SELECT latest(minionPhysicalMemoryUsedPercentage) from SyntheticsPrivateMinion FACET minionLocation SINCE 2 weeks ago TIMESERIES MaxMemory usage per hostname:
SELECT latest(minionPhysicalMemoryUsedPercentage) FROM SyntheticsPrivateMinion WHERE minionLocation = 'YOUR_PRIVATE_LOCATION' FACET minionHostname SINCE 2 weeks ago TIMESERIES Max
CPU usage troubleshooting
Identify potential periods of high cpu demand:
Overall CPU usage:
SELECT latest(minionProcessorsUsagePercentage) FROM SyntheticsPrivateMinion FACET minionLocation SINCE 2 weeks ago TIMESERIES MaxCPU usage for a specified location:
SELECT latest(minionProcessorsUsagePercentage) FROM SyntheticsPrivateMinion WHERE minionLocation = 'YOUR_PRIVATE_LOCATION' FACET minionHostname SINCE 2 weeks ago TIMESERIES Max
Network troubleshooting
Network access is red:
The private minion makes a check to New Relic to verify public network access. If public network access is not available, the Network Access icon on the private minion's Overview page will turn red, but the private minion will otherwise be fully functional. If this is the expected behavior, you can safely ignore this error.
You may also see log entries like this:
2016-01-21 21:47:09,401 [dw-22 - GET /private-location/status] c.n.s.m.util.PrivateMinionUtils WARN Unable to detect Public Network Access trying to fetch https://newrelic.com/synthetics: java.net.ConnectException: Connection refused
New Relic access is red:
The private minion must have access to https://synthetics-horde.nr-data.net
in order to retrieve the list of jobs to run and publish results that you can view in your New Relic user interface. If your firewall rules don't permit direct access, you must configure proxy access. If necessary, add the DNS endpoint to your allow list.
Network IOPS:
Network IOPS can also be a good place to troubleshoot if using a cloud provider for your CPM. Both Docker and Kubernetes CPMs require a decent amount of write throughput. Analyze your average IOPS and determine the appropriate volume size needed for your disk operations per second.
Other unexpected problems
If you encounter other unusual situations, recycle the application. The private minion is a stateless virtual appliance, so you can safely recycle or reboot it at any time with minimal impact to your private location. Even if all your minions are offline at the same time, synthetic monitoring will simply queue pending checks and re-run them once your minions are back online.
To recycle your minion, log into the console and issue this command:
sudo restart synthetics-minion
If recycling the minion does not resolve your problem, log into the console and restart the minion:
sudo reboot now
For more help
If you need more help, check out these support and learning resources:
- Browse the Explorers Hub to get help from the community and join in discussions.
- Find answers on our sites and learn how to use our support portal.
- Run New Relic Diagnostics, our troubleshooting tool for Linux, Windows, and macOS.
- Review New Relic's and and documentation.