• /
  • Log in
  • Free account

Elasticsearch monitoring integration

Our Elasticsearch integration collects and sends inventory and metrics from your Elasticsearch cluster to our platform, where you can see the health of your Elasticsearch environment. We collect metrics at the cluster, node, and index level so you can more easily find the source of any problems.

Read on to install the integration, and to see what data we collect.

Compatibility and requirements

Our integration is compatible with Elasticsearch 5.x through 7.x

If Elasticsearch is not running on Kubernetes or Amazon ECS, you must install the infrastructure agent on a host that's running Elasticsearch. Otherwise:

Quick start

Instrument your Elasticsearch cluster quickly and send your telemetry data with guided install. Our guided install creates a customized CLI command for your environment that downloads and installs the New Relic CLI and the infrastructure agent.

A screenshot of the guided install CLI.

Ready to get started? Click one of these button to try it out.

Guided install

Our guided install uses the infrastructure agent to set up the Elasticsearch integration. Not only that, it discovers other applications and log sources running in your environment and then recommends which ones you should instrument.

The guided install works with most setups. But if it doesn't suit your needs, you can find other methods below to get started monitoring your Elasticsearch cluster.

Install and activate

To install the Elasticsearch integration, follow the instructions for your environment:

Additional notes:

Configuration

Configure the integration

There are several ways to configure the integration, depending on how it was installed:

An integration's YAML-format configuration is where you can place required login credentials and configure how data is collected. Which options you change depend on your setup and preference.

The configuration file has common settings applicable to all integrations, such as interval, timeout, inventory_source. To read all about these common settings, refer to our Configuration Format document.

Important

If you are still using our legacy configuration/definition files, please refer to this document for help.

Specific settings related to Elasticsearch are defined using the env section of the configuration file. These settings control the connection to your Elasticsearch instance as well as other security settings and features. The list of valid settings is described in the next section of this document.

Elasticsearch Instance Settings

The Elasticsearch integration collects both metrics(M) and inventory(I) information. In the table, use the Applies To column for the settings available to each collection:

Setting

Description

Default

Applies to

HOSTNAME

Hostname or IP where Elasticsearch is running.

localhost

M/I

LOCAL_HOSTNAME

Hostname or IP of the Elasticsearch node from which to collect inventory data. Should only be set if you do not wish to collect inventory data against localhost.

localhost

M

PORT

Port on which Elasticsearch is listening.

9200

M/I

USERNAME

Username for accessing Elasticsearch Node.

N/A

M/I

PASSWORD

Password for the given user.

N/A

M/I

CLUSTER_ENVIRONMENT

A way to further specify which cluster we are gathering data for, example: 'staging'.

N/A

M/I

USE_SSL

Use SSL when communicating with the Elasticsearch node.

false

M/I

CA_BUNDLE_FILE

Location of SSL certificate on the host. Only required if USE_SSL is true.

N/A

M/I

CA_BUNDLE_DIR

Alternative Certificate Authority bundle directory.

N/A

M/I

SSL_ALTERNATIVE_HOSTNAME

Alternative server hostname that the integration will accept as valid for the purposes of SSL negotiation.

N/A

M/I

TLS_INSECURE_SKIP_VERIFY

Skip the verification of the server's certificate chain and hostname.

false

M/I

CONFIG_PATH

Path to the Elasticsearch configuration file.

/etc/elasticsearch/
elasticsearch.yml

I

TIMEOUT

Timeout for API requests, in seconds.

30

M/I

COLLECT_INDICES

Signals whether to collect indices metrics or not.

true

M

COLLECT_PRIMARIES

Signals whether to collect primaries metrics or not.

true

M

INDICES_REGEX

Filter which indices are collected.

N/A

M

MASTER_ONLY

Collect cluster metrics on the elected master only.

false

M

METRICS

Set to true to enable metrics-only collection.

false

INVENTORY

Set to true to enable inventory-only collection.

false

The values for these settings can be defined in several ways:

  • Adding the value directly in the config file. This is the most common way.
  • Replacing the values from environment variables using the {{}} notation. This requires infrastructure agent v1.14.0+. Read more here or see the example below.
  • Using secrets management. Use this to protect sensitive information, such as passwords that would be exposed in plain text on the configuration file. For more information, see Secrets management.

Labels/Custom attributes

You can further decorate your metrics using labels. Labels allow you to add key/value pair attributes to your metrics, which you can then use to query, filter, or group your metrics on.
Our default sample config file includes examples of labels; however, as they are not mandatory, you can remove, modify, or add new ones of your choice.

labels:
env: production
role: load_balancer

Example configurations

Find and use data

Data from this service is reported to an integration dashboard.

Elasticsearch data is attached to the following event types:

You can query this data for troubleshooting purposes or to create custom charts and dashboards.

For more on how to find and use your data, see Understand integration data.

Metric data

The Elasticsearch integration collects the following metric data attributes. Each metric name is prefixed with a category indicator and a period, such as cluster. or shards..

Elasticsearch cluster metrics

These attributes are attached to the ElasticsearchClusterSample event type:

Metric

Description

cluster.dataNodes

The number of data nodes in the cluster.

cluster.nodes

The number of nodes in the cluster.

cluster.status

The Elasticsearch cluster health: red, yellow, or green.

shards.active

The number of active shards in the cluster.

shards.initializing

The number of shards that are currently initializing.

shards.primaryActive

The number of active primary shards in the cluster.

shards.relocating

The number of shards that are relocating from one node to another.

shards.unassigned

The number of shards that are unassigned to a node.

Elasticsearch node metrics

These attributes are attached to the ElasticsearchNodeSample event type:

Metric

Description

activeSearches

The number of active searches.

activeSearchesInMilliseconds

The time spent on the search fetch.

breakers.estimatedSizeFieldDataCircuitBreakerInBytes

The estimated size of the field data circuit breaker, in bytes.

breakers.estimatedSizeParentCircuitBreakerInBytes

The estimated size of the parent circuit breaker, in bytes.

breakers.estimatedSizeRequestCircuitBreakerInBytes

The estimated size of the request circuit breaker, in bytes.

breakers.fieldDataCircuitBreakerTripped

The number of times the field data circuit breaker has tripped.

breakers.parentCircuitBreakerTripped

The number of times the parent circuit breaker has tripped.

breakers.requestCircuitBreakerTripped

The number of times the request circuit breaker has tripped.

cache.cacheSizeIDInBytes

The size of the id cache, in bytes.

flush.indexFlushDisk

The number of index flushes to disk since start.

flush.timeFlushIndexDiskInSeconds

The time spent flushing the index to disk.

fs.bytesAvailableJVMInBytes

Bytes available to this Java virtual machine on this file store, in bytes.

fs.bytesReadsInBytes

The total bytes read from the file store, in bytes.

fs.bytesUserIoOperationsInBytes

The total bytes used for all I/O operations on the file store, in bytes.

fs.iOOperations

The total I/O operations on the file store.

fs.reads

The total number of reads from the file store.

fs.totalSizeInBytes

The total size of the file store, in bytes.

fs.unallocatedBytesInBytes

The total number of unallocated bytes in the file store, in bytes.

fs.writes

The total number of writes to the file store.

fs.writesInBytes

The total bytes written to the file store, in bytes.

get.currentRequestsRunning

The number of get requests currently running.

get.requestsDocumentExists

The number of get requests where the document existed.

get.requestsDocumentExistsInMilliseconds

The time spent on get requests where the document existed.

get.requestsDocumentMissing

The number of get requests where the document was missing.

get.requestsDocumentMissingInMilliseconds

The time spent on get requests where the document was missing.

get.timeGetRequestsInMilliseconds

The time spent on get requests.

get.totalGetRequests

The number of get requests.

http.currentOpenConnections

The number of current open HTTP connections.

http.openedConnections

The number of opened HTTP connections.

indexing.docsCurrentlyDeleted

The number of documents currently being deleted from an index.

indexing.documentsCurrentlyIndexing

The number of documents currently being indexed to an index.

indexing.documentsIndexed

The number of documents indexed to an index.

indexing.timeDeletingDocumentsInMilliseconds

The time spent deleting documents from an index.

indexing.timeIndexingDocumentsInMilliseconds

The time spent indexing documents to an index.

indexing.totalDocumentsDeleted

The number of documents deleted from an index.

indices.indexingOperationsFailed

The number of failed indexing operations.

indices.indexingWaitedThrottlingInMilliseconds

The time indexing waited due to throttling.

indices.memoryQueryCacheInBytes

The memory used by the query cache, in bytes.

indices.numberIndices

The number of documents across all primary shards assigned to the node.

indices.queryCacheEvictions

The number of query cache evictions.

indices.queryCacheHits

The number of query cache hits.

indices.queryCacheMisses

The number of query cache misses.

indices.recoveryOngoingShardSource

The number of ongoing recoveries for which a shard serves as a source.

indices.recoveryOngoingShardTarget

The number of ongoing recoveries for which a shard serves as a target.

indices.recoveryWaitedThrottlingInMilliseconds

The total time recoveries waited due to throttling.

indices.requestCacheEvictions

The number of request cache evictions.

indices.requestCacheHits

The number of request cache hits.

indices.requestCacheMemoryInBytes

The memory used by the request cache, in bytes.

indices.requestCacheMisses

The number of request cache misses.

indices.segmentsIndexShard

The number of segments in an index shard.

indices.segmentsMaxMemoryIndexWriterInBytes

The maximum memory used by the index writer, in bytes.

indices.segmentsMemoryUsedDocValuesInBytes

The memory used by doc values, in bytes.

indices.segmentsMemoryUsedFixedBitSetInBytes

The memory used by fixed bit set, in bytes.

indices.segmentsMemoryUsedIndexSegmentsInBytes

The memory used by index segments, in bytes.

indices.segmentsMemoryUsedIndexWriterInBytes

The memory used by the index writer, in bytes.

indices.segmentsMemoryUsedNormsInBytes

The memory used by norm, in bytes.

indices.segmentsMemoryUsedSegmentVersionMapInBytes

The memory used by the segment version map, in bytes.

indices.segmentsMemoryUsedStoredFieldsInBytes

The memory used by stored fields, in bytes.

indices.segmentsMemoryUsedTermsInBytes

The memory used by terms, in bytes.

indices.segmentsMemoryUsedTermVectorsInBytes

The memory used by term vectors, in bytes.

indices.translogOperations

The number of operations in the transaction log.

indices.translogOperationsInBytes

The size of the transaction log, in bytes.

jvm.gc.collections

The number of garbage collections run by the JVM.

jvm.gc.collectionsInMilliseconds

The time spent on garbage collection in the JVM.

jvm.gc.concurrentMarkSweep

The number of concurrent mark & sweep GCs in the JVM.

jvm.gc.concurrentMarkSweepInMilliseconds

The time spent on concurrent mark & sweep GCs in the JVM.

jvm.gc.majorCollectionsOldGenerationObjects

The number of major GCs in the JVM that collect old generation objects.

jvm.gc.majorCollectionsOldGenerationObjectsInMilliseconds

The time spent in major GCs in the JVM that collect old generation objects.

jvm.gc.minorCollectionsYoungGenerationObjects

The number of minor GCs in the JVM that collects young generation objects.

jvm.gc.minorCollectionsYoungGenerationObjectsInMilliseconds

The time spent in minor GCs in the JVM that collects young generation objects.

jvm.gc.parallelNewCollections

The number of parallel new GCs in the JVM.

jvm.gc.parallelNewCollectionsInMilliseconds

The time spent on parallel new GCs in the JVM.

jvm.mem.heapCommittedInBytes

The amount of memory guaranteed to be available to the JVM heap, in bytes.

jvm.mem.heapMaxInBytes

The maximum amount of memory that can be used by the JVM heap, in bytes.

jvm.mem.heapUsed

The percentage of memory currently used by the JVM heap as a value between 0 and 1.

jvm.mem.heapUsedInBytes

The amount of memory currently used by the JVM heap, in bytes.

jvm.mem.maxOldGenerationHeapInBytes

The maximum amount of memory that can be used by the old generation heap, in bytes.

jvm.mem.maxSurvivorSpaceInBytes

The maximum amount of memory that can be used by the survivor space, in bytes.

jvm.mem.maxYoungGenerationHeapInBytes

The maximum amount of memory that can be used by the young generation heap, in bytes.

jvm.mem.nonHeapCommittedInBytes

The amount of memory guaranteed to be available to JVM non-heap, in bytes.

jvm.mem.nonHeapUsedInBytes

The amount of memory currently used by the JVM non-heap, in bytes.

jvm.mem.usedOldGenerationHeapInBytes

The amount of memory currently used by the old generation heap, in bytes.

jvm.mem.usedSurvivorSpaceInBytes

The amount of memory currently used by the survivor space, in bytes.

jvm.mem.usedYoungGenerationHeapInBytes

The amount of memory currently used by the young generation heap, in bytes.

jvm.ThreadsActive

The number of active threads in the JVM.

jvm.ThreadsPeak

The peak number of threads used by the JVM.

merges.currentActive

The number of currently active segment merges.

merges.docsSegmentsMerging

The number of documents across segments currently being merged.

merges.docsSegmentMerges

The number of documents across all merged segments.

merges.mergedSegmentsInBytes

The size of all merged segments, in bytes.

merges.segmentMerges

The number of segment merges.

merges.sizeSegmentsMergingInBytes

The size of the segments currently being merged, in bytes.

merges.totalSegmentMergingInMilliseconds

The time spent on segment merging.

openFD

The number of opened file descriptors associated with the current process, or-1 if not supported.

queriesTotal

The number of queries.

refresh.total

The number of index refreshes.

refresh.totalInMilliseconds

The time spent on index refreshes.

searchFetchCurrentlyRunning

The number of search fetches currently running.

searchFetches

The number of search fetches.

sizeStoreInBytes

The size of the store, in bytes.

threadpool.bulk.Queue

The number of queued threads in the bulk pool.

threadpool.bulkActive

The number of active threads in the bulk pool.

threadpool.bulkRejected

The number of rejected threads in the bulk pool.

threadpool.bulkThreads

The number of threads in the bulk pool.

threadpool.fetchShardStartedQueue

The number of queued threads in the fetch shard started pool.

threadpool.fetchShardStartedRejected

The number of rejected threads in the fetch shard started pool.

threadpool.fetchShardStartedThreads

The number of threads in the fetch shard started pool.

threadpool.fetchShardStoreActive

The number of active threads in the fetch shard store pool.

threadpool.fetchShardStoreQueue

The number of queued threads in the fetch shard store pool.

threadpool.fetchShardStoreRejected

The number of rejected threads in the fetch shard store pool.

threadpool.fetchShardStoreThreads

The number of threads in the fetch shard store pool.

threadpool.flushActive

The number of active threads in the flush queue.

threadpool.flushQueue

The number of queued threads in the flush pool.

threadpool.flushRejected

The number of rejected threads in the flush pool.

threadpool.flushThreads

The number of threads in the flush pool.

threadpool.forceMergeActive

The number of active threads for force merge operations.

threadpool.forceMergeQueue

The number of queued threads for force merge operations.

threadpool.forceMergeRejected

The number of rejected threads for force merge operations.

threadpool.forceMergeThreads

The number of threads for force merge operations.

threadpool.genericActive

The number of active threads in the generic pool.

threadpool.genericQueue

The number of queued threads in the generic pool.

threadpool.genericRejected

The number of rejected threads in the generic pool.

threadpool.genericThreads

The number of threads in the generic pool.

threadpool.getActive

The number of active threads in the get pool.

threadpool.getQueue

The number of queued threads in the get pool.

threadpool.getRejected

The number of rejected threads in the get pool.

threadpool.getThreads

The number of threads in the get pool.

threadpool.indexActive

The number of active threads in the index pool.

threadpool.indexQueue

The number of queued threads in the index pool.

threadpool.indexRejected

The number of rejected threads in the index pool.

threadpool.indexThreads

The number of threads in the index pool.

threadpool.listenerActive

The number of active threads in the listener pool.

threadpool.listenerQueue

The number of queued threads in the listener pool.

threadpool.listenerRejected

The number of rejected threads in the listener pool.

threadpool.listenerThreads

The number of threads in the listener pool.

threadpool.managementActive

The number of active threads in the management pool.

threadpool.managementQueue

The number of queued threads in the management pool.

threadpool.managementRejected

The number of rejected threads in the management pool.

threadpool.managementThreads

The number of threads in the management pool.

threadpool.mergeActive

The number of active threads in the merge pool.

threadpool.mergeQueue

The number of queued threads in the merge pool.

threadpool.mergeRejected

The number of rejected threads in the merge pool.

threadpool.mergeThreads

The number of threads in the merge pool.

threadpool.percolateActive

The number of active threads in the percolate pool.

threadpool.percolateQueue

The number of queued threads in the percolate pool.

threadpool.percolateRejected

The number of rejected threads in the percolate pool.

threadpool.percolateThreads

The number of threads in the percolate pool.

threadpool.refreshActive

The number of active threads in the refresh pool.

threadpool.refreshQueue

The number of queued threads in the refresh pool.

threadpool.refreshRejected

The number of rejected threads in the refresh pool.

threadpool.refreshThreads

The number of threads in the refresh pool.

threadpool.searchActive

The number of active threads in the search pool.

threadpool.searchQueue

The number of queued threads in the search pool.

threadpool.searchRejected

The number of rejected threads in the search pool.

threadpool.searchThreads

The number of threads in the search pool.

threadpool.snapshotActive

The number of active threads in the snapshot pool.

threadpool.snapshotQueue

The number of queued threads in the snapshot pool.

threadpool.snapshotRejected

The number of rejected threads in the snapshot pool.

threadpool.snapshotThreads

The number of threads in the snapshot pool.

threadpool.activeFetchShardStarted

The number of active threads in the fetch shard started pool.

transport.connectionsOpened

The number of connections opened for cluster communication.

transport.packetsReceived

The number of packets received in cluster communication.

transport.packetsReceivedInBytes

The size of data received in cluster communication, in bytes.

transport.packetsSent

The number of packets sent in cluster communication.

transport.packetsSentInBytes

The size of data sent in cluster communication, in bytes.

Elasticsearch common metrics

These attributes are attached to the ElasticsearchCommonSample event type:

Metric

Description

primaries.docsDeleted

The number of documents deleted from the primary shards.

primaries.docsnumber

The number of documents in the primary shards.

primaries.flushesTotal

The number of index flushes to disk from the primary shards since start.

primaries.flushTotalTimeInMilliseconds

The time spent flushing the index to disk from the primary shards.

primaries.get.documentsExist

The number of get requests on primary shards where the document existed.

primaries.get.documentsExistInMilliseconds

The time spent on get requests from the primary shards where the document existed.

primaries.get.documentsMissing

The number of get requests from the primary shards where the document was missing.

primaries.get.documentsMissingInMilliseconds

The time spent on get requests from the primary shards where the document was missing.

primaries.get.requests

The number of get requests from the primary shards.

primaries.get.requestsCurrent

The number of get requests currently running on the primary shards.

primaries.get.requestsInMilliseconds

The time spent on get requests from the primary shards.

primaries.index.docsCurrentlyDeleted

The number of documents currently being deleted from an index on the primary shards.

primaries.index.docsCurrentlyDeletedInMilliseconds

The time spent deleting documents from an index on the primary shards.

primaries.index.docsCurrentlyIndexing

The number of documents currently being indexed to an index on the primary shards.

primaries.index.docsCurrentlyIndexingInMilliseconds

The time spent indexing documents to an index on the primary shards.

primaries.index.docsDeleted

The number of documents deleted from an index on the primary shards.

primaries.index.docsTotal

The number of documents indexed to an index on the primary shards.

primaries.indexRefreshesTotal

The number of index refreshes on the primary shards.

primaries.indexRefreshesTotalInMilliseconds

The time spent on index refreshes on the primary shards.

primaries.merges.current

The number of currently active segment merges on the primary shards.

primaries.merges.docsSegmentsCurrentlyMerged

The number of documents across segments currently being merged on the primary shards.

primaries.merges.docsTotal

The number of documents across all merged segments on the primary shards.

primaries.merges.SegmentsCurrentlyMergedInBytes

The size of the segments currently being merged on the primary shards, in bytes.

primaries.merges.SegmentsTotal

The number of segment merges on the primary shards.

primaries.merges.segmentsTotalInBytes

The size of all merged segments on the primary shards, in bytes.

primaries.merges.segmentsTotalInMilliseconds

The time spent on segment merging on the primary shards.

primaries.queriesInMilliseconds

The time spent querying on the primary shards.

primaries.queriesTotal

The number of queries to the primary shards.

primaries.queryActive

The number of currently active queries on the primary shards.

primaries.queryFetches

The number of query fetches currently running on the primary shards.

primaries.queryFetchesInMilliseconds

The time spent on query fetches on the primary shards.

primaries.queryFetchesTotal

The number of query fetches on the primary shards.

primaries.sizeInBytes

The size of all the primary shards, in bytes.

Elasticsearch index metrics

These attributes are attached to the ElasticsearchIndexSample event type:

Metric

Description

index.docs

The number of documents in the index.

index.docsDeleted

The number of deleted documents in the index.

index.health

The status of the index: red, yellow, or green.

index.primaryShards

The number of primary shards in the index.

index.primaryStoreSizeInBytes

The store size of primary shards in the index.

index.replicaShards

The number of replica shards in the index.

index.storeSizeInBytes

The store size of primary and replica shards in the index, in bytes.

Inventory data

The Elasticsearch integration captures the configuration parameters of the Elasticsearch node, as specified in the YAML config file. It also collects node configuration information from the "_nodes/_local" endpoint. The data is available on the Inventory page, under the config/elasticsearch source. For more about inventory data, see Understand integration data.

Check the source code

This integration is open source software. That means you can browse its source code and send improvements, or create your own fork and build it.

For more help

If you need more help, check out these support and learning resources:

Create issueEdit page
Copyright © 2021 New Relic Inc.