Algorithmia MLOps integration

What is MLOps?

MLOps stands for machine-learning operations. As more companies invest in artificial intelligence and machine learning, there's a gap in understanding between the data science teams developing machine-learning models and the DevOps teams operating the applications that power those models.

MLOps provides a tool for monitoring and observing the performance and effectiveness of machine-learning models in a production environment. This increases the possibilities for collaboration between data science and DevOps teams, feeding into a continuous process of development, testing, and operational monitoring.

The Algorithmia integration

Algorithmia Insights provides tools for deploying your machine-learning models into production. By integrating, Algorithmia's integration with New Relic, you'll be able to instrument, analyze, troubleshoot, and optimize your machine-learning performance across your entire system. By rigorously observing your capabilities, you'll be able to react quickly to changes in the model's input or output and the relationship between the two.

Send your model performance metrics from Algorithmia Insights to New Relic and you'll have real-time monitoring for your algorithms. You'll explore your metrics data with user-friendly charts and learn the state of your algorithms at a glance for faster and more efficient troubleshooting.

Integrate Algorithmia with New Relic

First, Algorithmia uses a Kafka topic to stream Insights. Your machine-learning algorithm's performance metrics. Then, the New Relic connector (another algorithm) transforms the Kafka topic into a metrics data payload for a specific New Relic account.

Algorithmia uses Kafka and Event Flows to send data to New Relic.

With Algorithmia’s Event Flows, when a new message is set to your Kafka topic, your configured New Relic connector algorithm gets called. The connector transforms your metrics and sends them to your New Relic account.

Connect your Algorithmia data to New Relic

By integrating Incident Intelligence with your Algorithmia machine-learning models, you can monitor your machine learning model performance.

Start monitoring your Algorithmia event flows with New Relic.

Get your API key: From one.newrelic.com the account menu, click API keys and then create a user key for your account with a meaningful name. Make note of this name for later. For more on API keys, see our docs.
Create a dashboard: From one.newrelic.com go to Dashboards, then click the Import dashboards button. Copy and paste the JSON code into the Paste your JSON field code.

Update the YOUR_ACCOUNT_ID values with your account ID.

{
"name": "Algorithmia Dashboard for Default Metrics",
"description": null,
"permissions": "PUBLIC_READ_WRITE",
"pages": [
{
"name": "Algorithmia Dashboard for Default Metrics",
"description": null,
"widgets": [
{
  "visualization": {
    "id": "viz.line"
  },
  "layout": {
    "column": 1,
    "row": 1,
    "height": 3,
    "width": 4
  },
  "title": "Runtime Duration by Algorithm",
  "rawConfiguration": {
    "legend": {
      "enabled": true
    },
    "nrqlQueries": [
      {
        "accountId": YOUR_ACCOUNT_ID,
        "query": "SELECT average(algorithmia.duration_milliseconds) FROM Metric TIMESERIES FACET `algorithm_name` LIMIT 10 SINCE 1800 seconds ago"
      }
    ],
    "yAxisLeft": {
      "zero": true
    }
  },
  "linkedEntityGuids": null
},
{
  "visualization": {
    "id": "viz.line"
  },
  "layout": {
    "column": 5,
    "row": 1,
    "height": 3,
    "width": 4
  },
  "title": "Throughput by Algorithm",
  "rawConfiguration": {
    "legend": {
      "enabled": true
    },
    "nrqlQueries": [
      {
        "accountId": YOUR_ACCOUNT_ID,
        "query": "SELECT count(algorithmia.duration_milliseconds) FROM Metric TIMESERIES FACET `algorithm_name` LIMIT 10 SINCE 1800 seconds ago"
      }
    ],
    "yAxisLeft": {
      "zero": true
    }
  },
  "linkedEntityGuids": null
}
]
}
]
}

Configure Algorithmia Insights for New Relic: Use Algorithmia's docs for how to configure Algorithmia Insights for New Relic.
Create the New Relic connector algorithm: Use Python 3.8 to create a connector algorithm. If you're new to writing code to generate algorithms, see Algorithmia's getting started guide.

import Algorithmia
import json
from datetime import datetime
from newrelic_telemetry_sdk import GaugeMetric, MetricClient

client = Algorithmia.client()
metric_client = MetricClient(os.environ["newrelic_api_key"])


def convert_str_timestamp_to_epoch(str_time):
obj_time = datetime.strptime(str_time, "%Y-%m-%dT%H:%M:%S.%f")
return int(obj_time.timestamp() * 1000)

def get_operational_metrics(payload):
ALGORITHM_TAGS = {
"algorithm_version",
"request_id",
"time",
"algorithm_name",
"session_id",
"algorithm_owner"
}
inference_metrics = {
key: payload[key] for key in payload.keys() ^ ALGORITHM_TAGS
}
return inference_metrics

def send_to_newrelic(inference_metrics, insights_payload):
newrelic_metrics = []
for key, value in inference_metrics.items():
name = "algorithmia." + key
epoch_time = convert_str_timestamp_to_epoch(insights_payload["time"])
tags = {
    "algorithm_name": insights_payload["algorithm_name"],
    "algorithm_version": insights_payload["algorithm_version"],
    "algorithm_owner": insights_payload["algorithm_owner"],
    "request_id": insights_payload["request_id"],
    "session_id": insights_payload["session_id"],
}

newrelic_metrics.append(GaugeMetric(
    name=name, value=value, tags=tags, end_time_ms=epoch_time
))

response = metric_client.send_batch(newrelic_metrics)
response.raise_for_status()


def apply(input):
insights_payload = input
inference_metrics = get_operational_metrics(insights_payload)
send_to_newrelic(inference_metrics, insights_payload)
return None

Include these dependencies:

algorithmia>=1.0.0,<2.0
newrelic_telemetry_sdk==0.4.2

Once your algorithm build finishes, you can test it with this sample payload to make sure it runs successfully. Your output should look something like this.

{
"risk_score": 0.2,
"duration_milliseconds": 8,
"algorithm_version": "1.0.6",
"session_id": "rses-f28bb94a-5556-4aeb-a6d2-89493626bf4f",
"time": "2021-02-20T00:21:54.867231",
"algorithm_name": "credit_card_approval",
"request_id": "req-9f5345b4-a1cd-431c-a43a-bd2a06f4a6f4",
"algorithm_owner": "asli"
}

Configure with your API key: Add your New Relic API key to the Algorithmia secret store.
Set up Algorithmia Event Flows with New Relic: See Algorithmia's documentation on setting up your connector algorithm to send event-based machine learning flows to New Relic.

Monitor your machine learning models [#monitor]

Follow these steps to get the most of observing your machine-learning data in New Relic.

Get your API key: From one.newrelic.com the account menu, click API keys and then create a user key for your account with a meaningful name. Make note of this name for later. For more on API keys, see our docs.
Create a dashboard: From one.newrelic.com go to Dashboards, then click the Import dashboards button. Copy and paste the JSON code into the Paste your JSON field code.