{"repo": "azure/cloudopt", "files": {"docs/query_pack/newrelic.md": {"e5c6e53750f379fef3151dc83b504f9b9e01d870": {"t": 1785595778.5709903, "n": 7520, "h": "d159d2a2296575ad", "c": "# New Relic Query Pack\n\nExport VM metrics from New Relic in the canonical cloudopt CSV format.\n\n## Prerequisites\n\n- New Relic Infrastructure agent on target VMs\n- A New Relic User API key with `NerdGraph` access\n- Python `requests` library: `pip install requests`\n\n```bash\nexport NEW_RELIC_API_KEY=\"NRAK-...\"\nexport NEW_RELIC_ACCOUNT_ID=\"<account-id>\"\n```\n\n## Metric Mapping\n\n| cloudopt metric               | New Relic metric / NRQL                                          |\n| ----------------------------- | ---------------------------------------------------------------- |\n| `os.cpu.used_percent`         | `SystemSample.cpuPercent`                                        |\n| `os.memory.used_percent`      | `SystemSample.memoryUsedPercent`                                 |\n| `os.memory.available_mb`      | `SystemSample.memoryAvailableMegabytes`                          |\n| `os.memory.swap_used_percent` | `SystemSample.swapUsedPercent`                                   |\n| `os.disk.read_iops`           | `StorageSample.readIoOperationsPerSecond`                        |\n| `os.disk.write_iops`          | `StorageSample.writeIoOperationsPerSecond`                       |\n| `os.disk.read_mbps`           | `StorageSample.readBytesPerSecond` / 1048576                     |\n| `os.disk.write_mbps`          | `StorageSample.writeBytesPerSecond` / 1048576                    |\n| `os.network.receive_mbps`     | `NetworkSample.receiveBytesPerSecond` × 8 / 1000000              |\n| `os.network.send_mbps`        | `NetworkSample.transmitBytesPerSecond` × 8 / 1000000             |\n| `jvm.heap.used_percent`       | `JvmThreadSample.heapMemoryUsed / JvmThreadSample.heapMemoryMax` |\n| `jvm.gc.pause_ms_avg`         | `JvmThreadSample.gcConcurrentMarkSweepTime`                      |\n\n## Export Script\n\n```python\n#!/usr/bin/env python3\n\"\"\"Export New Relic VM metrics to cloudopt canonical CSV format.\"\"\"\n\nfrom __future__ import annotations\n\nimport csv\nimport os\nimport sys\nimport time\nfrom datetime import datetime, timedelta, timezone\nfrom pathlib import Path\n\nimport requests\n\n# ---------------------------------------------------------------------------\n# Configuration\n# ---------------------------------------------------------------------------\n\nNR_API_KEY = os.environ[\"NEW_RELIC_API_KEY\"]\nNR_ACCOUNT_ID = os.environ[\"NEW_RELIC_ACCOUNT_ID\"]\nNR_GRAPHQL_URL = \"https://api.newrelic.com/graphql\"\n\nPERIOD_DAYS = 30\nOUTPUT_PATH = Path(\"monitoring_newrelic.csv\")\nHOSTS_CSV = Path(\"cloudopt_hosts.csv\")\n\nSCHEMA_VERSION = \"1.0\"\nSOURCE_TOOL = \"newrelic\"\n\n# (cloudopt_metric_name, nrql_template, unit)\n# {host} is replaced with the VM name; {since} with the SINCE clause\nNRQL_DEFS: list[tuple[str, str, str]] = [\n    (\n        \"os.cpu.used_percent\",\n        \"SELECT average(cpuPercent) AS avg_value, percentile(cpuPercent, 95) AS p95_value, max(cpuPercent) AS max_value FROM SystemSample WHERE hostname = '{host}' SINCE {days} days AGO\",\n        \"percent\",\n    ),\n    (\n        \"os.memory.used_percent\",\n        \"SELECT average(memoryUsedPercent) AS avg_value, percentile(memoryUsedPercent, 95) AS p95_value, max(memoryUsedPercent) AS max_value FROM SystemSample WHERE hostname = '{host}' SINCE {days} days AGO\",\n        \"percent\",\n    ),\n    (\n        \"os.memory.available_mb\",\n        \"SELECT average(memoryAvailableMegabytes) AS avg_value, percentile(memoryAvailableMegabytes, 5) AS p95_value, min(memoryAvailableMegabytes) AS max_value FROM SystemSample WHERE hostname = '{host}' SINCE {days} days AGO\",\n        \"MB\",\n    ),\n    (\n        \"os.disk.read_iops\",\n        \"SELECT average(readIoOperationsPerSecond) AS avg_value, percentile(readIoOperationsPerSecond, 95) AS p95_value, max(readIoOperationsPerSecond) AS max_value FROM StorageSample WHERE hostname = '{host}' SINCE {days} days AGO\",\n        \"iops\",\n    ),\n    (\n        \"os.disk.write_iops\",\n        \"SELECT average(writeIoOperationsPerSecond) AS avg_value, percentile(writeIoOperationsPerSecond, 95) AS p95_value, max(writeIoOperationsPerSecond) AS max_value FROM StorageSample WHERE hostname = '{host}' SINCE {days} days AGO\",\n        \"iops\",\n    ),\n]\n\n_HEADERS = {\n    \"Content-Type\": \"application/json\",\n    \"API-Key\": NR_API_KEY,\n}\n\n\ndef _load_hostnames(path: Path) -> list[str]:\n    if not path.exists():\n        print(f\"ERROR: Host list not found: {path}\", file=sys.stderr)\n        sys.exit(1)\n    with path.open(encoding=\"utf-8\") as f:\n        reader = csv.DictReader(f)\n        return [row[\"vm_name\"] for row in reader if row.get(\"vm_name\")]\n\n\ndef _run_nrql(nrql: str) -> dict[str, float | None]:\n    \"\"\"Execute a NRQL query via NerdGraph and return result columns.\"\"\"\n    gql = {\n        \"query\": f\"\"\"\n            {{\n              actor {{\n                account(id: {NR_ACCOUNT_ID}) {{\n                  nrql(query: \"{nrql.replace('\"', '\\\\\"')}\") {{\n                    results\n                  }}\n                }}\n              }}\n            }}\n        \"\"\"\n    }\n    resp = requests.post(NR_GRAPHQL_URL, headers=_HEADERS, json=gql, timeout=30)\n    resp.raise_for_status()\n    data = resp.json()\n    results = (\n        data.get(\"data\", {})\n        .get(\"actor\", {})\n        .get(\"account\", {})\n        .get(\"nrql\", {})\n        .get(\"results\", [])\n    )\n    if not results:\n        return {\"avg_value\": None, \"p95_value\": None, \"max_value\": None}\n    r = results[0]\n    return {\n        \"avg_value\": r.get(\"avg_value\"),\n        \"p95_value\": r.get(\"p95_value\"),\n        \"max_value\": r.get(\"max_value\"),\n    }\n\n\ndef main() -> None:\n    now = datetime.now(timezone.utc)\n    period_end = now.replace(minute=0, second=0, microsecond=0)\n    period_end_utc = period_end.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n\n    hostnames = _load_hostnames(HOSTS_CSV)\n    print(f\"Exporting {len(hostnames)} host(s) over {PERIOD_DAYS} days…\")\n\n    rows: list[dict] = []\n    for host in hostnames:\n        print(f\"  {host}\")\n        for metric_name, nrql_template, unit in NRQL_DEFS:\n            nrql = nrql_template.replace(\"{host}\", host).replace(\"{days}\", str(PERIOD_DAYS))\n            try:\n                result = _run_nrql(nrql)\n            except Exception as exc:\n                print(f\"    WARNING: {metric_name}: {exc}\", file=sys.stderr)\n                continue\n            if result[\"avg_value\"] is None:\n                continue\n            rows.append({\n                \"schema_version\": SCHEMA_VERSION,\n                \"source_tool\": SOURCE_TOOL,\n                \"hostname\": host,\n                \"metric_name\": metric_name,\n                \"period_days\": PERIOD_DAYS,\n                \"period_end_utc\": period_end_utc,\n                \"avg_value\": round(result[\"avg_value\"], 4),\n                \"p95_value\": round(result[\"p95_value\"], 4) if result[\"p95_value\"] is not None else \"\",\n                \"max_value\": round(result[\"max_value\"], 4) if result[\"max_value\"] is not None else \"\",\n                \"unit\": unit,\n            })\n        time.sleep(0.05)  # New Relic rate limit: 3,000 req/min\n\n    with OUTPUT_PATH.open(\"w\", newline=\"\", encoding=\"utf-8\") as f:\n        writer = csv.DictWriter(f, fieldnames=[\n            \"schema_version\", \"source_tool\", \"hostname\", \"metric_name\",\n            \"period_days\", \"period_end_utc\", \"avg_value\", \"p95_value\", \"max_value\", \"unit\",\n        ])\n        writer.writeheader()\n        writer.writerows(rows)\n\n    print(f\"\\nWrote {len(rows)} rows to {OUTPUT_PATH}\")\n    print(f\"\\nNext step:\\n  cloudopt analyze --from cloudopt_export_<ts>.json --monitoring {OUTPUT_PATH}\")\n\n\nif __name__ == \"__main__\":\n    main()\n```\n"}}}}