1 line
7.9 KiB
JSON
1 line
7.9 KiB
JSON
{"repo": "azure/cloudopt", "files": {"docs/query_pack/newrelic.md": {"e5c6e53750f379fef3151dc83b504f9b9e01d870": {"t": 1785595778.5709903, "n": 7520, "h": "d159d2a2296575ad", "c": "# New Relic Query Pack\n\nExport VM metrics from New Relic in the canonical cloudopt CSV format.\n\n## Prerequisites\n\n- New Relic Infrastructure agent on target VMs\n- A New Relic User API key with `NerdGraph` access\n- Python `requests` library: `pip install requests`\n\n```bash\nexport NEW_RELIC_API_KEY=\"NRAK-...\"\nexport NEW_RELIC_ACCOUNT_ID=\"<account-id>\"\n```\n\n## Metric Mapping\n\n| cloudopt metric | New Relic metric / NRQL |\n| ----------------------------- | ---------------------------------------------------------------- |\n| `os.cpu.used_percent` | `SystemSample.cpuPercent` |\n| `os.memory.used_percent` | `SystemSample.memoryUsedPercent` |\n| `os.memory.available_mb` | `SystemSample.memoryAvailableMegabytes` |\n| `os.memory.swap_used_percent` | `SystemSample.swapUsedPercent` |\n| `os.disk.read_iops` | `StorageSample.readIoOperationsPerSecond` |\n| `os.disk.write_iops` | `StorageSample.writeIoOperationsPerSecond` |\n| `os.disk.read_mbps` | `StorageSample.readBytesPerSecond` / 1048576 |\n| `os.disk.write_mbps` | `StorageSample.writeBytesPerSecond` / 1048576 |\n| `os.network.receive_mbps` | `NetworkSample.receiveBytesPerSecond` × 8 / 1000000 |\n| `os.network.send_mbps` | `NetworkSample.transmitBytesPerSecond` × 8 / 1000000 |\n| `jvm.heap.used_percent` | `JvmThreadSample.heapMemoryUsed / JvmThreadSample.heapMemoryMax` |\n| `jvm.gc.pause_ms_avg` | `JvmThreadSample.gcConcurrentMarkSweepTime` |\n\n## Export Script\n\n```python\n#!/usr/bin/env python3\n\"\"\"Export New Relic VM metrics to cloudopt canonical CSV format.\"\"\"\n\nfrom __future__ import annotations\n\nimport csv\nimport os\nimport sys\nimport time\nfrom datetime import datetime, timedelta, timezone\nfrom pathlib import Path\n\nimport requests\n\n# ---------------------------------------------------------------------------\n# Configuration\n# ---------------------------------------------------------------------------\n\nNR_API_KEY = os.environ[\"NEW_RELIC_API_KEY\"]\nNR_ACCOUNT_ID = os.environ[\"NEW_RELIC_ACCOUNT_ID\"]\nNR_GRAPHQL_URL = \"https://api.newrelic.com/graphql\"\n\nPERIOD_DAYS = 30\nOUTPUT_PATH = Path(\"monitoring_newrelic.csv\")\nHOSTS_CSV = Path(\"cloudopt_hosts.csv\")\n\nSCHEMA_VERSION = \"1.0\"\nSOURCE_TOOL = \"newrelic\"\n\n# (cloudopt_metric_name, nrql_template, unit)\n# {host} is replaced with the VM name; {since} with the SINCE clause\nNRQL_DEFS: list[tuple[str, str, str]] = [\n (\n \"os.cpu.used_percent\",\n \"SELECT average(cpuPercent) AS avg_value, percentile(cpuPercent, 95) AS p95_value, max(cpuPercent) AS max_value FROM SystemSample WHERE hostname = '{host}' SINCE {days} days AGO\",\n \"percent\",\n ),\n (\n \"os.memory.used_percent\",\n \"SELECT average(memoryUsedPercent) AS avg_value, percentile(memoryUsedPercent, 95) AS p95_value, max(memoryUsedPercent) AS max_value FROM SystemSample WHERE hostname = '{host}' SINCE {days} days AGO\",\n \"percent\",\n ),\n (\n \"os.memory.available_mb\",\n \"SELECT average(memoryAvailableMegabytes) AS avg_value, percentile(memoryAvailableMegabytes, 5) AS p95_value, min(memoryAvailableMegabytes) AS max_value FROM SystemSample WHERE hostname = '{host}' SINCE {days} days AGO\",\n \"MB\",\n ),\n (\n \"os.disk.read_iops\",\n \"SELECT average(readIoOperationsPerSecond) AS avg_value, percentile(readIoOperationsPerSecond, 95) AS p95_value, max(readIoOperationsPerSecond) AS max_value FROM StorageSample WHERE hostname = '{host}' SINCE {days} days AGO\",\n \"iops\",\n ),\n (\n \"os.disk.write_iops\",\n \"SELECT average(writeIoOperationsPerSecond) AS avg_value, percentile(writeIoOperationsPerSecond, 95) AS p95_value, max(writeIoOperationsPerSecond) AS max_value FROM StorageSample WHERE hostname = '{host}' SINCE {days} days AGO\",\n \"iops\",\n ),\n]\n\n_HEADERS = {\n \"Content-Type\": \"application/json\",\n \"API-Key\": NR_API_KEY,\n}\n\n\ndef _load_hostnames(path: Path) -> list[str]:\n if not path.exists():\n print(f\"ERROR: Host list not found: {path}\", file=sys.stderr)\n sys.exit(1)\n with path.open(encoding=\"utf-8\") as f:\n reader = csv.DictReader(f)\n return [row[\"vm_name\"] for row in reader if row.get(\"vm_name\")]\n\n\ndef _run_nrql(nrql: str) -> dict[str, float | None]:\n \"\"\"Execute a NRQL query via NerdGraph and return result columns.\"\"\"\n gql = {\n \"query\": f\"\"\"\n {{\n actor {{\n account(id: {NR_ACCOUNT_ID}) {{\n nrql(query: \"{nrql.replace('\"', '\\\\\"')}\") {{\n results\n }}\n }}\n }}\n }}\n \"\"\"\n }\n resp = requests.post(NR_GRAPHQL_URL, headers=_HEADERS, json=gql, timeout=30)\n resp.raise_for_status()\n data = resp.json()\n results = (\n data.get(\"data\", {})\n .get(\"actor\", {})\n .get(\"account\", {})\n .get(\"nrql\", {})\n .get(\"results\", [])\n )\n if not results:\n return {\"avg_value\": None, \"p95_value\": None, \"max_value\": None}\n r = results[0]\n return {\n \"avg_value\": r.get(\"avg_value\"),\n \"p95_value\": r.get(\"p95_value\"),\n \"max_value\": r.get(\"max_value\"),\n }\n\n\ndef main() -> None:\n now = datetime.now(timezone.utc)\n period_end = now.replace(minute=0, second=0, microsecond=0)\n period_end_utc = period_end.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n\n hostnames = _load_hostnames(HOSTS_CSV)\n print(f\"Exporting {len(hostnames)} host(s) over {PERIOD_DAYS} days…\")\n\n rows: list[dict] = []\n for host in hostnames:\n print(f\" {host}\")\n for metric_name, nrql_template, unit in NRQL_DEFS:\n nrql = nrql_template.replace(\"{host}\", host).replace(\"{days}\", str(PERIOD_DAYS))\n try:\n result = _run_nrql(nrql)\n except Exception as exc:\n print(f\" WARNING: {metric_name}: {exc}\", file=sys.stderr)\n continue\n if result[\"avg_value\"] is None:\n continue\n rows.append({\n \"schema_version\": SCHEMA_VERSION,\n \"source_tool\": SOURCE_TOOL,\n \"hostname\": host,\n \"metric_name\": metric_name,\n \"period_days\": PERIOD_DAYS,\n \"period_end_utc\": period_end_utc,\n \"avg_value\": round(result[\"avg_value\"], 4),\n \"p95_value\": round(result[\"p95_value\"], 4) if result[\"p95_value\"] is not None else \"\",\n \"max_value\": round(result[\"max_value\"], 4) if result[\"max_value\"] is not None else \"\",\n \"unit\": unit,\n })\n time.sleep(0.05) # New Relic rate limit: 3,000 req/min\n\n with OUTPUT_PATH.open(\"w\", newline=\"\", encoding=\"utf-8\") as f:\n writer = csv.DictWriter(f, fieldnames=[\n \"schema_version\", \"source_tool\", \"hostname\", \"metric_name\",\n \"period_days\", \"period_end_utc\", \"avg_value\", \"p95_value\", \"max_value\", \"unit\",\n ])\n writer.writeheader()\n writer.writerows(rows)\n\n print(f\"\\nWrote {len(rows)} rows to {OUTPUT_PATH}\")\n print(f\"\\nNext step:\\n cloudopt analyze --from cloudopt_export_<ts>.json --monitoring {OUTPUT_PATH}\")\n\n\nif __name__ == \"__main__\":\n main()\n```\n"}}}} |