{
  "tags": [],
  "title": "Cluster cost & utilization metrics",
  "widgets": [
    {
      "layout": {
        "x": 0,
        "y": 0,
        "width": 12,
        "height": 1
      },
      "definition": {
        "type": "note",
        "content": "This dashboard shows monthly cost estimates for the cluster, based on **current** CPU, RAM and storage provisioned.",
        "tick_pos": "50%",
        "font_size": "14",
        "show_tick": false,
        "tick_edge": "left",
        "text_align": "left",
        "has_padding": true,
        "vertical_align": "top",
        "background_color": "white"
      }
    },
    {
      "layout": {
        "x": 0,
        "y": 1,
        "width": 3,
        "height": 1
      },
      "definition": {
        "type": "note",
        "content": "**CPU Cost**\n\nUnsupported panel (singlestat)\n\n- Cost multipliers $costpcpu/$costcpu/$costDiscount cannot appear in Datadog formulas (only query names and numeric literals are allowed, not template variables as scalars)\n- Preemptible vs non-preemptible conditional split via kube_node_labels info-join + `or` has no Datadog formula equivalent\n\nOriginal PromQL:\n```\nA: sum(\n (\n (\n sum(kube_node_status_capacity_cpu_cores) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible=\"true\"}\n ) * $costpcpu\n )\n or\n (\n (\n sum(kube_node_status_capacity_cpu_cores) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible!=\"true\"}\n ) * ($costcpu - ($costcpu / 100 * $costDiscount))\n )\n)\n```",
        "tick_pos": "50%",
        "font_size": "14",
        "show_tick": false,
        "tick_edge": "left",
        "text_align": "left",
        "has_padding": true,
        "vertical_align": "top",
        "background_color": "white"
      }
    },
    {
      "layout": {
        "x": 3,
        "y": 1,
        "width": 3,
        "height": 1
      },
      "definition": {
        "type": "note",
        "content": "**RAM Cost**\n\nUnsupported panel (singlestat)\n\n- Cost multipliers $costpram/$costram/$costDiscount cannot appear in Datadog formulas (only query names and numeric literals allowed)\n- Preemptible vs non-preemptible conditional split via info-join + `or` unsupported\n\nOriginal PromQL:\n```\nA: sum(\n (\n (\n sum(kube_node_status_capacity_memory_bytes) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible=\"true\"}\n ) /1024/1024/1024 * $costpram\n )\n or\n (\n (\n sum(kube_node_status_capacity_memory_bytes) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible!=\"true\"}\n ) /1024/1024/1024 * ($costram - ($costram / 100 * $costDiscount))\n)\n)\n```",
        "tick_pos": "50%",
        "font_size": "14",
        "show_tick": false,
        "tick_edge": "left",
        "text_align": "left",
        "has_padding": true,
        "vertical_align": "top",
        "background_color": "white"
      }
    },
    {
      "layout": {
        "x": 6,
        "y": 1,
        "width": 3,
        "height": 1
      },
      "definition": {
        "type": "note",
        "content": "**Storage Cost (Cluster and PVC)**\n\nUnsupported panel (singlestat)\n\n- Cost multipliers $costStorageSSD/$costStorageStandard cannot be used as scalars in Datadog formulas\n- Complex `or` fallback and group_right storageclass joins have no Datadog equivalent\n\nOriginal PromQL:\n```\nA: sum (\n sum(kube_persistentvolumeclaim_info{storageclass=~\".*ssd.*\"}) by (persistentvolumeclaim, namespace, storageclass)\n + on (persistentvolumeclaim, namespace) group_right(storageclass)\n sum(kube_persistentvolumeclaim_resource_requests_storage_bytes) by (persistentvolumeclaim, namespace) or up * 0\n) / 1024 / 1024 /1024 * $costStorageSSD\n\n+\n\nsum (\n sum(kube_persistentvolumeclaim_info{storageclass!~\".*ssd.*\"}) by (persistentvolumeclaim, namespace, storageclass)\n + on (persistentvolumeclaim, namespace) group_right(storageclass)\n sum(kube_persistentvolumeclaim_resource_requests_storage_bytes) by (persistentvolumeclaim, namespace) or up * 0\n) / 1024 / 1024 /1024 * $costStorageStandard\n\n+ \n\nsum(container_fs_limit_bytes{id=\"/\"}) / 1024 / 1024 / 1024 * 1.03 * $costStorageStandard\n```",
        "tick_pos": "50%",
        "font_size": "14",
        "show_tick": false,
        "tick_edge": "left",
        "text_align": "left",
        "has_padding": true,
        "vertical_align": "top",
        "background_color": "white"
      }
    },
    {
      "layout": {
        "x": 9,
        "y": 1,
        "width": 3,
        "height": 1
      },
      "definition": {
        "type": "note",
        "content": "**Network Egress Cost**\n\nUnsupported panel (singlestat)\n\n- Cost multiplier $costEgress cannot be used as a scalar in a Datadog formula (only query names and numeric literals allowed)\n\nOriginal PromQL:\n```\nA: SUM(rate(node_network_transmit_bytes_total{device=\"eth0\"}[60m]) / 1024 / 1024 / 1024 ) * (60 * 60 * 24 * 30) * $costEgress\n```",
        "tick_pos": "50%",
        "font_size": "14",
        "show_tick": false,
        "tick_edge": "left",
        "text_align": "left",
        "has_padding": true,
        "vertical_align": "top",
        "background_color": "white"
      }
    },
    {
      "layout": {
        "x": 0,
        "y": 2,
        "width": 1,
        "height": 1
      },
      "definition": {
        "type": "query_value",
        "title": "CPU Utilization",
        "requests": [
          {
            "queries": [
              {
                "name": "query1",
                "query": "avg:system.cpu.idle{*}",
                "aggregator": "last",
                "data_source": "metrics"
              }
            ],
            "formulas": [
              {
                "formula": "100 - query1"
              }
            ],
            "response_format": "scalar",
            "conditional_formats": [
              {
                "value": 30,
                "palette": "white_on_gray",
                "comparator": ">="
              },
              {
                "value": 80,
                "palette": "white_on_gray",
                "comparator": ">="
              }
            ]
          }
        ],
        "autoscale": true,
        "precision": 2,
        "text_align": "center",
        "custom_unit": "%"
      }
    },
    {
      "layout": {
        "x": 1,
        "y": 2,
        "width": 2,
        "height": 1
      },
      "definition": {
        "type": "query_value",
        "title": "CPU Requests",
        "requests": [
          {
            "queries": [
              {
                "name": "query1",
                "query": "sum:kubernetes_state.container.cpu_requested{*}",
                "aggregator": "last",
                "data_source": "metrics"
              },
              {
                "name": "query2",
                "query": "sum:kubernetes_state.node.cpu_allocatable{*}",
                "aggregator": "last",
                "data_source": "metrics"
              }
            ],
            "formulas": [
              {
                "formula": "query1 / query2 * 100"
              }
            ],
            "response_format": "scalar",
            "conditional_formats": [
              {
                "value": 30,
                "palette": "white_on_gray",
                "comparator": ">="
              },
              {
                "value": 80,
                "palette": "white_on_gray",
                "comparator": ">="
              }
            ]
          }
        ],
        "autoscale": true,
        "precision": 2,
        "text_align": "center",
        "custom_unit": "%"
      }
    },
    {
      "layout": {
        "x": 3,
        "y": 2,
        "width": 1,
        "height": 1
      },
      "definition": {
        "type": "query_value",
        "title": "RAM Utilization",
        "requests": [
          {
            "queries": [
              {
                "name": "query1",
                "query": "sum:kubernetes.memory.usage{*}",
                "aggregator": "last",
                "data_source": "metrics"
              },
              {
                "name": "query2",
                "query": "sum:kubernetes_state.node.memory_allocatable{*}",
                "aggregator": "last",
                "data_source": "metrics"
              }
            ],
            "formulas": [
              {
                "formula": "query1 / query2 * 100"
              }
            ],
            "response_format": "scalar",
            "conditional_formats": [
              {
                "value": 30,
                "palette": "white_on_gray",
                "comparator": ">="
              },
              {
                "value": 80,
                "palette": "white_on_gray",
                "comparator": ">="
              }
            ]
          }
        ],
        "autoscale": true,
        "precision": 2,
        "text_align": "center",
        "custom_unit": "%"
      }
    },
    {
      "layout": {
        "x": 4,
        "y": 2,
        "width": 2,
        "height": 1
      },
      "definition": {
        "type": "query_value",
        "title": "RAM Requests",
        "requests": [
          {
            "queries": [
              {
                "name": "query1",
                "query": "sum:kubernetes_state.container.memory_requested{*}",
                "aggregator": "last",
                "data_source": "metrics"
              },
              {
                "name": "query2",
                "query": "sum:kubernetes_state.node.memory_allocatable{*}",
                "aggregator": "last",
                "data_source": "metrics"
              }
            ],
            "formulas": [
              {
                "formula": "query1 / query2 * 100"
              }
            ],
            "response_format": "scalar",
            "conditional_formats": [
              {
                "value": 30,
                "palette": "white_on_gray",
                "comparator": ">="
              },
              {
                "value": 80,
                "palette": "white_on_gray",
                "comparator": ">="
              }
            ]
          }
        ],
        "autoscale": true,
        "precision": 2,
        "text_align": "center",
        "custom_unit": "%"
      }
    },
    {
      "layout": {
        "x": 6,
        "y": 2,
        "width": 1,
        "height": 1
      },
      "definition": {
        "type": "query_value",
        "title": "Storage Utilization",
        "requests": [
          {
            "queries": [
              {
                "name": "query1",
                "query": "sum:kubernetes.kubelet.volume.stats.used_bytes{*}",
                "aggregator": "last",
                "data_source": "metrics"
              },
              {
                "name": "query2",
                "query": "sum:kubernetes.filesystem.usage{*}",
                "aggregator": "last",
                "data_source": "metrics"
              },
              {
                "name": "query3",
                "query": "sum:kubernetes_state.persistentvolumeclaim.request_storage{*}",
                "aggregator": "last",
                "data_source": "metrics"
              },
              {
                "name": "query4",
                "query": "sum:kubernetes.filesystem.usage_pct{*}",
                "aggregator": "last",
                "data_source": "metrics"
              }
            ],
            "formulas": [
              {
                "formula": "(query1 + query2) / (query3 + query4) * 100"
              }
            ],
            "response_format": "scalar",
            "conditional_formats": [
              {
                "value": 30,
                "palette": "white_on_gray",
                "comparator": ">="
              },
              {
                "value": 80,
                "palette": "white_on_gray",
                "comparator": ">="
              }
            ]
          }
        ],
        "autoscale": true,
        "precision": 2,
        "text_align": "center",
        "custom_unit": "%"
      }
    },
    {
      "layout": {
        "x": 7,
        "y": 2,
        "width": 2,
        "height": 1
      },
      "definition": {
        "type": "query_value",
        "title": "SSD Utilization",
        "requests": [
          {
            "queries": [
              {
                "name": "query1",
                "query": "sum:kubernetes.kubelet.volume.stats.used_bytes{*}",
                "aggregator": "last",
                "data_source": "metrics"
              },
              {
                "name": "query2",
                "query": "sum:kubernetes_state.persistentvolumeclaim.request_storage{*}",
                "aggregator": "last",
                "data_source": "metrics"
              }
            ],
            "formulas": [
              {
                "formula": "query1 / query2 * 100"
              }
            ],
            "response_format": "scalar",
            "conditional_formats": [
              {
                "value": 30,
                "palette": "white_on_gray",
                "comparator": ">="
              },
              {
                "value": 80,
                "palette": "white_on_gray",
                "comparator": ">="
              }
            ]
          }
        ],
        "autoscale": true,
        "precision": 2,
        "text_align": "center",
        "custom_unit": "%"
      }
    },
    {
      "layout": {
        "x": 9,
        "y": 2,
        "width": 3,
        "height": 1
      },
      "definition": {
        "type": "note",
        "content": "**Total Monthly Cost**\n\nUnsupported panel (singlestat)\n\n- Combines CPU, RAM, Storage and Network cost sub-expressions, all of which multiply by template variables ($costcpu, $costpcpu, $costram, $costpram, $costStorageSSD, $costStorageStandard, $costEgress, $costDiscount); Datadog formulas cannot use template variables as numeric literals\n\nOriginal PromQL:\n```\nA: # CPU\nsum(\n (\n (\n sum(kube_node_status_capacity_cpu_cores) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible=\"true\"}\n ) * $costpcpu\n )\n or\n (\n (\n sum(kube_node_status_capacity_cpu_cores) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible!=\"true\"}\n ) * ($costcpu - ($costcpu / 100 * $costDiscount))\n )\n) \n\n+ \n\n# Storage\nsum (\n sum(kube_persistentvolumeclaim_info{storageclass=~\".*ssd.*\"}) by (persistentvolumeclaim, namespace, storageclass)\n + on (persistentvolumeclaim, namespace) group_right(storageclass)\n sum(kube_persistentvolumeclaim_resource_requests_storage_bytes) by (persistentvolumeclaim, namespace) or up * 0\n) / 1024 / 1024 /1024 * $costStorageSSD\n\n+\n\nsum (\n sum(kube_persistentvolumeclaim_info{storageclass!~\".*ssd.*\"}) by (persistentvolumeclaim, namespace, storageclass)\n + on (persistentvolumeclaim, namespace) group_right(storageclass)\n sum(kube_persistentvolumeclaim_resource_requests_storage_bytes) by (persistentvolumeclaim, namespace) or up * 0\n) / 1024 / 1024 /1024 * $costStorageStandard\n\n+ \n\nsum(container_fs_limit_bytes{id=\"/\"}) / 1024 / 1024 / 1024 * 1.03 * $costStorageStandard \n\n+\n\n# END STORAGE\n# RAM \nsum(\n (\n (\n sum(kube_node_status_capacity_memory_bytes) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible=\"true\"}\n ) /1024/1024/1024 * $costpram\n )\n or\n (\n (\n sum(kube_node_status_capacity_memory_bytes) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible!=\"true\"}\n ) /1024/1024/1024 * ($costram - ($costram / 100 * $costDiscount))\n)\n)\n\n+\n\n#Network \nSUM(rate(node_network_transmit_bytes_total{device=\"eth0\"}[60m]) / 1024 / 1024 / 1024 ) * (60 * 60 * 24 * 30) * $costEgress\n```",
        "tick_pos": "50%",
        "font_size": "14",
        "show_tick": false,
        "tick_edge": "left",
        "text_align": "left",
        "has_padding": true,
        "vertical_align": "top",
        "background_color": "white"
      }
    },
    {
      "layout": {
        "x": 0,
        "y": 3,
        "width": 6,
        "height": 2
      },
      "definition": {
        "type": "note",
        "content": "**Total monthly cost**\n\nUnsupported panel (graph)\n\n- Same as Total Monthly Cost stat panel: relies on cost template variables as numeric multipliers, unsupported in Datadog formulas\n\nOriginal PromQL:\n```\nA: # CPU\nsum(\n (\n (\n sum(kube_node_status_capacity_cpu_cores) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible=\"true\"}\n ) * $costpcpu\n )\n or\n (\n (\n sum(kube_node_status_capacity_cpu_cores) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible!=\"true\"}\n ) * ($costcpu - ($costcpu / 100 * $costDiscount))\n )\n) \n\n+ \n\n# Storage\nsum (\n sum(kube_persistentvolumeclaim_info{storageclass=~\".*ssd.*\"}) by (persistentvolumeclaim, namespace, storageclass)\n + on (persistentvolumeclaim, namespace) group_right(storageclass)\n sum(kube_persistentvolumeclaim_resource_requests_storage_bytes) by (persistentvolumeclaim, namespace) or up * 0\n) / 1024 / 1024 /1024 * $costStorageSSD\n\n+\n\nsum (\n sum(kube_persistentvolumeclaim_info{storageclass!~\".*ssd.*\"}) by (persistentvolumeclaim, namespace, storageclass)\n + on (persistentvolumeclaim, namespace) group_right(storageclass)\n sum(kube_persistentvolumeclaim_resource_requests_storage_bytes) by (persistentvolumeclaim, namespace) or up * 0\n) / 1024 / 1024 /1024 * $costStorageStandard\n\n+ \n\nsum(container_fs_limit_bytes{id=\"/\"}) / 1024 / 1024 / 1024 * 1.03 * $costStorageStandard \n\n+\n\n# END STORAGE\n# RAM \nsum(\n (\n (\n sum(kube_node_status_capacity_memory_bytes) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible=\"true\"}\n ) /1024/1024/1024 * $costpram\n )\n or\n (\n (\n sum(kube_node_status_capacity_memory_bytes) by (node)\n * on (node) group_left (label_cloud_google_com_gke_preemptible)\n kube_node_labels{label_cloud_google_com_gke_preemptible!=\"true\"}\n ) /1024/1024/1024 * ($costram - ($costram / 100 * $costDiscount))\n)\n) \n\n+\n\n#Network \nSUM(rate(node_network_transmit_bytes_total{device=\"eth0\"}[60m]) / 1024 / 1024 / 1024 ) * (60 * 60 * 24 * 30) * $costEgress\n```",
        "tick_pos": "50%",
        "font_size": "14",
        "show_tick": false,
        "tick_edge": "left",
        "text_align": "left",
        "has_padding": true,
        "vertical_align": "top",
        "background_color": "white"
      }
    },
    {
      "layout": {
        "x": 6,
        "y": 3,
        "width": 6,
        "height": 2
      },
      "definition": {
        "type": "note",
        "content": "**Namespace cost allocation**\n\nUnsupported panel (table)\n\n- Uses $costcpu/$costpcpu/$costDiscount as numeric multipliers, which Datadog formulas cannot express (only query names and literals)\n- preemptible split via label filter + `or` fallback also unsupported\n- Uses $costram/$costpram/$costDiscount as numeric multipliers, unsupported in Datadog formulas\n- Uses $costStorageSSD/$costStorageStandard as numeric multipliers, unsupported in Datadog formulas\n- Combines CPU/RAM/Storage cost sub-expressions, all relying on cost template variables as numeric multipliers; unsupported in Datadog formulas\n\nOriginal PromQL:\n```\nA: (\n sum(kube_pod_container_resource_requests_cpu_cores{namespace!=\"\",namespace!=\"kube-system\",cloud_google_com_gke_preemptible!=\"true\"}*($costcpu - ($costcpu / 100 * $costDiscount))) by(namespace)\n or\n count(\n count(container_spec_cpu_shares{namespace!=\"\",namespace!=\"kube-system\"}) by(namespace)\n ) by(namespace) -1\n)\n\n+\n\n(\n sum(kube_pod_container_resource_requests_cpu_cores{namespace!=\"\",namespace!=\"kube-system\",cloud_google_com_gke_preemptible=\"true\"}*$costpcpu) by(namespace)\n or\n count(\n count(container_spec_cpu_shares{namespace!=\"\",namespace!=\"kube-system\"}) by(namespace)\n ) by(namespace) -1\n)\nB: (\n sum(kube_pod_container_resource_requests_memory_bytes{namespace!=\"\",namespace!=\"kube-system\",cloud_google_com_gke_preemptible!=\"true\"} / 1024 / 1024 / 1024*($costram- ($costram / 100 * $costDiscount))) by (namespace) \n or\n count(\n count(container_spec_memory_limit_bytes{namespace!=\"\",namespace!=\"kube-system\"}) by(namespace)\n ) by(namespace) -1\n)\n\n+\n\n(\n sum(kube_pod_container_resource_requests_memory_bytes{namespace!=\"\",namespace!=\"kube-system\",cloud_google_com_gke_preemptible=\"true\"} / 1024 / 1024 / 1024 * $costpram ) by (namespace) \n or\n count(\n count(container_spec_memory_limit_bytes{namespace!=\"\",namespace!=\"kube-system\"}) by(namespace)\n ) by(namespace) -1\n)\nC: sum (\n sum(kube_persistentvolumeclaim_info{storageclass=~\".*ssd.*\"}) by (persistentvolumeclaim, namespace, storageclass)\n + on (persistentvolumeclaim, namespace) group_right(storageclass)\n sum(kube_persistentvolumeclaim_resource_requests_storage_bytes) by (persistentvolumeclaim, namespace) \n) by (namespace) / 1024 / 1024 /1024 * $costStorageSSD \n\nor\n\nsum (\n sum(kube_persistentvolumeclaim_info{storageclass!~\".*ssd.*\"}) by (persistentvolumeclaim, namespace, storageclass)\n + on (persistentvolumeclaim, namespace) group_right(storageclass)\n sum(kube_persistentvolumeclaim_resource_requests_storage_bytes) by (persistentvolumeclaim, namespace) \n) by (namespace) / 1024 / 1024 /1024 * $costStorageStandard\nD: # CPU \n(\n sum(kube_pod_container_resource_requests_cpu_cores{namespace!=\"\",namespace!=\"kube-system\",cloud_google_com_gke_preemptible!=\"true\"}*($costcpu - ($costcpu / 100 * $costDiscount))) by(namespace)\n or\n count(\n count(container_spec_cpu_shares{namespace!=\"\",namespace!=\"kube-system\"}) by(namespace)\n ) by(namespace) -1\n)\n\n+\n\n(\n sum(kube_pod_container_resource_requests_cpu_cores{namespace!=\"\",namespace!=\"kube-system\",cloud_google_com_gke_preemptible=\"true\"}*$costpcpu) by(namespace)\n or\n count(\n count(container_spec_cpu_shares{namespace!=\"\",namespace!=\"kube-system\"}) by(namespace)\n ) by(namespace) -1\n)\n\n+\n\n#END CPU \n# Memory \n\n(\n sum(kube_pod_container_resource_requests_memory_bytes{namespace!=\"\",namespace!=\"kube-system\",cloud_google_com_gke_preemptible!=\"true\"} / 1024 / 1024 / 1024*($costram- ($costram / 100 * $costDiscount))) by (namespace) \n or\n count(\n count(container_spec_memory_limit_bytes{namespace!=\"\",namespace!=\"kube-system\"}) by(namespace)\n ) by(namespace) -1\n)\n\n+\n\n(\n sum(kube_pod_container_resource_requests_memory_bytes{namespace!=\"\",namespace!=\"kube-system\",cloud_google_com_gke_preemptible=\"true\"} / 1024 / 1024 / 1024 * $costpram ) by (namespace) \n or\n count(\n count(container_spec_memory_limit_bytes{namespace!=\"\",namespace!=\"kube-system\"}) by(namespace)\n ) by(namespace) -1\n)\n\n+\n\n# ",
        "tick_pos": "50%",
        "font_size": "14",
        "show_tick": false,
        "tick_edge": "left",
        "text_align": "left",
        "has_padding": true,
        "vertical_align": "top",
        "background_color": "white"
      }
    },
    {
      "layout": {
        "x": 0,
        "y": 5,
        "width": 12,
        "height": 7
      },
      "definition": {
        "type": "group",
        "title": "CPU Metrics",
        "widgets": [
          {
            "layout": {
              "x": 0,
              "y": 0,
              "width": 12,
              "height": 2
            },
            "definition": {
              "type": "timeseries",
              "title": "Cluster CPUs",
              "yaxis": {
                "max": "auto",
                "min": "auto",
                "label": "",
                "scale": "linear",
                "include_zero": true
              },
              "requests": [
                {
                  "style": {
                    "palette": "dog_classic",
                    "line_type": "solid",
                    "line_width": "normal"
                  },
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:kubernetes_state.node.cpu_capacity{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:kubernetes_state.container.cpu_requested{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query3",
                      "query": "sum:kubernetes.cpu.usage.total{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query4",
                      "query": "sum:kubernetes_state.container.cpu_limit{*}",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "alias": "capacity",
                      "formula": "query1"
                    },
                    {
                      "alias": "requests",
                      "formula": "query2"
                    },
                    {
                      "alias": "usage",
                      "formula": "query3 / 1000000000"
                    },
                    {
                      "alias": "limits",
                      "formula": "query4"
                    }
                  ],
                  "display_type": "area",
                  "response_format": "timeseries"
                }
              ],
              "show_legend": true,
              "legend_layout": "auto",
              "legend_columns": [
                "avg",
                "min",
                "max",
                "value",
                "sum"
              ]
            }
          },
          {
            "layout": {
              "x": 0,
              "y": 2,
              "width": 12,
              "height": 2
            },
            "definition": {
              "type": "timeseries",
              "title": "CPU Mode",
              "yaxis": {
                "max": "auto",
                "min": "auto",
                "label": "",
                "scale": "linear",
                "include_zero": true
              },
              "requests": [
                {
                  "style": {
                    "palette": "dog_classic",
                    "line_type": "solid",
                    "line_width": "normal"
                  },
                  "queries": [
                    {
                      "name": "query1",
                      "query": "avg:system.cpu.user{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "avg:system.cpu.system{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query3",
                      "query": "avg:system.cpu.iowait{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query4",
                      "query": "avg:system.cpu.stolen{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query5",
                      "query": "avg:system.cpu.interrupt{*}",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "formula": "query1 + query2 + query3 + query4 + query5"
                    }
                  ],
                  "display_type": "area",
                  "response_format": "timeseries"
                }
              ],
              "show_legend": true,
              "legend_layout": "auto",
              "legend_columns": [
                "avg",
                "min",
                "max",
                "value",
                "sum"
              ]
            }
          },
          {
            "layout": {
              "x": 0,
              "y": 4,
              "width": 6,
              "height": 3
            },
            "definition": {
              "type": "query_table",
              "title": "CPU request utilization by namespace",
              "requests": [
                {
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:kubernetes_state.container.cpu_requested{*} by {kube_namespace}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:kubernetes.cpu.usage.total{*} by {kube_namespace}",
                      "aggregator": "avg",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "limit": {
                        "count": 50,
                        "order": "desc"
                      },
                      "formula": "query1"
                    },
                    {
                      "formula": "query2 / 1000000000"
                    }
                  ],
                  "response_format": "scalar"
                }
              ],
              "has_search_bar": "auto"
            }
          },
          {
            "layout": {
              "x": 6,
              "y": 4,
              "width": 6,
              "height": 3
            },
            "definition": {
              "type": "query_table",
              "title": "Cluster cost & utilization by node",
              "requests": [
                {
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:kubernetes.cpu.usage.total{*} by {host}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:kubernetes_state.node.cpu_capacity{*} by {host}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query3",
                      "query": "sum:kubernetes_state.container.cpu_requested{*} by {host}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query4",
                      "query": "sum:kubernetes_state.node.cpu_capacity{*} by {host}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "limit": {
                        "count": 50,
                        "order": "desc"
                      },
                      "formula": "(query1 / 1000000000) / query2"
                    },
                    {
                      "formula": "query3 / query4"
                    }
                  ],
                  "response_format": "scalar"
                }
              ],
              "has_search_bar": "auto"
            }
          }
        ],
        "show_title": true,
        "layout_type": "ordered"
      }
    },
    {
      "layout": {
        "x": 0,
        "y": 12,
        "width": 12,
        "height": 7
      },
      "definition": {
        "type": "group",
        "title": "Memory Metrics",
        "widgets": [
          {
            "layout": {
              "x": 0,
              "y": 0,
              "width": 12,
              "height": 2
            },
            "definition": {
              "type": "timeseries",
              "title": "Cluster memory (GB)",
              "yaxis": {
                "max": "auto",
                "min": "auto",
                "label": "",
                "scale": "linear",
                "include_zero": true
              },
              "requests": [
                {
                  "style": {
                    "palette": "dog_classic",
                    "line_type": "solid",
                    "line_width": "normal"
                  },
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:kubernetes_state.node.memory_capacity{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:kubernetes_state.container.memory_requested{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query3",
                      "query": "sum:kubernetes.memory.usage{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query4",
                      "query": "sum:kubernetes_state.container.memory_limit{*}",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "alias": "capacity",
                      "formula": "query1 / 1024 / 1024 / 1024"
                    },
                    {
                      "alias": "requests",
                      "formula": "query2 / 1024 / 1024 / 1024"
                    },
                    {
                      "alias": "usage",
                      "formula": "query3 / 1024 / 1024 / 1024"
                    },
                    {
                      "alias": "limits",
                      "formula": "query4 / 1024 / 1024 / 1024"
                    }
                  ],
                  "display_type": "area",
                  "response_format": "timeseries"
                }
              ],
              "show_legend": true,
              "legend_layout": "auto",
              "legend_columns": [
                "avg",
                "min",
                "max",
                "value",
                "sum"
              ]
            }
          },
          {
            "layout": {
              "x": 0,
              "y": 2,
              "width": 12,
              "height": 2
            },
            "definition": {
              "type": "timeseries",
              "title": "Cluster Memory Utilization",
              "yaxis": {
                "max": "auto",
                "min": "auto",
                "label": "",
                "scale": "linear",
                "include_zero": true
              },
              "requests": [
                {
                  "style": {
                    "palette": "dog_classic",
                    "line_type": "solid",
                    "line_width": "normal"
                  },
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:system.mem.usable{*} by {host}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:system.mem.total{*} by {host}",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "alias": "usage",
                      "formula": "(1 - (query1 / query2)) * 100"
                    }
                  ],
                  "display_type": "area",
                  "response_format": "timeseries"
                }
              ],
              "show_legend": true,
              "legend_layout": "auto",
              "legend_columns": [
                "avg",
                "min",
                "max",
                "value",
                "sum"
              ]
            }
          },
          {
            "layout": {
              "x": 0,
              "y": 4,
              "width": 6,
              "height": 3
            },
            "definition": {
              "type": "query_table",
              "title": "Memory requests & utilization by namespace",
              "requests": [
                {
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:kubernetes_state.container.memory_requested{*} by {kube_namespace}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:kubernetes.memory.usage{*} by {kube_namespace}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "limit": {
                        "count": 50,
                        "order": "desc"
                      },
                      "formula": "query1 / 1024 / 1024 / 1024"
                    },
                    {
                      "formula": "query2 / 1024 / 1024 / 1024"
                    }
                  ],
                  "response_format": "scalar"
                }
              ],
              "has_search_bar": "auto"
            }
          },
          {
            "layout": {
              "x": 6,
              "y": 4,
              "width": 6,
              "height": 3
            },
            "definition": {
              "type": "query_table",
              "title": "Node utilization of allocatable RAM",
              "requests": [
                {
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:kubernetes.memory.usage{*} by {host}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:kubernetes_state.node.memory_capacity{*} by {host}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query3",
                      "query": "sum:kubernetes_state.container.memory_requested{*} by {host}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query4",
                      "query": "sum:kubernetes_state.node.memory_capacity{*} by {host}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "limit": {
                        "count": 50,
                        "order": "desc"
                      },
                      "formula": "query1 * 100 / query2"
                    },
                    {
                      "formula": "query3 / query4"
                    }
                  ],
                  "response_format": "scalar"
                }
              ],
              "has_search_bar": "auto"
            }
          }
        ],
        "show_title": true,
        "layout_type": "ordered"
      }
    },
    {
      "layout": {
        "x": 0,
        "y": 19,
        "width": 12,
        "height": 7
      },
      "definition": {
        "type": "group",
        "title": "Storage Metrics",
        "widgets": [
          {
            "layout": {
              "x": 0,
              "y": 0,
              "width": 6,
              "height": 2
            },
            "definition": {
              "type": "query_table",
              "title": "Local Storage",
              "requests": [
                {
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:kubernetes.filesystem.usage_pct{*} by {host}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "avg:kubernetes.filesystem.usage_pct{*} by {host}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "limit": {
                        "count": 50,
                        "order": "desc"
                      },
                      "formula": "query1 / 1024 / 1024 / 1024 * 1.03"
                    },
                    {
                      "formula": "query2"
                    }
                  ],
                  "response_format": "scalar"
                }
              ],
              "has_search_bar": "auto"
            }
          },
          {
            "layout": {
              "x": 6,
              "y": 0,
              "width": 6,
              "height": 2
            },
            "definition": {
              "type": "timeseries",
              "title": "Local storage utilization",
              "yaxis": {
                "max": "auto",
                "min": "auto",
                "label": "",
                "scale": "linear",
                "include_zero": true
              },
              "requests": [
                {
                  "style": {
                    "palette": "dog_classic",
                    "line_type": "solid",
                    "line_width": "normal"
                  },
                  "queries": [
                    {
                      "name": "query1",
                      "query": "avg:kubernetes.filesystem.usage_pct{*}",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "alias": "reads",
                      "formula": "query1"
                    }
                  ],
                  "display_type": "area",
                  "response_format": "timeseries"
                }
              ],
              "show_legend": true,
              "legend_layout": "auto",
              "legend_columns": [
                "avg",
                "min",
                "max",
                "value",
                "sum"
              ]
            }
          },
          {
            "layout": {
              "x": 0,
              "y": 2,
              "width": 6,
              "height": 3
            },
            "definition": {
              "type": "query_table",
              "title": "Persistent Volume Claims",
              "requests": [
                {
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:kubernetes_state.persistentvolumeclaim.request_storage{*} by {kube_namespace,persistentvolumeclaim,storageclass}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:kubernetes.kubelet.volume.stats.used_bytes{*} by {persistentvolumeclaim,kube_namespace}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query3",
                      "query": "sum:kubernetes_state.persistentvolumeclaim.request_storage{*} by {persistentvolumeclaim,kube_namespace}",
                      "aggregator": "last",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "limit": {
                        "count": 50,
                        "order": "desc"
                      },
                      "formula": "query1 / 1073741824"
                    },
                    {
                      "formula": "query2 / query3"
                    }
                  ],
                  "response_format": "scalar"
                }
              ],
              "has_search_bar": "auto"
            }
          },
          {
            "layout": {
              "x": 6,
              "y": 2,
              "width": 6,
              "height": 3
            },
            "definition": {
              "type": "timeseries",
              "title": "Disk IOPS",
              "yaxis": {
                "max": "auto",
                "min": "auto",
                "label": "",
                "scale": "linear",
                "include_zero": true
              },
              "requests": [
                {
                  "style": {
                    "palette": "dog_classic",
                    "line_type": "solid",
                    "line_width": "normal"
                  },
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:system.io.r_s{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:system.io.w_s{*}",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "alias": "reads",
                      "formula": "query1"
                    },
                    {
                      "alias": "writes",
                      "formula": "query2"
                    }
                  ],
                  "display_type": "area",
                  "response_format": "timeseries"
                }
              ],
              "show_legend": true,
              "legend_layout": "auto",
              "legend_columns": [
                "avg",
                "min",
                "max",
                "value",
                "sum"
              ]
            }
          },
          {
            "layout": {
              "x": 0,
              "y": 5,
              "width": 12,
              "height": 2
            },
            "definition": {
              "type": "timeseries",
              "title": "Inode usage",
              "yaxis": {
                "max": "auto",
                "min": "auto",
                "label": "",
                "scale": "linear",
                "include_zero": true
              },
              "requests": [
                {
                  "style": {
                    "palette": "dog_classic",
                    "line_type": "solid",
                    "line_width": "normal"
                  },
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:kubernetes.kubelet.volume.stats.inodes_used{*} by {persistentvolumeclaim}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:kubernetes.kubelet.volume.stats.inodes{*} by {persistentvolumeclaim}",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "formula": "(query1 / query2) * 100"
                    }
                  ],
                  "display_type": "area",
                  "response_format": "timeseries"
                }
              ],
              "show_legend": true,
              "legend_layout": "auto",
              "legend_columns": [
                "avg",
                "min",
                "max",
                "value",
                "sum"
              ]
            }
          }
        ],
        "show_title": true,
        "layout_type": "ordered"
      }
    },
    {
      "layout": {
        "x": 0,
        "y": 26,
        "width": 12,
        "height": 2
      },
      "definition": {
        "type": "group",
        "title": "Network",
        "widgets": [
          {
            "layout": {
              "x": 0,
              "y": 0,
              "width": 12,
              "height": 2
            },
            "definition": {
              "type": "timeseries",
              "title": "Node network transmit",
              "yaxis": {
                "max": "auto",
                "min": "auto",
                "label": "",
                "scale": "linear",
                "include_zero": true
              },
              "requests": [
                {
                  "style": {
                    "palette": "dog_classic",
                    "line_type": "solid",
                    "line_width": "normal"
                  },
                  "queries": [
                    {
                      "name": "query1",
                      "query": "sum:system.net.bytes_sent{*}",
                      "data_source": "metrics"
                    },
                    {
                      "name": "query2",
                      "query": "sum:system.net.bytes_sent{device:eth0}",
                      "data_source": "metrics"
                    }
                  ],
                  "formulas": [
                    {
                      "alias": "node_out",
                      "formula": "query1"
                    },
                    {
                      "alias": "eth0 out",
                      "formula": "query2"
                    }
                  ],
                  "display_type": "area",
                  "response_format": "timeseries"
                }
              ],
              "show_legend": true,
              "legend_layout": "auto",
              "legend_columns": [
                "avg",
                "min",
                "max",
                "value",
                "sum"
              ]
            }
          }
        ],
        "show_title": true,
        "layout_type": "ordered"
      }
    }
  ],
  "description": "Converted from Grafana dashboard https://grafana.com/grafana/dashboards/8670 (by kubecost). A dashboard to help manage Kubernetes cluster costs and resources",
  "layout_type": "ordered",
  "reflow_type": "fixed",
  "template_variables": [
    {
      "name": "costcpu",
      "prefix": "costcpu",
      "default": "${VAR_COSTCPU}",
      "available_values": []
    },
    {
      "name": "costpcpu",
      "prefix": "costpcpu",
      "default": "${VAR_COSTPCPU}",
      "available_values": []
    },
    {
      "name": "costram",
      "prefix": "costram",
      "default": "${VAR_COSTRAM}",
      "available_values": []
    },
    {
      "name": "costpram",
      "prefix": "costpram",
      "default": "${VAR_COSTPRAM}",
      "available_values": []
    },
    {
      "name": "costStorageStandard",
      "prefix": "costStorageStandard",
      "default": "${VAR_COSTSTORAGESTANDARD}",
      "available_values": []
    },
    {
      "name": "costStorageSSD",
      "prefix": "costStorageSSD",
      "default": "${VAR_COSTSTORAGESSD}",
      "available_values": []
    },
    {
      "name": "costEgress",
      "prefix": "costEgress",
      "default": "${VAR_COSTEGRESS}",
      "available_values": []
    },
    {
      "name": "costDiscount",
      "prefix": "costDiscount",
      "default": "${VAR_COSTDISCOUNT}",
      "available_values": []
    }
  ]
}