Monitoring the Cluster
Monitoring the Kubernetes Cluster
Dgraph exposes Prometheus metrics to monitor the state of various components involved in the cluster, including Dgraph Alpha and Zero nodes. You can setup Prometheus monitoring for your cluster.
You can use Helm to install kube-prometheus-stack chart. This Helm chart is a collection of Kubernetes manifests, Grafana dashboards, Prometheus rules combined with scripts to provide monitoring with Prometheus using the Prometheus Operator. This Helm chart also installs Grafana, node_exporter, kube-state-metrics.
Before you begin:
- Install kubectl.
- Ensure that you have a production-ready Kubernetes cluster with atleast three worker nodes running in a cloud provider of your choice.
- Install Helm
Install using Helm Chart
dgraph-prometheus-operator.yamland edit the values as appropriate for adding endpoints, adding alert rules, adjusting alert manager configuration, adding Grafana dashboard, and others. For more information see, Dgraph helm chart values.
prometheusOperator: createCustomResource: true grafana: enabled: true persistence: enabled: true accessModes: ["ReadWriteOnce"] size: 5Gi defaultDashboardsEnabled: true service: type: ClusterIP alertmanager: service: labels: app: dgraph-io alertmanagerSpec: storage: volumeClaimTemplate: spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 5Gi replicas: 1 logLevel: debug config: global: resolve_timeout: 2m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' routes: - match: alertname: Watchdog receiver: 'null' receivers: - name: 'null' prometheus: service: type: ClusterIP serviceAccount: create: true name: prometheus-dgraph-io prometheusSpec: storageSpec: volumeClaimTemplate: spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 25Gi resources: requests: memory: 400Mi enableAdminAPI: false additionalServiceMonitors: - name: zero-dgraph-io endpoints: - port: http-zero path: /debug/prometheus_metrics namespaceSelector: any: true selector: matchLabels: monitor: zero-dgraph-io - name: alpha-dgraph-io endpoints: - port: http-alpha path: /debug/prometheus_metrics namespaceSelector: any: true selector: matchLabels: monitor: alpha-dgraph-io
secrets.yamlthat has the credentials for Grafana.
grafana: adminPassword: <GRAFANA-PASSWORD>
helm repo add stable https://charts.helm.sh/stable helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update
Install kube-prometheus-stack with the
<MY-RELEASE-NAME>in the namescape named
helm install <MY-RELEASE-NAME>\ --values dgraph-prometheus-operator.yaml \ --values secrets.yaml \ prometheus-community/kube-prometheus-stack --namespace monitoring
An output similar to the following appears:
NAME: dgraph-prometheus-release LAST DEPLOYED: Sun Feb 5 21:35:45 2023 NAMESPACE: monitoring STATUS: deployed REVISION: 1 NOTES: kube-prometheus-stack has been installed. Check its status by running: kubectl --namespace monitoring get pods -l "release=dgraph-prometheus-release" Visit https://github.com/prometheus-operator/kube-prometheus instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.
Check the list of services in the
kubectl get svc -n monitoring:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 29s dgraph-prometheus-release-alertmanager ClusterIP 10.128.239.240 <none> 9093/TCP 32s dgraph-prometheus-release-grafana ClusterIP 10.128.213.70 <none> 80/TCP 32s dgraph-prometheus-release-kube-state-metrics ClusterIP 10.128.139.145 <none> 8080/TCP 32s dgraph-prometheus-release-operator ClusterIP 10.128.6.5 <none> 443/TCP 32s dgraph-prometheus-release-prometheus ClusterIP 10.128.255.88 <none> 9090/TCP 32s dgraph-prometheus-release-prometheus-node-exporter ClusterIP 10.128.103.131 <none> 9100/TCP 32s prometheus-operated ClusterIP None <none> 9090/TCP 29s
kubectl port-forward svc/dgraph-prometheus-release-prometheus -n monitoring 9090to access Prometheus at
kubectl --namespace monitoring port-forward svc/grafana 3000:80to access Grafana at
Log in to Grafna using the password that you had set in the
In the Dashboards menu of Grafana, select Import.
In the Dashboards/Import dashboard page copy the contents of the dgraph-kubernetes-grafana-dashboard.json file in Import via panel json and click Load.
You can visualize all Dgraph Alpha and Zero Kubernetes Pods, using the regex pattern
"/dgraph-.*-[0-9]*$/. You can change this in the dashboard configuration and select the variable Pod. For example, if you have multiple releases, and only want to visualize the current release named
my-release-3, change the regex pattern to
"/my-release-3.*dgraph-.*-[0-9]*$/"in the Pod variable of the dashboard configuration. By default, the Prometheus that you installed is configured as the
The Kubernetes configurations in the previous sections were configured to run
Dgraph with any storage type (
storage-class: anything). On the common cloud
environments like AWS, GCP, and Azure, the default storage type are slow disks
like hard disks or low IOPS SSDs. We highly recommend using faster disks for
ideal performance when running Dgraph.
The AWS storage-optimized i-class instances provide locally attached NVMe-based SSD storage which provide consistent very high IOPS. The Dgraph team uses i3.large instances on AWS to test Dgraph.
You can create a Kubernetes
StorageClass object to provision a specific type
of storage volume which you can then attach to your Dgraph Pods. You can set up
your cluster with local SSDs by using Local Persistent
This Kubernetes feature is in beta at the time of this writing (Kubernetes
v1.13.1). You can first set up an EC2 instance with locally attached storage.
Once it is formatted and mounted properly, then you can create a StorageClass to
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: <your-local-storage-class-name> provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer
Currently, Kubernetes does not allow automatic provisioning of local storage. So a PersistentVolume with a specific mount path should be created:
apiVersion: v1 kind: PersistentVolume metadata: name: <your-local-pv-name> spec: capacity: storage: 475Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete storageClassName: <your-local-storage-class-name> local: path: /data nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <node-name>
Then, in the StatefulSet configuration you can claim this local storage in .spec.volumeClaimTemplate:
kind: StatefulSet ... volumeClaimTemplates: - metadata: name: datadir spec: accessModes: - ReadWriteOnce storageClassName: <your-local-storage-class-name> resources: requests: storage: 500Gi
You can repeat these steps for each instance that’s configured with local node storage.
Non-local persistent disks
EBS volumes on AWS and PDs on GCP are persistent disks that can be configured with Dgraph. The disk performance is much lower than locally attached storage but can be sufficient for your workload such as testing environments.
When using EBS volumes on AWS, we recommend using Provisioned IOPS SSD EBS
volumes (the io1 disk type) which provide consistent IOPS. The available IOPS
for AWS EBS volumes is based on the total disk size. With Kubernetes, you can
request io1 disks to be provisioned with this config with 50 IOPS/GB using the
kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: <your-storage-class-name> provisioner: kubernetes.io/aws-ebs parameters: type: io1 iopsPerGB: "50" fsType: ext4
Example: Requesting a disk size of 250Gi with this storage class would provide 12.5K IOPS.
Removing a Dgraph Pod
In the event that you need to completely remove a Pod (e.g., its disk got
corrupted and data cannot be recovered), you can use the
/removeNode API to
remove the node from the cluster. With a Kubernetes StatefulSet, you’ll need to
remove the node in this order:
- On the Zero leader, call
/removeNodeto remove the Dgraph instance from the cluster (see More about Dgraph Zero). The removed instance will immediately stop running. Any further attempts to join the cluster will fail for that instance since it has been removed.
- Remove the PersistentVolumeClaim associated with the Pod to delete its data. This prepares the Pod to join with a clean state.
- Restart the Pod. This will create a new PersistentVolumeClaim to create new data directories.
When an Alpha Pod restarts in a replicated cluster, it will join as a new member of the cluster, be assigned a group and an unused index from Zero, and receive the latest snapshot from the Alpha leader of the group.
When a Zero Pod restarts, it must join the existing group with an unused index
ID. You set the index ID with the
idx option. This might
require you to update the StatefulSet configuration.
Kubernetes and Bulk Loader
You may want to initialize a new cluster with an existing data set such as data from the Dgraph Bulk Loader. You can use Init Containers to copy the data to the Pod volume before the Alpha process runs.
initContainers configuration in
to learn more.