Highly Available Cluster Setup

Before you begin:

• Install kubectl.
• Ensure that you have a production-ready Kubernetes cluster with at least three worker nodes running in a cloud provider of your choice.
• (Optional) To run Dgraph Alpha with TLS, see TLS Configuration.

Installing a highly available Dgraph cluster

1. Verify that you are able to access the nodes in the Kubernetes cluster:

   kubectl get nodes
   

   An output similar to this appears:

     NAME                                          STATUS   ROLES    AGE   VERSION
    <aws-ip-hostname>.<region>.compute.internal   Ready    <none>   1m   v1.15.11-eks-af3caf
    <aws-ip-hostname>.<region>.compute.internal   Ready    <none>   1m   v1.15.11-eks-af3caf
    <aws-ip-hostname>.<region>.compute.internal   Ready    <none>   1m   v1.15.11-eks-af3caf
   

   After your Kubernetes cluster is up, you can use dgraph-ha.yaml to start the cluster.

2. Start a StatefulSet that creates Pods with Zero, Alpha, and Ratel UI:

   kubectl create --filename https://raw.githubusercontent.com/dgraph-io/dgraph/main/contrib/config/kubernetes/dgraph-ha/dgraph-ha.yaml
   

   An output similar to this appears:

   service/dgraph-zero-public created
   service/dgraph-alpha-public created
   service/dgraph-ratel-public created
   service/dgraph-zero created
   service/dgraph-alpha created
   statefulset.apps/dgraph-zero created
   statefulset.apps/dgraph-alpha created
   deployment.apps/dgraph-ratel created
   

3. Confirm that the Pods were created successfully.

   kubectl get pods
   

   An output similar to this appears:

    NAME                  READY   STATUS    RESTARTS   AGE
   dgraph-alpha-0        1/1     Running   0          6m24s
   dgraph-alpha-1        1/1     Running   0          5m42s
   dgraph-alpha-2        1/1     Running   0          5m2s
   dgraph-ratel-<pod-id> 1/1     Running   0          6m23s
   dgraph-zero-0         1/1     Running   0          6m24s
   dgraph-zero-1         1/1     Running   0          5m41s
   dgraph-zero-2         1/1     Running   0          5m6s
   

   You can check the logs for the Pod using kubectl logs --follow <POD_NAME>..

4. Port forward from your local machine to the Pod:

      kubectl port-forward service/dgraph-alpha-public 8080:8080
      kubectl port-forward service/dgraph-ratel-public 8000:8000
   

5. Go to http://localhost:8000 to access Dgraph using the Ratel UI.

Deleting highly available Dgraph resources

Delete all the resources using:

kubectl delete --filename https://raw.githubusercontent.com/dgraph-io/dgraph/main/contrib/config/kubernetes/dgraph-ha/dgraph-ha.yaml
kubectl delete persistentvolumeclaims --selector app=dgraph-zero
kubectl delete persistentvolumeclaims --selector app=dgraph-alpha

Before you begin

• Install kubectl.
• Ensure that you have a production-ready Kubernetes cluster with atleast three worker nodes running in a cloud provider of your choice.
• Install Helm.
• (Optional) To run Dgraph Alpha with TLS, see TLS Configuration.

Installing a highly available Dgraph cluster using Helm

1. Verify that you are able to access the nodes in the Kubernetes cluster:

   kubectl get nodes
   

   An output similar to this appears:

     NAME                                          STATUS   ROLES    AGE   VERSION
    <aws-ip-hostname>.<region>.compute.internal   Ready    <none>   1m   v1.15.11-eks-af3caf
    <aws-ip-hostname>.<region>.compute.internal   Ready    <none>   1m   v1.15.11-eks-af3caf
    <aws-ip-hostname>.<region>.compute.internal   Ready    <none>   1m   v1.15.11-eks-af3caf
   

   After your Kubernetes cluster is up and running, you can use of the Dgraph Helm chart to install a highly available Dgraph cluster

2. Add the Dgraph helm repository::

   helm repo add dgraph https://charts.dgraph.io
   

3. Install the chart with <RELEASE-NAME>:

   helm install <RELEASE-NAME> dgraph/dgraph
   

   You can also specify the version using:

      helm install <RELEASE-NAME> dgraph/dgraph --set image.tag="v24.0.1"
   

   When configuring the Dgraph image tag, be careful not to use latest or main in a production environment. These tags may have the Dgraph version change, causing a mixed-version Dgraph cluster that can lead to an outage and potential data loss.

   An output similar to this appears:

    ```bash
    NAME: <RELEASE-NAME>
    LAST DEPLOYED: Wed Feb  1 21:26:32 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    1. You have just deployed Dgraph, version 'v21.12.0'.
   
       For further information:
        * Documentation: https://dgraph.io/docs/
        * Community and Issues: https://discuss.dgraph.io/   
    2. Get the Dgraph Alpha HTTP/S endpoint by running these commands.
       export ALPHA_POD_NAME=$(kubectl get pods --namespace default --selector "statefulset.kubernetes.io/pod-name=<RELEASE-NAME>-dgraph-alpha-0,release=<RELEASE-NAME>-dgraph" --output jsonpath="{.items[0].metadata.name}")
       echo "Access Alpha HTTP/S using http://localhost:8080"
       kubectl --namespace default port-forward $ALPHA_POD_NAME 8080:8080
   
      NOTE: Change "http://" to "https://" if TLS was added to the Ingress, Load Balancer, or Dgraph Alpha service.
    ```
   

4. Get the name of the Pods in the cluster using kubectl get pods:

      NAME                          READY   STATUS    RESTARTS   AGE
     <RELEASE-NAME>-dgraph-alpha-0   1/1     Running   0          4m48s
     <RELEASE-NAME>-dgraph-alpha-1   1/1     Running   0          4m2s
     <RELEASE-NAME>-dgraph-alpha-2   1/1     Running   0          3m31s
     <RELEASE-NAME>-dgraph-zero-0    1/1     Running   0          4m48s
     <RELEASE-NAME>-dgraph-zero-1    1/1     Running   0          4m10s
     <RELEASE-NAME>-dgraph-zero-2    1/1     Running   0          3m50s
   
   

5. Get the Dgraph Alpha HTTP/S endpoint by running these commands:

       export ALPHA_POD_NAME=$(kubectl get pods --namespace default --selector "statefulset.kubernetes.io/pod-name=<RELEASE-NAME>-dgraph-alpha-0,release=<RELEASE-NAME>-dgraph" --output jsonpath="{.items[0].metadata.name}")
       echo "Access Alpha HTTP/S using http://localhost:8080"
       kubectl --namespace default port-forward $ALPHA_POD_NAME 8080:8080
   

Deleting the resources from the cluster

1. Delete the Helm deployment using:

     helm delete my-release
   

2. Delete associated Persistent Volume Claims:

   kubectl delete pvc --selector release=my-release
   

Public Internet

To use an external load balancer, set the service type to LoadBalancer.

1. To expose Alpha service to the Internet use:

helm install <RELEASE-NAME> dgraph/dgraph --set alpha.service.type="LoadBalancer"

2. To expose Alpha and Ratel services to the Internet use:

helm install <RELEASE-NAME> dgraph/dgraph --set alpha.service.type="LoadBalancer" --set ratel.service.type="LoadBalancer"

Private Internal Network

An external load balancer can be configured to face internally to a private subnet rather the public Internet. This way it can be accessed securely by clients on the same network, through a VPN, or from a jump server. In Kubernetes, this is often configured through service annotations by the provider. Here’s a small list of annotations from cloud providers:

As an example, using Amazon EKS as the provider.

1. Create a Helm chart configuration values file <MY-CONFIG-VALUES>.yaml file:

# <MY-CONFIG-VALUES>.yaml
alpha:
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
ratel:
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"

1. To expose Alpha and Ratel services privately, use:

helm install <RELEASE-NAME> dgraph/dgraph --values <MY-CONFIG-VALUES>.yaml

You can expose Alpha and Ratel using an ingress resource that can route traffic to service resources. Before using this option you may need to install an ingress controller first, as is the case with AKS and EKS, while in the case of GKE, this comes bundled with a default ingress controller. When routing traffic based on the hostname, you may want to integrate an addon like ExternalDNS so that DNS records can be registered automatically when deploying Dgraph.

As an example, you can configure a single ingress resource that uses ingress-nginx for Alpha and Ratel services.

1. Create a Helm chart configuration values file, <MY-CONFIG-VALUES>.yaml file:

# <MY-CONFIG-VALUES>.yaml
global:
  ingress:
    enabled: false
    annotations:
      kubernetes.io/ingress.class: nginx
    ratel_hostname: "ratel.<my-domain-name>"
    alpha_hostname: "alpha.<my-domain-name>"

2. To expose Alpha and Ratel services through an ingress:

helm install <RELEASE-NAME> dgraph/dgraph --values <MY-CONFIG-VALUES>.yaml

You can run kubectl get ingress to see the status and access these through their hostname, such as http://alpha.<my-domain-name> and http://ratel.<my-domain-name>