Ask a Question Export data from Dgraph As an Administrator you can export data on all nodes, configure the Alpha server, specify the export format, export to an object store, disable HTTPS for exports, and encrypt exports Export data using the GraphQL admin endpoint You can export the entire data by executing a GraphQL mutation on the /admin endpoint of any Alpha node. Before you begin: Ensure that there is sufficient space on disk to store the export. Each Dgraph Alpha leader for a group writes output as a gzipped file to the export directory specified through the --export flag (defaults to an export directory). If any of the groups fail because of insufficient space on the disk, the entire export process is considered failed and an error is returned. Make a note of the export directories of the Alpha server nodes. For more information about configuring the Dgraph Alpha server, see Config. This mutation triggers the export from each of the Alpha leader for a group. Depending on the Dgraph configuration several files are exported. It is recommended that you copy the files from the Alpha server nodes to a safe place when the export is complete. mutation { export(input: {}) { response { message code } } } The export data of the group: in the Aplha instance is stored in the Alpha. in every other group is stored in the Alpha leader of that group. You need to retrieve the right export files from the Alpha instances in the cluster. Dgraph does not copy all files to the Alpha that initiated the export. When the export is complete a response similar to this appears: {"data":{ "export":{ "response":{ "message":"Export completed.", "code":"Success" } } }, "extensions":{ "tracing":{ "version":1, "startTime":"2022-12-14T07:39:51.061712416Z","endTime":"2022-12-14T07:39:51.129431494Z", "duration":67719080 } } } Export data format By default, Dgraph exports data in RDF format. Replace <FORMAT>with json or rdf in this GraphQL mutaion: mutation { export(input: { format: "<FORMAT>" }) { response { message code } } } Export to NFS or a file path You can override the default folder path by adding the destination input field to the directory where you want to export data. Replace <PATH> in this GraphQL mutaion with the absolute path of the directory to export data. mutation { export(input: { format: "<FORMAT>" destination: "<PATH>" }) { response { message code } } } Export to an object store You can export to an AWS S3, Azure Blob Storage or Google Cloud Storage. Example mutation to export to AWS S3 mutation { export(input: { destination: "s3://s3.<region>.amazonaws.com/<bucket-name>" accessKey: "<aws-access-key-id>" secretKey: "<aws-secret-access-key>" }) { response { message code } } } Note The Dgraph URL used for S3 is different than the AWS CLI tools with the aws s3 command, which uses a shortened format: s3://<bucket-name>. Example mutation to export to MinIO mutation { export(input: { destination: "minio://<address>:9000/<bucket-name>" accessKey: "<minio-access-key>" secretKey: "<minio-secret-key>" }) { response { message code } } } Export to a MinIO gateway You can use MinIO as a gateway to other object stores, such as Azure Blob Storage or Google Cloud Storage. Azure Blob Storage You can use Azure Blob Storage through the MinIO Azure Gateway. Before you begin: Configure a storage account and a Blob container to organize the blobs. Make a note the name of the blob container. It is the <bucket-name> when specifying the destination in the GraphQL mutation. Retrieve storage accounts keys to configure MinIO. Because, MinIO Azure Gateway uses MINIO_ACCESS_KEY and MINIO_SECRET_KEY to correspond to Azure Storage Account AccountName and AccountKey. You can access Azure Blob Storage locally using one of these methods: Using MinIO Azure Gateway with the MinIO Binary export MINIO_ACCESS_KEY="<AccountName>" export MINIO_SECRET_KEY="<AccountKey>" minio gateway azure Using MinIO Azure Gateway with Docker docker run --detach --rm --name gateway \ --publish 9000:9000 \ --env MINIO_ACCESS_KEY="<AccountName>" \ --env MINIO_SECRET_KEY="<AccountKey>" \ minio/minio gateway azure Using MinIO Azure Gateway with the MinIO Helm chart for Kubernetes: helm repo add minio https://helm.min.io/ helm install my-gateway minio/minio \ --set accessKey="<AccountName>",secretKey="<AccountKey>" \ --set azuregateway.enabled=true You can use the MinIO GraphQL mutation with MinIO configured as a gateway. Google Cloud Storage You can use Google Cloud Storage through the MinIO GCS Gateway. Before you begin: Create storage buckets Create a Service Account key for GCS and get a credentials file. For more information, see Create a Service Account key. When you have a credentials.json, you can access GCS locally using one of these methods: Using MinIO GCS Gateway with the MinIO Binary export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json" export MINIO_ACCESS_KEY="<minio-access-key>" export MINIO_SECRET_KEY="<minio-secret-key>" minio gateway gcs "<project-id>" Using MinIO GCS Gateway with Docker docker run --detach --rm --name gateway \ --publish 9000:9000 \ --volume "</path/to/credentials.json>":/credentials.json \ --env GOOGLE_APPLICATION_CREDENTIALS=/credentials.json \ --env MINIO_ACCESS_KEY="<minio-access-key>" \ --env MINIO_SECRET_KEY="<minio-secret-key>" \ minio/minio gateway gcs "<project-id>" Using MinIO GCS Gateway with the MinIO Helm chart for Kubernetes: ## create MinIO Helm config cat <<-EOF > myvalues.yaml accessKey: <minio-access-key> secretKey: <minio-secret-key> gcsgateway: enabled: true projectId: <project-id> gcsKeyJson: | $(IFS='\n'; while read -r LINE; do printf ' %s\n' "$LINE"; done < "</path/to/credentials.json>") EOF ## deploy MinIO GCS Gateway helm repo add minio https://helm.min.io/ helm install my-gateway minio/minio \ --values myvalues.yaml You can use the MinIO GraphQL mutation with MinIO configured as a gateway. Disable HTTPS for exports to S3 and Minio By default, Dgraph assumes the destination bucket is using HTTPS. If that is not the case, the export fails. To export to a bucket using HTTP (insecure), set the query parameter secure=false with the destination endpoint in the destination field: mutation { export(input: { destination: "minio://<address>:9000/<bucket-name>?secure=false" accessKey: "<minio-access-key>" secretKey: "<minio-secret-key>" }) { response { message code } } } Use anonymous credentials When exporting to S3 or MinIO where credentials are not required, can set anonymous to true. mutation { export(input: { destination: "s3://s3.<region>.amazonaws.com/<bucket-name>" anonymous: true }) { response { message code } } } Encrypt exports Export is available wherever an Alpha is running. To encrypt an export, the Alpha must be configured with the --encryption key-file=value. Note The --encryption key-file was used for Encryption at Rest and will now also be used for encrypted exports. Use curl to trigger an export This is an example of how you can use curl to trigger an export. Create GraphQL file for the desired mutation: cat <<-EOF > export.graphql mutation { export(input: { destination: "s3://s3.<region>.amazonaws.com/<bucket-name>" accessKey: "<aws-access-key-id>" secretKey: "<aws-secret-access-key>" }) { response { message code } } } EOF Trigger an export with curl curl http://localhost:8080/admin --silent --request POST \ --header "Content-Type: application/graphql" \ --upload-file export.graphql ← Export data Export data from Dgraph Cloud →