How to troubleshoot backup failures

Andrii_Logvin · ‎03-31-2025

How to troubleshoot backup failures

How to start a backup

Sisense supports two options to initiate the creation of an application backup:

Via Sisense SI cli command: https://docs.sisense.com/main/SisenseLinux/backing-up-and-restoring-sisense.htm
Via REST API: on-demand and scheduled backups.

Any other ways to start a backup are not supported and may not work as intended.

Considerations

When running the SI commands, Sisense connects to the instance specified in the dns_application_name during the installation. If you create a clone of the environment, ensure this parameter points to the correct DNS of the clone instead of a production environment to avoid running commands in the wrong application.
A backup with farms (Elasticube data) requires much more free space than a regular backup. Check the size of the /opt/sisense/storage/tenants/*/farms folders and the available space on the shared drive before running the backup command with farms.
Based on the scale of your application, creating a backup may require considerable RAM resources. Consider scheduling backups when users’ activity in the application is low.

Execution steps

The execution of the source_add_completion.sh script requires the admin user credentials in the Sisense application. The Sisense web application must be reachable from the machine on which you execute this script (usually, one of the cluster machines or a bastion).
You can validate whether Sisense received an incoming authentication request by tailing the API-gateway logs during the script execution:
kubectl logs -n sisense -l app=api-gateway--tail=1 -f | grep "/api/v1/authentication/login"
You should look for the following message:
Finished handling request => POST /api/v1/authentication/login 200
If the request was not received, check DNS resolution, allowed outbound ports, security groups of your Load Balancer, etc.
Once the user is authenticated, the subsequent SI commands will be proxied from the API gateway to the management pod. In case of a failure, you can review the logs of the management service to check for any errors during the execution:
kubectl logs -n sisense -l app=management --tail=1 -f
For example, when running an si system backup-list command to view available backups in the system, you will see the following logs:
BE#587104 Finished handling POST request - url=https://test.sisense.com/api/v2/cli/execute
BE#820886 cli command started ParsedCli(incompleteWord=null, incompleteArg=null, commands=[system, backup-list], args={}, options=[], payload=)
BE#374860 cli command finishedParsedCli(incompleteWord=null, incompleteArg=null, commands=[system, backup-list], args={}, options=[], payload=)
If the si system backup command was executed successfully, Sisense will create a sisense-restore-hash pod. You can use the following commands to locate the pod and trace its logs:
kubectl get po -n sisense | grep backup

sisense-backup-xlsw7 1/1 Running 0 12s

Ensure the status of the pod becomes Running. If the pod is stuck in the Pending state, use kubectl describe po -n sisense -l job-name=sisense-backup command to find out why the pod was not assigned to a cluster node yet. Pay attention to the events at the bottom of the output.

Once the pod has entered the Running state, use the following command to view the logs of the backup:
kubectl logs -n sisense -l job-name=sisense-backup --tail=-1 -f

The last line in the sisense-backup logs indicates whether the backup creation was successful. If the task exited with a code other than 0, the backup creation has failed, and you can review the previous logs of the pod to identify the reason.
An example of a successful execution should end with
exited with code '0'.
The backup is created in the /opt/sisense/storage/system_backups folder. You can access the system_backups folder using the File Management feature: https://docs.sisense.com/main/SisenseLinux/uploading-files-to-customize-your-linux-deployment.htm. If you use a multi-node installation and would like to access the file from the server-side, the system_backups folder can be accessed by executing into the management pod.

Retry attempts

The created sisense-backup job is configured to retry the backup creation 6 times before failing. If the initial backup failed, you may see other pods with the sisense-backup-<hash> name in the Sisense namespace.

Common failures

1. A_CUBE_IS_BUILDING
ERROR: A cube is being build (A_CUBE_IS_BUILDING)

This error may happen when you initiate a full backup (including farms) while there is an active Elasticube build in the system. You can stop the build and retry the backup attempt.

2. error mkdir ./kube: permission denied

This error could occur in some Sisense versions when Sisense was installed with a non-default (other than 1000-id) user. Consider upgrading to the latest versions to address this issue.

As a workaround in the current version, you can adjust the backup to run it as root:

3. ENOSPC
Error: ENOSPC: no space left on device, write

The error indicates that the system ran out of disk space during the backup creation.

You can do the following to reduce the disk space used:

- Consider creating a backup without farms. Elasticube data (farms) usually takes the most storage space. If you restore a backup without farms, you can then rebuild all your Elasticube models to pull Elasticube data directly from your data sources.

- Remove outdated images as described here: https://community.sisense.com/t5/knowledge-base/optimizing-docker-storage/ta-p/26340

- Review the Elasticubes on the Data page and remove models that are no longer in use.

- Check that logrotate runs without issues in the cluster:
kubectl get po -n sisense | grep logrotate

4. Error: EACCES: permission denied, open '/opt/sisense/storage/translations/en-US/.navver.js.jbnEkp'

This error may happen after an unexpected interruption in the file upload that created an unreadable file. The indications of such an occurrence are

Extra filename path after the file format (.jbnEkp)
The file is 0 bytes

In such cases, you can delete this file and retry the backup creation.

5. Error getting usage [Error: ENOENT: no such file or directory, lstat '/opt/sisense/storage/usage/build/buildCustomTableStart/build-74877998b-xt69m-2023_01_29.csv']

The backup process might fail if the Usage Analytics Model elasticube is scheduled to build at the same time. Consider rescheduling the Usage Analytics elasticube build to another time frame.

This article provides a guide on troubleshooting backup failures in Sisense. It details the methods to initiate backups using the Sisense SI CLI command and REST API

Related Resources

Disclaimer: This post outlines a potential custom workaround for a specific use case or provides instructions regarding a specific task. The solution may not work in all scenarios or Sisense versions, so we strongly recommend testing it in your environment before deployment. If you need further assistance with this, please let us know.

Sisense Community

How to troubleshoot backup failures