This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Cookie Policy. Click Preferences to customize your cookie settings.
on - edited on by
DRay
The Safe-Mode is triggered when the pod (query/build) OR overall server RAM consumption gets to 85% of usage.
If Safe Mode is triggered on build - it will cancel the build due to OOM. On the dashboard - the application will restart the Query pod of the cube (delete and start it again) in order to release memory. The safe mode also has a grace period that cancels all new queries for 30 seconds. On the next dashboard page restart after 30 seconds, you will see the results in case of correct Data Groups settings.
Log location:
single node, (MultiNode logs located on a first node (the one that is the first in the list of nodes in config.yaml that been used to install Sisense.)
/var/log/sisense/sisense/
- build - ec-<name of cube>-bld-hash.log
for example ec-sample-ecommerce-bld-a63510c3-3672-0.log
- query - ec-<name of cube>-qry-hash.log
for example ec-sample-ecommerce-qry-a63510c3-3672-0.log
In case you received the Safe-mode triggered while BUILD.
Check out the build error message and identify if it's a pod or server overall issue.
- Build pod limits issue BE#521691 :
In order to fix the POD limitation issue, consider increasing the Build Node RAM limits in the Data Groups:
- Build server overall OOM issue BE#636134:
- Other builds. If so please consider changing the build schedule accordingly.
- In most cases, the RAM is consumed by Query pods. If so, try to stop the cubes of the query pod to release the RAM.
- Consider RAM upgrade
- Increase MAX RAM in Data Group settings
Settings of the Elasticube Build Safe-Mode are located in the Configuration Manager, Elasticube Build Params, where you can enable/disable Safe mode and change the % of RAM that should be saved. It's not recommended to disable the Safe-mode as it supposes to save the 15% of RAM for Technicians to login and fix issues. Note, in case disabling Safe Mode the server may become unresponsive and even server restart might not fix the issue. Do not disable Safe-mode without urgent need!
In case you received the Safe-mode triggered while surfing the Dashboard.
It's also can be or due to POD limitation, or server OOM.
The Dashboards uses RAM of the query pods where Sisense keeps results of Dashboard's query execution results for further reuse by another dashboard user. Sisense collects results for 85% of the RAM allocated in the Data group
settings after it tries to remove old results and replace them with new, at this time if the huge quey will be received from a dashboard, the Safe Mode will be triggered that will delete the query pod (remove all saved results) and will start a new pod to calculate results. At this moment the user will see the error on a dashboard, however, after the pod will be restarted (usually about 14 sec, however, the more RAM allocated to the query pod the more it will take to release the RAM and start a new pod) user can restart the page to get the results.
UPD: starting from Sisense L2021.5, the Soft-Restart of the Elasticubes has been implemented (beta, please test it before go live). The Soft Restart of the qry pod will restart only Monet DB and not the entire pod, which will speed up restart in case hitting Safe-Mode. to enable Safe Restart:
- go to the control panel
- click 5 times on Sisense Logo at the top left corner
- navigate to Management section on the left-hand side menu
- scroll down to the end of a page
- enable Soft-Restart
in order to fix:
- Consider changing Data Group Settings
- Consider RAM upgrade
- increase amount of Instances in the Data Group. So when one of the instances would be restarted du to a Safe-Mode the other will handle the requests
Settings of the Elasticube Query Safe-Mode located in the Configuration Manager, Elasticube Query Params:
Cleanup Threshold Percentage - the % of the Safe-Mode (default 15%)
Safe-Mode Grace Period - after safe-mode is triggered, Sisense will cancel all queries for 30 sec.
Disable/Enable Safe-mode - NOT RECOMENDED!!!
