Build Stability Improvements on Heavy Loaded Systems
Symptoms Relevant for Linux Below are a couple of select symptoms your system could be experiencing under a big load with simultaneous builds. Please note the list is not exhaustive: Build failures over random cubes or a couple of cubes that are being rebuilt manually successfully CubeIsUnreachable error Failure due to Build Service restart Diagnosis Build flow is connected to three main services: Build: Takes care of triggering builds and moving logs between the Pod and UI Management: Creates ec-bld pod and takes care of Kubernetes level communication over the flow Ec: <name_of_the_cube>-bld - actual cube import process which creates the folder, makes import If memory consumption of the ec-bld pod is being controlled by the DataGroup Max RAM for Build (more in-depth article here) and affects just this cube, then Build/Management works for all builds running on the system. Sometimes when there are 4+ cubes that run in parallel, Build/Management services could be under heavy load and require additional RAM. Since both of the services are Java based they have a default memory limit mechanism that allows 500 MB of RAM to be used. If the service needs more, it could be throttling which can affect defined timeouts or cause service restart. If you want to ensure that this is the case, please check graphana for build/management services when builds fail. Linked is additional information on how to use Grafana to troubleshoot performance issues. Keep in mind that by default services are limited to 500 MB but could consume more since there are additional parts of the service that are not Java-based and Java limits can peak from time to time. However, if you see that the service is under load, it is a good idea to allocate more RAM for stability improvements. Solution Increase the Memory Limits for the Build and Management services. Since this is a Java service there are two places to update: Kubernetes deployment limits Java service on the Sisense Configuration side To update Memory Limits please follow the steps below: 1. SSH to the server 2. Execute line 1 for build or line 2 for management: kubectl edit deployment -n <namespace> build kubectl edit deployment -n <namespace> management 3. Find "resources: limits" and update memory size to 4000 (press “i” on the keyboard to enter edit mode → edit value → ESC to exit edit mode → use “:wq!” to exit and save changes). 4. Build/management pod will automatically restart after deployment modification. 5. Navigate to Admin → System management → Configuration → Build → Memory limit for build pod and Navigate to Admin → System management → Configuration → Advanced Management params → Memory limit for management pod. Set the value to be 1000 lower than the value set in step #3 for the deployment. 6. Save settings. Tips: Please ensure that the Memory Limit value is correctly entered. If there is a problem with the value, the service will not start. Please keep in mind that the Deployment value should be 1000m bigger than the Configuration value. If you need any additional help, please contact Sisense Support.Sisense application logs
Sisense application logs The Sisense Support team typically uses the application log data to investigate outages, troubleshoot bugs, or analyze reported issues. Analyzing application logs can help the Support team determine the root cause of incidents. The location of log directories on a single node: /var/log/sisense/namespace/ /var/log/sisense/sisense/query.log /var/log/sisense/sisense/combined.log -- logs of all services If your environment does not have this setup, here is how you can gather the appropriate logs to share For Example, if interested to Access galaxy logs: kubectl -n sisense logs $(kubectl -n sisense get pods -l app="galaxy" -o custom-columns=":.metadata.name" ) Tail the log and print the last 100 lines: kubectl -n sisense -f --tail=100 logs $(kubectl -n sisense get pods -l app="galaxy" -o custom-columns=":.metadata.name" ) Get logs from specific pod: sisense@node1: kubectl logs $(PODNAME) -n $(NAMESPACE) Helpful Tips: In all Sisense installs, it would be very helpful to generate the Support File in the Admin tab under 'System Management'. When the issue occurs, you can download the file; this application log is a file that contains information about events that have occurred within a software application. This file contains logs of all services. Additional Resources: Guideline for submitting a Support inquiry Overview to Interpreting Sisense 7.2+ LogsHow to update style sheets in the branding folder for dashboards
In this article, we will address the issue of updates made to style sheets within a branding folder not reflecting on dashboards. Users often encounter this problem due to browser caching, which prevents the most updated CSS files from loading. This guide provides solutions to ensure your dashboard reflects the latest version of your style sheets.
Error in ingress-nginx-controller pod after upgrading Sisense to L2024.1
Error in ingress-nginx-controller pod after upgrading Sisense to L2024.1: main.go:64] port 80 is already in use. Please check the flag --http-port !NOTE! - this guide is relevant for non-cloud managed Sisense deployments i.e. installed with Sisense Kubernetes (RKE). After upgrading Sisense to version L2024.1 you might face the following error in ingress-nginx-controller pod’s logs (the default namespace): main.go:64] port 80 is already in use. Please check the flag --http-port Such an issue might be the result of incompatibility between the ingress-Nginx release version that was updated in Sisense L2024.1 and the Kubernetes version in case it is lower than v1.26. In case Sisense L2024.1 was freshly installed on the non-cloud instance (i.e. with Sisense RKE) as a fresh deployment (and was not upgraded from the previous version) this issue should not reveal since L2024.1 package already comes with v1.26 Kubernetes by default. However, if Sisense was upgraded to L2024.1 or it was deployed/installed on Kubernetes lower than v1.26 there will be an incompatibility between the ingress-Nginx release implemented in L2024.1 (4.10.0). To resolve the issue it is necessary to re-run the Sisense upgrade with the following parameters in the configuration yaml file: update: true update_k8s_version: true In case you are still facing the same issue after that, please open a ticket for Sisense Support. This article provides a brief explanation regarding possible Nginx<->Kubernetes compatibility issues in Sisense L2024.1. The article also provides the steps to resolve the issue described. Disclaimer: This post outlines a potential custom workaround for a specific use case or provides instructions regarding a specific task. The solution may not work in all scenarios or Sisense versions, so we strongly recommend testing it in your environment before deployment. If you need further assistance with this please let us know.How to Troubleshoot UI Issues Using DevTools (HAR File & Console Logs)
If a webpage, dashboard, or widget isn't loading properly, encounters errors during exporting, importing, editing, or opening, or if buttons are unresponsive and error messages appear, you can use Developer Tools (DevTools) in your browser to diagnose the issue. This guide will show you how to: - Check the Network tab for failed requests. - Use the Preview and Response tabs to see details of errors. - Check the Console tab for other issues. - Save a HAR file to share with support.
How to identify and download logs from the Sisense server
To identify and download ec-bld logs in Sisense, first, log into the Sisense server and run kubectl -n sisense get pods | grep bld during the cube build to find the relevant pod name. Once the build is complete, locate the log file in /var/log/sisense/sisense/ (for single-node) or use kubectl cp commands to retrieve logs from fluentd or management pods (for multi-node). Finally, access the Admin File Manager > Data folder to download logs and clean up local copies if needed.
Error on installation "cp: cannot create regular file '/usr/local/bin/kubectl': Text file busy"
The error "cp: cannot create regular file '/usr/local/bin/kubectl': Text file busy" typically occurs when multiple upgrade processes are running simultaneously, causing a conflict when trying to update the kubectl binary. [2024-10-30 12:33:34] Getting binaries kubectl (v1.30.3) and helm (v3.12.3) [2024-10-30 12:33:34] Downloading them from the internet % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 138 100 138 0 0 1367 0 --:--:-- --:--:-- --:--:-- 1380 100 49.0M 100 49.0M 0 0 3478k 0 0:00:14 0:00:14 --:--:-- 3525k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 15.2M 100 15.2M 0 0 3730k 0 0:00:04 0:00:04 --:--:-- 3731k linux-amd64/helm cp: cannot create regular file '/usr/local/bin/kubectl': Text file busy [2024-10-30 12:33:53] ** Error occurred during command sudo cp installer/05_post_infra/files/kubectl installer/05_post_infra/files/helm /usr/local/bin section ** [2024-10-30 12:33:53] ** Exiting Installation ... ** For example, in one case, the error occurred because multiple upgrades were happening on the same environment simultaneously (the same bastion for several cloud environments), which led to the Kubectl binary being in use by another upgrade process. The recommended solution is to check if any process is using Kubectl or Helm before proceeding with the upgrade. Watching an upgrade with "kubectl ... -w" command can cause this issue also. To prevent this error, it is advisable to: Ensure that no other upgrade or deployment processes are running in parallel. Use commands like lsof -t $(which kubectl) and lsof -t $(which helm) to check if these binaries are in use. If any command returns a PID, print out the process details using ps -ef | grep <pid number> and fail the pipeline if necessary. By following these steps, you can avoid the "Text file busy" error and ensure a smooth upgrade process. Related Content: https://academy.sisense.com/sisense-administration https://docs.sisense.com/main/SisenseLinux/upgrading-sisense.htm
