Performance Improvements On The Single-Node Linux Servers Under Heavy Usage
Symptoms Relevant for Linux Single-node Sisense In a default Sisense single-node deployment, there is one pod per service deployment. In cases of heavy usage, which is usually caused by parallel requests, the number of services processing the request could result in slowness or a delay when accessing dashboard objects, opening admin pages, etc. Diagnosis A good test would be to compare how the system behaves under heavy load and during non-working hours. If you see a difference in the time it takes to load the dashboard layout or access pages in admin, this could be caused by the amount of the load placed on the workers of each pod. This should not be an issue in multi-node environments as services are already scaled up, but in a single node environment, if there is enough RAM/CPU to handle the load, Sisense can scale services to boost the performance of parallel requests. Important services include: api-gateway - webserver service. Takes a request first and communicates with other requests. galaxy - service which serves the object of the dashboard, widgets, data security, shares, navigation panel, and alerts. identity - service which provides detail of users, groups, and authentification. configuration - service which provides system settings and configuration Solution If you have enough RAM/CPU resources on the server, you can scale services by running the commands below. Please note that each additional pod replica will consume up to 8.5 GB of RAM, so keep this in mind when scaling. Below is a command to double the amount of services. Remember to change the namespace and to set the amount of --replicas to the correct amount. kubectl -n <namespace> scale deployment identity --replicas=2 kubectl -n <namespace> scale deployment galaxy --replicas=2 kubectl -n <namespace> scale deployment api-gateway --replicas=2 kubectl -n <namespace> scale deployment configuration --replicas=2 The above steps can help if you notice common api-gateway restarts caused by the OOMemory. Sisense's RnD team is working on a solution to the problem, but in the interim, scaling the api-gateway can help prevent service disruptions.Demonstrating ComposeSDK Styling Of Sisense Dashboard Widgets
Demonstrating ComposeSDK Styling Of Existing Sisense Dashboard Widgets When developing ComposeSDK embedded applications, there are three principal techniques for embedding or converting an existing Sisense widget. Discussing each method, and some of the benefits associated with them: Direct Rendering existing Sisense Dashboard Widgets with the DashboardWidget Function: The expedited approach involves rendering an existing Sisense widget directly using the DashboardWidget function. While this method ensures swift integration, it does not allow all data options of the widget to be edited within ComposeSDK or defined as variables, one of the features of native ComposeSDK widgets. Nevertheless, customization of other parameters, including styling, remains feasible based on the props used. Mandatory parameters include the dashboard and widget ID, which allow the rendering of an existing Sisense widget. Widget Recreation using ComposeSDK to Create Native ComposeSDK widgets An alternative method, detailed in this article, entails recreating the widget as a native ComposeSDK widget by employing an existing Sisense widget as a template for a new not directly linked ComposeSDK widget. Though potentially more time-intensive, this approach yields a fully customizable native ComposeSDK widget. It stands independent of any specific widget or dashboard on the Sisense server, enabling complete independence to changes made to the original dashboard or widget, or complete deletion of the widget used as the model. Loading Widget Metadata with the useGetWidgetModel function Leveraging the useGetWidgetModel ComposeSDK function provides a middle option, detailed in this article. It allows automating the return of widget metadata from an existing Sisense widget, facilitating dynamic modifications within ComposeSDK. This method balances somewhat the autonomy of entirely recreating a widget as a native ComposeSDK widget and rendering a widget as a Dashboard Widget. In this article, we will demonstrate and discuss the DashboardWidget rendering feature, a powerful capability within ComposeSDK that allows the embedding and rendering of existing widgets. The focus will be on exploring the large number of styling options provided by this feature. Among the properties of a DashboardWidget component, the styleOptions parameter determines the styling. This parameter accepts a DashboardWidgetStyleOptions object, which includes a large number of parameters documented in detail below and in this documentation page. To demonstrate a practical implementation of using these styling options, the following code example showcases a Compose DashboardWidget utilizing all documented parameters within the DashboardWidgetStyleOptions object. import { DashboardWidget } from "@sisense/sdk-ui"; export function ComposeSDKChart(props) { let styleOptions = { "backgroundColor": "lightblue", "border": true, "borderColor": "green", "cornerRadius": "Large", "shadow": "Dark", "spaceAround": "Large", "header": { "hidden": false, "titleAlignment": "Center", "backgroundColor": "lightblue", "titleTextColor": "blue" }, "height": 200, "width": 200 }; return (<DashboardWidget widgetOid={'65ab8958857ff900335db870'} dashboardOid={'65ab8948857ff900335db86e'} styleOptions={styleOptions} title={"My Chart"} />) } The final result in this example is the widget below, showing the effect of this styling in this specific case: Discussing each style option individually, including the type of parameter accepted by each style setting. backgroundColor: Type: string - Specifically a string containing an HTML color Description: This option sets the background color of the widget. In the example, the background color is set to "lightblue," a preset standard HTML color code that is equal to #ADD8E6 as a hex color code. border: Type: boolean Description: This option determines whether the widget container has a border or not. In the example, the border is enabled with the value set to true, adding a defined boundary to the widget that can be styled. borderColor: Type: string - Specifically a string containing an HTML color, Description: Specifies the color of the widget container's border. cornerRadius: Type: "Large" | "Medium" | "Small" - Three specific options set as specific strings Description: Defines the corner radius of the widget container, similar to the CSS property of the same name, allowing for customization of the widget container's shape. header: Type: object- Contains various parameters that control the styling of the header and title. Description: A style object to customize the widget header. It includes options like background color, divider line toggle, divider line color, header visibility toggle, title alignment, and title text color. header.backgroundColor: Type: string - Specifically a string containing an HTML color Description: Sets the background color of the widget header. header.dividerLine: Type: boolean Description: Controls the visibility of the divider line between the widget header and the chart. header.dividerLineColor Type: string - Specifically a string containing an HTML color Description: Specifies the color of the divider line, if visible. header.hidden Type: boolean Description: Toggles the visibility of the header and title. header.titleAlignment Type: "Left" | "Center" - Two specific options set as specific strings Description: Alignment of the title within the header. The example centers the title using the value "Center. header.titleTextColor Type: string- Specifically a string containing an HTML color Description: Specifies the text color of the header title. height: Type: number - Number in pixels Description: Sets the total height of the widget in pixels. If not explicitly set will use the height of the container. width: Type: number - Number in pixels Description: Sets the total width of the widget in pixels. If not explicitly set will use the width of the container. shadow: Type: "Medium" | "Light" | "Dark" - Three specific options set as specific strings Description: Defines the shadow level of the widget container, similar to the CSS style of the same name. Effects styling only when spaceAround is defined. spaceAround: Type: "Large" | "Medium" | "Small" - Three specific options set as specific strings Description: Specifies the space between the widget container edge and the chart. With these styling options, one can modify the visual styling of a ComposeSDK DashboardWidget, enabling a large amount of style customization for an otherwise unchanged existing dashboard widget selected from an existing dashboard on the Sisense server. While ComposeSDK DashboardWidgets may not offer the extensive modification capabilities available to native ComposeSDK widgets, such as the ability to completely modify all of widget data options, they still provide a large degree of extensive visual customization through the use of styling options. Share your experience in the comments!Sisense Kubernetes Cluster Health Check
Sisense Kubernetes Cluster Health Check Check the pods status 1. Check if there are pods that are not in a Running or Competed state. kubectl get po -A -o wide | egrep -v 'Running|Completed' -A is used to get pods from all namespaces (Sisense is usually installed in sisense one) -o wide is used to get the extended output The response should be empty: 2. If there is no output (all pods are in a Running or Competed state), check if all containers of the Running pods are READY. kubectl get po -A -o wide | egrep 'Running' You should be looking for numbers in the READY column, x/y, where x is the number of ready containers and y is the total number of containers. Please note if x is less than y, then not all containers are ready! Please refer to the sections below for instructions on troubleshooting this issue. 3. If all pods are in a Running or Completed state, and all Running pods containers are READY, check the status of the nodes: kubectl get nodes. All nodes should have the status ‘Ready.’ If it is a single-node environment, then you will see just one node. If a multi-node environment, then you should see several nodes. 4. If the node is not in a ‘Ready’ state, get details by ‘describing’ the node: kubectl describe node <node-name>. 5. You may also check storage health by running: kubectl -n sisense get PVC. 6. If all pods are in a Running or Completed state, all Running pods containers are READY, and all nodes are in a Ready state, the basic Kubernetes troubleshooting is complete, and the issue is not in the Kubernetes infrastructure. What if kubectl is not running? 1. If Linux doesn't recognize the kubectl command, there is an issue with the Kubernetes installation, or the user doesn’t have permissions to run kubectl: 2. The main Kubernetes component is kubelet. Check the status of the kubelet service (does not apply to RKE deployments): systemctl status kubelet It should be in an active/running state. 3. If the kubelet is not in an active (running) state, try restarting it 4. You may check kubelet logs by running: journalctl -u kubelet Click shift+g to go to the end of the list 5. If the kubelet is missing, then there is an issue with the Kubernetes installation 6. If the kubelet is in an active (running) state, check if there is a .kube directory in the home directory of your current user: cd && ls -la .kube If the .kube directory is missing or empty, the current user is not configured to run kubectl, and there is a problem with the Kubernetes configuration. What if the pods are not running correctly? 1. If you have a meaningful output, but you have pods in a state other than Running or Completed, or not all containers are READY like the below screenshot, then you will have to describe the pod to understand the reason why it is unhealthy. 2. For example, you have a pod with 0/1 containers READY: Assuming the pod is in the sisense namespace, copy the name of the pod, in this case, ‘external-plugins-5dcf494b77-gtsfk’, and run kubectl -n sisense describe pod external-plugins-5dcf494b77-gtsfk The output will look like this: The two main sections we are interested in evaluating are Conditions and Events. Conditions will give you True/False values if the pod is: -Initialized -Ready -ContainerReady -PodScheduled (pod is placed on the node) In the example above, the pod had been placed on the node (PodScheduled: True), but it is not ready, although it has been initialized, because its container is not Ready. Events will give you an excerpt from the kubelet log showing events related to the current pod. In the example above, the Readiness probe for the pod failed, so the problem is in the application itself and not in the Kubernetes infrastructure. 3. You can check the logs of the pod with the command: kubectl -n sisense logs external-plugins-5dcf494b77-gtsfk and look for errors given the clue about the root cause of the problem. 4. The container may be in a state other than Running: Use describe pod to check its Conditions and Events as we did in the previous case: kubectl -n sisense describe po external-plugins-5dcf494b77-gtsfk 5. If conditions and events don’t give you enough information about the root cause of the problem, look at the State/Last state section: In the example above, the Last State is ‘Terminated,’ and the Reason is OOMKilled, which means Out of Memory, Killed. This means that the Kubernetes has killed the container because the latter has exceeded the memory limit. To increase the memory limit, find the problematic pod: kubectl -n sisense get po Then find the Kubernetes object managing the pod. In our example kubectl -n sisense get all | grep connectors Find a resource without additional random letters/digits in the name: In our case, it’s a deployment. Then edit the resource: kubectl -n sisense edit deployment connectors And search for resources: Increase the ‘limits’ for the ‘memory’ in our example. 6. Let’s consider another example: We have a pod in a CrashLoopBackOff state. Let’s describe the pod: kubectl -n sisense describe po sisense-dgraph-alpha-0 It doesn’t give us anything obvious. Let’s check the logs: kubectl -n sisense logs sisense-dgraph-alpha-0 (add –previous if you don’t have any output). You can see that the root cause of the issue is the fact that there is “no space left on device.” This means we should allocate more space to the pod. In this case, you may check the status of persistent volumes and persistent volume claims with kubectl get pv and kubectl -n sisense get PVC. You are looking for statuses other than Bound: If you see a status other than Bound, you may “describe” the resource: kubectl -n sisense describe pvc data-dgraph-0 as in the case above. What else to check? 1. If the cluster looks healthy, but performance suffers, you may check the resource consumption by Sisense services. Start with checking nodes. kubectl top nodes The output should look like: Note if the CPU% is close to 100% or memory% is close to 85% 2. To check the resource consumption of the individual pods run kubectl -n sisense top po Note the pods with abnormally high memory consumption: Conclusion In conclusion, if you are a Kubernetes pro then this article will help you quickly grasp what infrastructure components are involved in Sisense deployment and what to check next. If you are a Kubernetes newbie, these basic instructions will let you troubleshoot the issue quickly and identify the issue to seek further help. If you need any additional help, please contact Sisense Support.Sisense Configuration Tips
Sisense Configuration Tips What You’ll Learn In this course you will learn how to: Export Sisense configuration Search for settings in the configuration Change the configuration settings Compare different configurations Import configuration Use Cases For Applying What You Will Learn The following are some scenarios where this content can be applied when working with Sisense configurations: You see the operation in the logs hitting the timeout equaling 300000, but you are not sure if there is a setting for such a timeout in the configuration. You are performing an upgrade and want to check if the upgrade affected some of your configurations. You want to back up the configuration. You want to compare the configurations of two different environments. Prerequisites Familiarity with Sisense UI Basic knowledge of Linux Basic knowledge of Kubernetes Configuration Types In Sisense Sisense configuration settings may be logically divided into a couple of main categories: Fixed: Those that are ‘fixed’ for the current deployment like the ones you set during Sisense's installation. Configurable: Those that can be changed using the Configuration UI or the SI Client commands, without having to reinstall or update the Sisense application. Add-ons: Those that are not directly related to the core application, but apply to add-ons like plugins or connectors In this article, we will focus on Configurable settings that can be accessed from the Admin tab > System Management > Configuration To get access to the full configuration menu, click on the Sisense logo 5 times: Services Responsible For Configuration Management Two services in Sisense are responsible for configuration management: Zookeeper stores the configuration settings. Configuration is responsible for manipulating configuration parameters and providing the configuration data to other services using ConfigurationSDK. Configurations are set when the configuration service starts. The service sets all of the base (shared) configurations under the base configurations path in Zookeeper. All other services copy their required configurations during boot time using the configuration manifest file each service has stored using the Sisense-configuration SDK. Base Vs Specific Configuration The base configuration is shared between services. For example, if you change the log level in the base configuration it will change for all the services. Specific configuration settings for services affect those services only. You should be mindful of the Base and specific configuration when setting configuration parameters using tools outside of the UI! Hidden UI Configuration “File Explorer” As we already learned, we cannot change the base configuration parameters for specific services using the standard UI configuration menu. But what if we want to change the log level just for the Galaxy service? The hidden UI configuration, “file explorer” will come in handy. It will display all available Zookeeper configuration parameters. To access the hidden configuration “file explorer” click on the General section header 5 times. The menu will expand and then, you will be able to see the File Explorer sub-menu Find the galaxy service folder in the file tree. From there you will be able to see the base configuration settings for galaxy services: Please note, if you change the base configuration setting, it will reset the individual value. Changing Configuration Settings There are 4 ways to change configuration settings: From the UI regular menu Admin > System Management > Configuration From the UI hidden file explorer (see above) From inside the Zookeeper pod using zkCli.sh Using SI CLI commands Using the UI and hidden file explorer is quite straightforward, so below is a review on how to use zkCli.sh and SI CLI commands. From Inside The Zookeeper Pod Using Zkcli.Sh Exec into Zookeeper: kubectl -n sisense exec -it $(kubectl -n sisense get po -l=app=zookeeper -o name) -- bash Run zkCli.sh You will see the prompt: help — to see the help with all commands ls — to list configurations, Sisense configurations are located at ls /Sisense/S1/configuration/production/ get — to get the particular value for example get /Sisense/S1/configuration/production/translation-service/SwitchOffSameGuids set — to set the particular value in the format, set the parameter value. For example, set /Sisense/S1/configuration/production/translation-service/SwitchOffSameGuids true Using SI CLI commands si configuration is responsible for manipulating the configuration settings: get — to get the particular value in the format of si configuration get -key category.entry -type key set — to set the particular value in the format of si configuration set -key category.entry -new-value 1412 Read more about SI commands in Sisense’s documentation site: Using Sisense CLI Commands. Exporting Configuration Settings: Say you see in the logs that the request is hitting the 300000 msec timeout threshold, but you don’t know where the timeout is set and if there is such a timeout setting at all. It would be great to search the entire configuration set for the value 300000 and see if anything comes out, but the UI doesn’t provide that functionality yet, since you can only search per service. Please note, that at this time you cannot search through all configuration settings via the comprehensive configuration menu nor in the file explorer. To perform a global search in all configuration settings, we can export the configurations and perform the search in the output file. Remember that configurations are stored in Zookeeper while the configuration service is responsible for interacting with the Zookeeper database. Therefore, we will use configuration commands to perform the export. 1. Run this command to get the Zookeeper service ClusterIP: kubectl get svc -A | grep zookeeper. 2. Put the IP in this command (but don’t run it yet!). node /usr/src/app/node_modules/@sisense/sisense-configuration/bin/sisense-conf export -p -c 10.43.198.82:2181 3. Then exec into the configuration pod. kubectl -n sisense get po | grep configuration kubectl exec -it -n sisense configuration-84777bdfd9-n2wn5 -- /bin/bash 4. Then run the above command (with the correct cluster IP). It will dump the output into sisenseConfiguration.json, then run exit. 5. Copy sisenseConfiguration.json out of the pod, to the host, so you could work with the file kubectl -n sisense cp configuration-84777bdfd9-n2wn5:/usr/src/app/sisenseConfiguration.json sisenseConfiguration.json Or simply run this command from the host, which will do everything for you. kubectl -n sisense exec -it $(kubectl -n sisense get po -l=app=configuration -o name) -- node /usr/src/app/node_modules/@sisense/sisense-configuration/bin/sisense-conf export -p -c sisense-zookeeper:2181 > sisenseConfiguration.json It will put sisenseConfiguration.json into the folder you are running it from. Now you can search for the 300000 value in the entire configuration. To perform an extensive search, you can use your favorite editor like vim or nano to open the file or use the Linux command “less”. Importing Configuration Settings Copy sisenseConfiguration.json to the new environment and to the configuration pod. Then exec into the configuration pod on the new environment using the following command: kubectl exec -it -n sisense configuration-84777bdfd9-n2wn5 -- /bin/bash and run: node /usr/src/app/node_modules/@sisense/sisense-configuration/bin/sisense-conf import -p -c sisense-zookeeper:2181 -i sisenseConfiguration.json Comparing Configurations Of Different Environments Say you want to compare configurations of two different environments or compare configurations before and after the upgrade to make sure nothing has changed. Now it is easy to do! You can export configurations into files and compare files using diff or your favorite online text-comparing tool like https://text-compare.com/ Summary I hope this article helps you perform searches in Sisense’s configuration, compare configurations and check if a configuration setting exists at all in Sisense. Using this article, we hope to empower Sisense users and show that users are not limited by UI limitations when working with configurations.
