Managing Linux Build Service Settings for Optimal Performance and Stability

[Darwin]

Managing Linux Build Service Settings for Optimal Performance and Stability

An Elasticube build is a complex process that invokes multiple services.  This article explains some of the settings that are used for controlling the build service which affects performance and availability. The primary services used by the build process are connectivity, management, and build. This article is specific to the build service within the ElastiCube build process. Additional documentation is also available for Troubleshooting Memory Issues and Data Groups.

The correct values for these settings vary from deployment to deployment and depends on many factors. The primary factors are available memory, available CPU, and whether it is a single or multi-node configuration. Changes to these settings must always be within the context of the overall resources available across the system.

For an overview of the Sisense system configuration go to Admin>System Management.

Base Table Max Threads

The Base Table Max Threads setting controls parallel processing (threads), and is set to 4 by default. This setting can be changed to 1 temporarily for troubleshooting table builds without parallel processing. Correct settings will vary depending on CPU availability on the build node (if multi-node) while balancing against other processes that require system resources. This is an especially important setting if you are seeing any warnings or errors related to parallel processing.

Admin > System Management > Configuration (top right) > scroll down > Build. 

Maximum Memory Limit

The maximum memory limit setting controls the amount of JVM memory available for all currently running build processes combined. Changing this setting will cause a very short outage as services restart. 

Admin > System Management > Configuration (top right) > 5 clicks on Sisense Logo (top left) > General > System Configuration > scroll down > Build. 

Build Pod Memory and CPU Limits (Linux)

This controls resources available for the build service. If you increase the value too much you may not be able to schedule. Any value above 3000 is not recommended without Sisense consultation.

Linux > SSH > Kubectl -n <namespace> describe deployment build > Containers: build: Limits

To view this setting:

1. SSH to the Sisense server
2. View the build configuration:
   kubectl -n <namespace> describe deployment build
3. Find CPU and memory under Containers:build:Limits

To edit this setting:

1. SSH to the Sisense server
2. Make a backup of the current build deployment yaml
   kubectl -n <namespace> get deployment build -o yaml > build_deployment.yaml 
3. Edit the build deployment:
   kubectl -n <namespace> edit deployment build
4. Edit the memory limit to 1000 less than the Build MaxMemoryLimit setting:
   Find "resources: limits" and update memory size (press “i” on the keyboard to enter edit mode → edit value → ESC to exit edit mode → use “:wq!” to exit and save changes).
5. Build pod will be automatically restarted after deployment modification.

While these settings are the foundation for the build service, they are only a part of the overall build process.  Additional documentation is also available for Troubleshooting Memory Issues and Data Groups.

If you need additional help, please contact Sisense Support.

[dvuke]

Hello, 

Nice article @Darwin 

We're self hosting in a linux environment and currently run an ETL job hourly that makes requests to build ElastiCubes in Sisense when it's done moving data over to redshift.

We just recently upgraded from L2022.6 to L2022.7, didn't change any part of our ETL Process or any build configurations, just a simple upgrade, and our Sisense Server is slowly running out of memory and eventually becomes unresponsive/dead after a few days.  

Are there any known memory leaks in L2022.7 or any additional things we could look into to see what's happening?  You can see in graphana it's slowly consuming more and more and then gets erratic when less and less memory is available.  At this rate, it'll die.  

Thanks for any insight in advance,

Damian

[Darwin]

Hi @dvuke,

There are no known memory leaks in L2022.7.  Please submit a ticket for us to evaluate this.  If you would like you can mention me in the ticket for it to be routed, but all of the Sisense Support Team members are excellent at this type of evaluation.

Regards,

Darwin