on - edited on by
DRay
Logging and Monitoring in Sisense
TL;DR
This article covers the basics of the logging and monitoring infrastructure in Sisense on Linux.
What You’ll Learn
- What logging and monitoring information Sisense collects
- What tools Sisense uses
- How to access logging and monitoring tools
Use Cases For Applying What You Will Learn
- Check Sisense cluster health
- Access logs for troubleshooting
- Assess resource consumption and resource needs
Prerequisites
Basic knowledge of Linux; basic knowledge of Kubernetes will be a plus.
General concept: Logging and monitoring in Sisense
What does Siesense do in terms of logging and monitoring?
- Sisense collects logs from all its microservices so we could narrow down the issue if it arises and troubleshoot Sisense components one by one
- Sisense also collects live resources monitoring information like CPU utilization, RAM consumption, disk usage, network traffic, etc. on the computers (nodes) Sisense runs on
- In the end, Sisense makes logging and monitoring information accessible to clients and Sisense employees
Logging in Sisense
(image taken from https://sisenseglobal.atlassian.net/wiki/spaces/SAS/pages/20265339422/Service+-+Logging+-+Fluentd)
General overview of logging in Sisense:
- Sisense collects logs from all its services
- Sisense stores logs locally in /var/log/sisense/sisense
- Sisense sends out logs to external storages
How to access service logs?
ssh to Sisense server and cd /var/log/sisense/sisense/ (on a fluentd node)
Each .log file corresponds to a particular service except for combined.log, which stores all logs in one file
Technologies behind logging in Sisense
- Sisense uses fluent-bit to collect logs from Sisense services
- Fluent-bit forwards logs to Fluentd
- Fluentd records logs to /var/log/sisense/sisense
- Individual logs to files, which corresponds to service containers (query.log, build.log etc.)
- All logs to one combined.log file
- Fluentd sends out logs to external storage to store/analyze logs:
- To Amazon Kinesis and then to Logz.io
- To elasticsearch
Where can I find logging tools and how are they configured?
- Logging tools are installed in the Sisense namespace:
kubectl -n sisense get all | grep fluent
- Fluent-bit and fluentd configuration is done via configmaps:
kubectl -n sisense get cm | grep fluent
Is there a way to access logs externally and not directly in the system?
Yes, Sisense by default uses Logz.io for these purposes, but you also have the option to use Loki and Elasticsearch. The external log provider may be set during the installation or during a Sisense upgrade.
Use configuration file: "./kubespray/extra_values/installer/installer-values.yaml" from the installation package to set external log provider:
- LogzIO: set by default
- Loki: open the configuration file "./kubespray/extra_values/installer/installer-values.yaml" and set the configuration to "loki.enabled: true". Run the installation/upgrade
- Elasticsearch: open the configuration file "./kubespray/extra_values/installer/installer-values.yaml" and set the configuration "external_log_provider: elasticsearch". Run the installation/upgrade
Monitoring in Sisense
(image taken from https://github.com/srinisbook/Prometheus-grafana)
General concepts of monitoring in Sisense:
- Sisense collects monitoring information like CPU utilization, RAM consumption, disk usage, network traffic, etc. directly from the computers (nodes) Sisense runs on
- Sisense stores monitoring information in a database
- Sisense makes the monitoring information available to users by providing a graphical interpretation of the monitoring data
How to access monitoring data?
Log into the Sisense application and go to Admin > System Management > Monitoring
The monitoring app, Grafana, will open in a separate tab.
How to switch between monitoring data?
Click on Dashboards in the left menu and select “Browse”.
You will see the list of available dashboards, each with their corresponding metrics
Technologies behind monitoring in Sisense
- Sisense uses Prometheus to collect the monitoring data
- Prometheus:
- collects metrics
- executes rules to filter out metrics
- saves results to the database
- Sisense uses Grafana to present monitoring data
- Grafana connects to the Prometheus database, reads metrics, and presents data on dashboards with widgets similar to Sisense dashboards
Where can I find logging tools and how are they configured?
- Monitoring tools are installed in the Monitoring namespace:
kubectl -n monitoring get all
Additional reading about Sisense Logging/Monitoring:
If you need additional help, please contact Sisense Support.
