Prometheus storage & retention changes for Sisense EKS monitoring [Linux]

Step-by-Step Guide: 

1. Adjust the Prometheus Custom Resource (CR)
The Prometheus CR controls runtime configuration, including retention and storage settings. Updating the CR applies changes immediately.

• View the existing Prometheus CR:
  kubectl -n monitoring get prometheus sisense-prom-operator-kube-prometheus -o yaml
• Edit the CR to update retention and PVC size:
  kubectl -n monitoring edit prometheus sisense-prom-operator-kube-prometheus
• Update the following values under spec:
  retention: 60d
  retentionSize: 50GB
• Increase the persistent volume claim size:
  storage:
  volumeClaimTemplate:
  spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: gp3
  resources:
  requests:
  storage: 100Gi
• Save the file. The Prometheus Operator will automatically expand the PVC, update the StatefulSet, and restart the Prometheus pod.
• Verify the PVC size:
  kubectl -n monitoring get pvc | grep prometheus

2. Make changes persistent across Sisense upgrades
Sisense upgrades regenerate monitoring resources based on Helm templates. To ensure the changes are preserved, the new values must be added to the extra values file.

• Open the extra values file:
  sisense-L2025.3.0.397/installer/extra_values/helm/kube-prometheus-stack.yaml
• Add the following content:
  prometheus:
  prometheusSpec:
  retention: 60d
  retentionSize: 50GB
  storageSpec:
  volumeClaimTemplate:
  spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: gp3
  resources:
  requests:
  storage: 100Gi
• Save the file. These settings will now override defaults during future upgrades.

Disclaimer: This post outlines a potential custom workaround for a specific use case or provides instructions regarding a specific task. The solution may not work in all scenarios or Sisense versions, so we strongly recommend testing it in your environment before deployment. If you need further assistance with this, please let us know.