Assessing the Quality of Your Dashboard
Introduction The following article discusses how to assess the quality and adoption of your dashboard. Table of Contents How to Measure a Dashboard's Quality? Let's begin by defining what a "Good Dashboard" is. To assess the quality of your dashboard, you should measure the following aspects: Is the dashboard practical? Does it serve its purpose? Does the dashboard display relevant and correct information? Is the information in the dashboard displayed correctly? Was the dashboard adopted by the end-users? Is the dashboard intuitive for use? Planning Makes Perfect Correct dashboard planning is the key to its success! I recommend you read this article discussing a dashboard's (high-level) development cycle. It breaks the process into easy measurable steps that start from the initial KPI planning to maintaining and adjusting your end product. Reading through it might help identify flaws during the dashboard's initial rollout process. Assessing a Dashboard's Quality The "Practical" Test The practicality of a dashboard focuses on whether the dashboard serves its purpose. Going back to a dashboard's planning phase, try to recall the reason for building this dashboard. What were the end-users looking to achieve? What insight were they seeking? Possible answers are: Monitoring a person's or a group's achievements Monitoring a measurable process Help decide what to do next based on historical information Assess future steps in various scenarios (What-If Analysis) A good dashboard: Answers its purpose! How to Test This? To check whether your dashboard is practical, you should: Set up a call with one (or more) end-users and/or stakeholders using the dashboard Ask them whether they can make the intended decision based on it If they are - What is the process of making the decision? If they aren't - What are the obstacles that stand in the way of making it? (e.g., missing/incorrect data, bad user experience, etc.) Next, revisit the dashboard's goals: What question should it answer? What action can the users make based on it? Compare the information you got from your stakeholders and the plan made when designing the dashboard - Define the gaps. How to Resolve This Type of Issue? Work towards re-planning your dashboard to meet its goals. Focus on: Aligning the "Call for Action" Redefining the KPIs and the user story Redesigning and restructuring the KPIs Optimizing/adjusting the data model to be able to answer the different KPIs Resolving this type of issue will most likely require a complete dashboard redesign, followed by an entire UAT cycle. The "Data Relevance" Test The term "Data Relevance" breaks into two aspects: Am I displaying relevant data - Considering the audience of the dashboard? Am I displaying relevant data - Considering the decision to be made? Does It Address the Right Crowd? Like any deliverable, one of the fundamental things to know is its audience. The answer to this question will affect the following: The type of dashboard you create The granularity of data you present The way information is presented A good dashboard: Is built with the right audience in mind. Does it Contain the Right Amount of Data? Making a decision requires having the right amount of data: Too little data - Could prevent the person from making the right call. Too much data - Could confuse the person and throw them off-track. A good dashboard: Has the right amount of data required to serve its purpose. How to Test This? To check whether your dashboard has relevant data, you'll have to: Assess who are the end-users and/or stakeholders using this dashboard Set up a call with one (or more) end-users and/or stakeholders using it Ask them what the process of making their decision is? Check whether the information they require is presented in the dashboard. Check whether the dashboard has redundant or irrelevant data. How to Resolve This Type of Issue? Incorrect Dashboard Type Check your dashboard's type - Based on the person using this dashboard, should it be operational, analytical, tactical, or strategic? Compare the dashboard type to the intended crowd to determine if it was designed correctly (for example, a "Tactical Dashboard" is aimed towards upper management and will usually present long-term KPIs and high-level metrics). Resolving this type of issue will most likely require a complete dashboard redesign, followed by an entire UAT cycle. Missing/Redundant Information If you've identified that certain information is redundant or missing, consider adding, modifying, or removing widgets from the dashboard. Doing so might be a simple task (especially when removing data). However, it might also end up as a more extensive project requiring a partial redesign of the dashboard and a partial/complete UAT cycle. The "Data Correctness" Test Presenting inaccurate or outdated data is worse than displaying partial or excessive data. Two possible outcomes of stakeholders, basing their decisions on incorrect data, are: Experiencing negative events (such as revenue loss) Compromising the trust relationship between Sisense and the end-users Frequent causes for showing outdated data in your dashboard include: An ETL process that is not run frequently enough An ETL process that often/occasionally fails An ETL process using an incorrect table update behavior (e.g., "Accumulative" where it should be "Full") An ETL process that isn't synchronized with your Data Warehouse A source database being offline for an extended period Frequent causes for showing incorrect data in your dashboard: Wrong data modeling leading to unexpected Many-to-Many relationships Business questions that don't align with the data model (e.g., causing "Random Paths") Unexpected / Missing inheritance of filters Wrong data security configuration The use of multiple data models (in the same dashboard) built at different schedules A good dashboard: Displays precise and up-to-date data How to Test This? To check whether your dashboard has correct data, you'll have to: Assess the different widgets on the dashboard to see the granularity of data displayed For each widget, check what data model it relies on - Use the Usage Analytics "Usage - Builds" dashboard to monitor the behavior of historical builds. Set up a call with one (or more) end-users and/or stakeholders using this dashboard Ask them if they trust the data on the dashboard and if the data refresh frequency is sufficient. How to Resolve This Type of Issue? Showing Outdated Data Ask your stakeholder how frequently they expect the data to be refreshed. Perform the following actions: Check the data model's build frequency - Should the build frequency be modified? Check the build time - Should the data model be optimized? Is the data model too heavy? Check the build success rate - Why are builds failing? Is the system running low on resources? Should "Data Groups" be applied? If using a Data Warehouse (DWH) and an Elasticube - Check the DWH build frequency; Check the synchronization between the DWH ETL finish and the Sisense ETL beginning. If you can't achieve the required build frequency, check whether all widgets require the same data refresh rate - Can a "Live Model" or a "Hybrid Dashboard" be considered? Showing Incorrect Information Ask a stakeholder to generate a report with correct data or ask them to point you to "defective widgets." Perform the following actions: Check the formula behind the figure(s) showing the incorrect data and correct them. Consider the different filters, inheritance behavior, etc. Examine the query path used to calculate each figure (look out for unexpected "Many-to-Many relationships" or "Random Paths") - Use the "Visualize Queries" add-on (Link) to compare Sisense behavior against the expected query path. Run the calculation in the data model using a custom table and a SQL statement that emulates the dashboard's calculation. The "Visual Correctness" Test Data alone isn't enough; it must be visualized correctly. Visualizing data requires using the correct widget type and visual aids to convey the message. For example, say a KPI is showing the company's revenue. See the different visualization options below: Option #1 Option #2 Option #3 Transitioning from option #1 to #2 provides an added value of "Good" / "Bad" Transitioning from option #2 to #3 provides an added value of revenue ranges A good dashboard: Has widgets that are visualized correctly and convey a clear message. How to Test This? To check whether your widgets are visualized correctly, you'll have to analyze each widget individually: Categorize each widget to find out its aim (e.g., show a single figure, compare values, show behavior over time, visualize data to show distribution, etc.) Verify the visualization matches the widget type (e.g., An indicator widget is perfect for displaying a single figure, a line chart is ideal for showing a behavior over time, etc.) Make sure each widget has visual aids (such as conditional formatting) to convey its message clearly - Each widget should tell a small puzzle of the story (which the dashboard should put together to a larger picture) Set up a call with people who are not familiar with the dashboard Ask them to describe each widget (separately) and their conclusion from looking at it. How to Resolve This Type of Issue? Wrong Widget Type Resolve the issue by fixing the visualization type - Use the following chart to help out: Type Use Case Indicator Show a single figure (numerical) Show a single figure and a gauge representing its range Column Chart Show a comparison among different sets of data Track data sets over time Track individual values + Their sum (stacking) Bar Chart Compare many items Track individual values + Their sum (stacking) Line Chart Reveal trends, progress, or changes that occur over time The data set is continuous rather than full of starts and stops Area Chart Displaying absolute or relative (stacked) values over a time period Analyzing a Part-to-whole Relationship Pie Chart One static number, divided into categories that constitute Represent numerical amounts in percentages Table Display RAW granular data Pivot Display RAW granular data Display aggregative data in a table format Scatter Plot Comparing large numbers of data points without regard to time Identify a potential relationship between two variables Calendar Heatmap Show relative number of events for each day in a calendar view Missing Visual Aids Visual aids help the end-users identify "Good" or "Bad" values. Possible visual aids include: Adding colors to numerical labels (e.g., Green figure vs. a red one) Adding background colors to table cells (e.g., Color negative cells red) Adding a line chart representing a threshold (e.g., A red line to indicate a lower threshold) Converting numerical values with text (e.g., Showing a "Revenue increased" label rather than a positive figure) Adding trend lines Adding a forecast range The "Intuitiveness" Test Dashboard intuitiveness refers to the ability of a non-technical person to: Access the dashboard Understand and conclude the dashboard Understand how to customize the dashboard (using filters, drilling, etc.) Easy Access to Data There are two methods of accessing a dashboard: Sisense offers a complete HTML5 platform (a.k.a. Sisense Web Application) that can be white-labeled and customized to meet the company's look & feel (i.e., color pallet, logo, links to internal support and documentation pages, etc.). Sisense offers three different embedding deployment options (conventional iFrames, an embedding SDK, and a complete JS-based embedding solution). Embedding the dashboard (or individual widgets) allows infusing analytics into web pages and applications. Choosing the proper access method will affect how end-users access data and benefit the BI solution. To improve intuitiveness - Make sure to streamline the process of consuming data as much as possible. The need to switch between multiple applications will result in a lack of efficiency, a complex adoption, or even a lack of adoption. Ease of Comprehension The comprehension of individual widgets was discussed earlier. However, is "the whole" greater than the sum of the parts? Does the dashboard tell a story? An excellent example for a dashboard: Widget #1 shows the revenue is low. Widget #2 shows a revenue breakdown per department and points out one department losing money. Widget #3 provides an income/expanse category breakdown and points out non-proportional marketing expenses. Widget #4 provides a detailed transactional income/expanse breakdown and points out the individual expenses Customizability and Interactivity Stale reports tell one story. However, a person looking at the dashboard may want to filter, sort, and pivot the data to tell the same story about a specific segment or individual. The tools for customizing/interacting with a dashboard include: Filtering data to a specific segment of interest (Link) Drilling into a measure to be able to extract more information about it (Link) Adding explanations (Link) and narratives (Link) A good dashboard: Is easy to access, self-explanatory, easy to customize, and play around with. How to Test This? To check whether your dashboard is interactive, you'll have to: Set up a call with one (or more) end-users and/or stakeholders using it Ask them when they use this dashboard and their workflow of consuming its information (e.g., A customer requires the data when building a financial report, and his workflow includes opening another browser tab and logging into the Sisense Web Application). Open the dashboard and ask them to explain it. Try to identify widgets that were over-explained and widgets they skipped. Track the questions you have to ask to understand what you see. Track the amount of "mouse scrolling" they perform when explaining the dashboard flow. Ask them what interaction they have with their dashboard and lay out the tools to increase interaction. Note the tools they are interested in (e.g., extra filters, adding narratives to widgets, etc.) How to Resolve This Type of Issue? Data is Hard to Access Ease of access is easy to measure - Count the number of clicks the user has to make when they want to consume the data in this dashboard - Fewer clicks = Easier to access. If data is hard to access or requires too much "Clicking around": Integrate SSO or WAT to prevent the user from logging in to the Sisense Web Application Embed the dashboard (or a single widget) into the application/webpage Have additional data imported to Sienese - Allowing the user to have a "One Stop Shop" for their entire workflow Enable sending periodic reports to the user's email Enable NLQ to allow the user to interact with Sisense easily (Link) Infuse data into G-Suite applications (Link) Enable Pulse alerts to send push notifications to the user (Link) Data is Hard to Comprehend If your widgets tell the right story, but the complete picture doesn't make sense: Move widgets around to make the story clearer Add visual separators between certain widgets (requires scripting) Reduce the number of widgets by splitting your dashboard into multiple stand-alone dashboards Use add-ons to simplify dashboard navigation (e.g., Accordion) Bring in a UI/UX designer to help visualize correctly Redesign the dashboard to avoid scrolling the mouse The Dashboard isn't Customizable / Interactive Make the dashboard more customizable by allowing the user to: Filter data based on predefined filters Define hierarchies to make filtering more intuitive Integrate filtering abilities into the dashboard (e.g., BloX buttons, drilling options, clickable widget values) Add additional widgets (e.g., Accordion, Switchable Dimensions, Tabber, etc.) Add premium widgets (e.g., Advanced Input Parameters - Link) The "Adoption" Test So you've created a great dashboard; it has all the correct data and visualizations, a person can access it quickly and draw the proper conclusion by just looking at it, but it wasn't adopted. Has the company done enough to drive its adoption (e.g., sufficient training, decommissioning the old dashboard, integrating it into the users' application, etc.)? A good dashboard: Is measured by its adoption and usage. How to Test This? To check whether your dashboard has correct data, you'll have to: Find out who this dashboard's audience is Use the Usage Analytics "Usage - Dashboards" dashboard to monitor who accesses this dashboard and how often. Set up a call with one (or more) end-users and/or stakeholders who are not using this dashboard Find out why this dashboard isn't being used. Find out what alternative ways they use to collect the data How to Resolve This Type of Issue? Once you're sure the dashboard is perfect and the only thing missing is adoption: Involve users in the design meetings and UAT loop to make them feel they are part of the process Create an adoption plan! Get executives to buy-in - Causing them to promote their usage Monitor what people are using the dashboard and which aren't - Target the right people Reeducate your end-users on the benefits of the dashboard and the value of using it Decommission old dashboards and reports Refresh your dashboards from time to time (design, contents, etc.)
How to configure Data Groups
Introduction This article will guide you on how to configure the Data Groups’ parameters based on the current performance and business requirements. What are "Data Groups”? The “Data Groups” administrative feature allows configuring Quality-Of-Service (QoS) to Sisense instances and limiting/reserving the resources of individual data models. By utilizing data groups you’ll make sure available resources are controlled and managed to support your business needs. Why Should One Configure “Data Groups”? The main benefits of configuring “Data Groups” are: Controlling which cluster nodes are used for building and querying Limiting the amount of RAM and CPU cores a data model uses Configuring indexing, parallel processing, and caching behavior of a data model Mitigating “Out-Of-Memory" issues caused by lack of resources Preventing “Safe Mode” exceptions caused by lack of resources Read this article to learn about the different “Data Groups” parameters “Data Group” Use Cases The following two scenarios are good examples of using "Data Groups”: Scenario A customer embeds Sisense (OEM use case) and has many tenants that can design dashboards (a.k.a. Self-Service BI). Each tenant has a dedicated Elasticube data model. Challenge A single tenant is capable of creating a large number of dashboards - Causing the underlying data model to utilize a significant portion of the available CPU/RAM resources. The excessive use of resources by a single tenant may lead to resource depletion and degrade the user experience for them and other tenants. Solution Resource Governance – Set up resource utilization limitations using "Data Groups”. Doing so will limit each tenant’s usable resources and prevent tenants from affecting each other. Scenario A customer has many Elasticube data models created by various departments of the organization. Some of the data models are used to generate high-priority strategical dashboards (used by C-Level managers). Other dashboards are prioritized as well (e.g., operational dashboards vs. dashboards used for testing) Challenge C-Level dashboards must have the highest priority and should always be available. Other dashboards should still be operational and behave based on their priority. In case of conflict or a temporary lack of resources, a critical Elasticube may run out of memory or trigger a ‘safe mode’ event. Solution Resource Governance – Setting up priority-based Data Groups would result in allocating business-critical data models with more resources and limiting the less critical ones. Data Groups Resource Governance Strategies Limiting a Data Model’s Resources - Governance rules can be used to limit the resources used by a single data model or a group of multiple data models. This can be done by configuring a "Maximal” amount of allocated CPU/RAM. Note that the data model would be limited to the configured resource restrictions even though additional resources are available for use. Pre-Allocating a Data Model’s Resources - Governance rules can be used to pre-allocate the resources used by a single data model or a group of multiple data models. This can be done by configuring a “Reserved” amount of allocated CPU/RAM. Note that other data models would be limited to partial server resources even though a pre-allocated resource might be idle. Prioritizing Data Models - Governance rules can be used to prioritize certain data models by allocating a different amount of resources to different data models. High-priority data models would be allocated with more resources than lower-level data models. You may also choose not to limit the data model’s resources (no Data Group configuration). However, this will re-introduce the initial risk of resource depletion. How To Configure Data Groups? Configuring “Data Groups” requires high expertise and attention to detail. A dedicated tool is introduced to assist you in this task. Follow the directions below to acquire the tool and implement Data Groups: Prerequisite To base your calculations and make a decision on how to configure data groups you’ll require data monitoring enabled. To learn more about Sisense monitoring and the data reported read the following article: https://support.sisense.com/kb/en/article/enable-sending-sisense-monitoring-logs Step #1 – Download the “Data Groups Configuration Tool” The data groups tool (Excel Document) is attached to this article. Download a local copy to your computer. Step #2 – Access Logz.IO To access the Logz.IO platform: Log in to the Logz.io website Navigate to the “SA - Linux - Data Groups“ dashboard Set the timeframe filter to a 21-day timeframe Step #3 – Fill out the “Data Groups Configuration Tool” Document The “Elasticubes” sheet holds the data for the decision making regarding the different groups: Field Description Comment CubeName The ElastiCube name Size GB The ElastiCube’s size on disk The measure is taken from the “Data” tab ElastiCube list Estimated size in memory The estimated maximal ElastiCube size in memory Auto-calculated ([Size GB] X 2.5) Peak query memory consumption (GB) The actual maximal ElastiCube size in memory (should be smaller than the estimated maximal size) The measure is taken from the logz.io dashboard: (“Max Query Memory” field) Build frequency The frequency of the ETL Sisense performs The measure is taken from Sisense’s build scheduler Average of concurrent Query Average concurrent queries sent from dashboards to the ElastiCube The measure is taken from the logz.io dashboard: Search for the ‘Max concurrent queries per Cube’ widget and download the CSV file with the data. Calculate the average value of the “Max concurrent queries per Cube: column Business Criticality This is a business measure that determines the QoS of the ElastiCube True (High) / False (Low) Data security Is “Data Security” applied to this ElastiCube This column will help determine if the “ElastiCube Query Recycler” parameter will improve the performance or not. Explanation here under “ElastiCube Query Recycler” Data Group The Data Group this ElastiCube should be a part of Try to fill in the column by classifying your ElastiCubes according to common characteristics both in terms of business and in terms of resource consumption. Step #4 – Define your Data Groups Using the information you’ve collected and the explanations in this article – Define the data groups you wish to use. Fill in the information in the “Required data groups” sheet. The “Required data groups” sheet provides the name and configuration for each data group. Use this to tab describe the Data Groups that meet your business needs, use the “Intent” column to describe each group's purpose. The configuration in this sheet will later be used to configure the Data Groups in the Sisense Admin console: Field Description Comment Group name The Data Group’s name Intent The Data Group’s description (in the business point of view) Instances The number of query instances (pods) created in the Sisense cluster This parameter is very useful, however, increasing this value will result in increasing the Elasticube’s memory footprint. You should only consider changing this value if your CPU usage reaches 100%. The high CPU consumption is mostly caused by high users concurrency Build on Local Storage Is enabled, for Multi-node. Using local storage can decrease the time required to query the cube, after the first build on local storage Simultaneous Query Executions The maximum number of queries processed simultaneously In case your widget contains heavy formulas it is worth reducing the number to make the pressure lower. Used when experiencing out-of-memory (OOM) issues. Parallelization is a trade-off between memory usage & query performance. 8 is the optimal amount of queries % Concurrent Cores per Query Related to Mitosis and Parallelism technology. To minimize the risk of OOM, set this value to the equivalent of 8 cores. e.g. if 32-core, set to 25; for 64 cores set to 14; the value should be an even number. Can be increased up to a maximum of half the total cores - i.e. treat the default as the recommended max that can be adjusted down only. Change it when experiencing out-of-memory (OOM) issues Max Cores Maximum cores allowed to be used by the qry pod Limit for less critical groups Query: Max RAM (MB) The Max RAM will be consumed per each of the ElastiCubes in each group. Take from the column “MAX of Peak query memory consumption (GB)” in the Summary of data groups sheet Maximum RAM that is allowed to be used by the qry pod (dashboards). By each of the query instances if were increased in the Instances. Please note that 85% of the pod usage OR overall server RAM will cause a Safe-Mode exception. In the case of Dashboards qry pod) will delete and start the pod again. At the Safe Mode, the users of a dashboard would see the error of a Safe Mode and the qry pod will be deleted and started again. So the next dashboard refresh will bring the data Step #5 – Verify The “Summary of Data Groups” sheet includes a pivot chart that will calculate the max memory consumption from each data group. This value correlates to the “Query: Max RAM in MB” configuration. We need to take the maximal value of Peak query memory consumption (GB) from the Summary of data groups tab and multiply it by 1.20 to avoid Safe mode. Step #6 – Process the results and configure Sisense Data Groups Prerequisite - Read this article on how to create data-groups (5 min read): https://documentation.sisense.com/docs/creating-data-groups On the Sisense Web Application, for each new data group: Navigate to “Admin” --> Data Groups” and click the “+ Data Group” button. Fill out the information from the “Required data groups” tab. Step #7 – Monitor the Sisense Instance The final step is to follow the information in the “Sisense Monitor” and to make sure the performance is improving. Review the documentation below for more details regarding monitoring https://docs.sisense.com/main/SisenseLinux/monitoring-sisense-on-linux.htm Good luck!
Introducing Sisense Fusion 2021.10 — Enhanced Personalization, Streamlined Support, and More!
Dig into the latest features to enhance your Sisense analytics platform with custom fonts, new Web Access Tokens that allow sharing without logging in, and for managed services customers, a secure support connection to help troubleshoot without interruptions.
