Can Explanations be Relevant?
We are interested in rolling out Explanations to our clients, but I've hit a block and need some guidance. I think that the source of the issue may be our use of null replacement values in our data models. We don't allow data to join out of a query and instead ensure that all fact data joins to a null-replacement row in our dimensional tables. For example, given a fact table of product PURCHASES, imagine it was possible to sell those products at a discount in some orders. Assume that the fact table then joins to a dimension table of DISCOUNTS. Most purchases do not have a discount, but some do. Without a null-replacement row in the DISCOUNTS table, adding a field from that dimensional table onto a widget will limit the results in the widget to only fact rows with a discount. In our models, we would replace the null discount key in the fact table with a value that joins to a null-replacement row in the DISCOUNTS table. For example, replace null with -999999 in the fact table and have a row in DISCOUNTS with [id] = -999999 and a [description] = "(none)". The problem we're running into with Explanations is that the feature is presenting only the most useless information as being the most probable explanations. For example, when a single discount exists in the data set, then 99.9% of the PURCHASE rows have a "(none)" DISCOUNT, which is deemed this highest scoring field. More generally it seems that Explanations just reports back the most highly probably explanations as those fields where the member in the field is the commonly present in the data anyway, and it's just reflecting the raw change in fact data. For example, here's an image where the drop in revenue is explained by the drop in revenue from web sales (BRestAPI). But web sales constitute a larger over proportion of sales after the drop (89% up from 82% in the prior period). And another example in which an "Initiator Specified Flag" is whether or not an alternate order "owner" has been specified on a order. Specifying an alternat owner is allowed ("Y"), but not a common practice ("N"). Again, we're just seeing a reflection of sales in the explanation. Is there anything we can do to make the explanations discovered by Sisense relevant? The way it's functioning now, it would be better if Sisense DID NOT recommend possible explanations and just let the user Explore Other Fields immediately without having to wait for Sisense to return the list of completely irrelevant fields. I've tested this across many of our clients' data sets and found the same behavior. Hopefully there are options for us to improve the recommendations or maybe prevent it from recommending.
Data prep essentials for AI-driven analytics - part 3
Discover how to improve data quality for AI and machine learning. In Part 3 of our Data Preparation series, learn how to fix missing values, remove duplicates, correct data types, and standardize formats—with SQL and Python scripts to help you clean your data for accurate, AI-ready analytics.
Have you heard about Sisense Intelligence?
Sisense Intelligence is a new suite of AI-powered tools that help product managers and developers build smarter, more intuitive analytics experiences. Features like natural language dashboards, automated insights, forecasting, and data explanations accelerate every stage of the analytics journey. Designed for seamless integration, it empowers teams to deliver fast, user-friendly insights at scale.
Use dashboard API to get data in python dataframe
I have a widget which displays users with points and I want to fetch top 10 users from that widget.Also I need to select current month "01/2025" from the filter.Let me know how this can be done But I'm getting weired error.Please help! I'm an admin so I have a token from my profile section. import requests url = "https://bp.sisense.com/api/v1/dashboards/4577fc4415dd7d003339bfdc/widgets/4577c7716dd7d554441bjae" # authentication headers = { "Content-Type": "application/json", "Authorization": "" } # Send the request response = requests.get(url, headers=headers) response # Check if the request was successful data = response.json() data Error: {'error': {'code': 5002, 'message': 'Invalid token.', 'status': 401, 'httpMessage': 'unauthorized'}}Exploring RAG: A Sisense-Based Approach with BigQuery and Gemini
Exploring RAG: A Sisense-Based Approach with BigQuery and Gemini Continuing from our previous article, Automated Machine Learning with Sisense Fusion: A Practical Guide, where we discussed how Sisense and AutoML simplify complex machine learning workflows, I am now excited to introduce an advanced feature: a chatbot interface powered by Retrieval-Augmented Generation (RAG) and a large language model (LLM) like Gemini. This enhancement allows users to interact with their data in natural language through Sisense's Native Blox widget. Imagine asking questions like "How many products did I sell per category?" and receiving insightful answers instantly. This not only improves data accessibility but also bridges the gap between technical data repositories and business decision-making. In this article, I’ll cover the first step of enabling this functionality: setting up a seamless integration pipeline between Sisense and Google BigQuery, embedding table and column metadata, and preparing the data for efficient querying. Important Note: Experimental Project: This implementation is an experimental project and not an official Sisense feature. It demonstrates how Sisense’s functionalities can be utilized to build a custom Retrieval-Augmented Generation (RAG) pipeline or how you can use this example by importing the provided notebooks. Google Ecosystem: The example provided is specific to the Google ecosystem, utilizing services like BigQuery (BQ) and Gemini for query generation and processing. Associated Costs: Please note that each prompt sent to the chatbot incurs a cost, which will be billed to your Google Cloud Platform (GCP) account. This includes charges for LLM processing and database queries. Building the Foundation: Data Preparation and Embeddings To enable natural language interactions with your data, we first need to establish a robust data infrastructure. This includes creating a vector store for embeddings in Google BigQuery (BQ) and preparing metadata about tables and columns. The process is fully automated using a pre-built Sisense custom code notebook, which simplifies embedding generation and management. Dataset in BigQuery The journey begins with your dataset in Google BigQuery. The notebook retrieves metadata such as table and column names, descriptions, and schema information from BigQuery’s Information_Schema. If any descriptions are missing, the system automatically generates them, ensuring comprehensive metadata coverage. Setting Up the Vector Store in BigQuery The vector store acts as the backbone for similarity searches within the RAG system. Using the custom notebook, the system: Creates the vector store: A structured repository to store embeddings for tables and columns. Organizes metadata: Ensures all table and column descriptions are structured and accessible. Generating Table and Column Descriptions Missing descriptions can hinder data understanding. The notebook includes a Description Agent that automatically generates meaningful text for: Tables: Contextual descriptions based on schema and usage. Columns: Descriptions highlighting their role within the dataset. These enhancements ensure the metadata is both informative and ready for embedding generation. Creating Embeddings with Vertex AI To enable semantic search, metadata descriptions are converted into numerical embeddings using Vertex AI’s TextEmbeddingModel. This is facilitated by the EmbedderAgent, which: Accepts strings or lists of strings (e.g., column descriptions). Generates embeddings through Vertex AI. Handles both single and batch processing for efficiency. Efficient Embedding with Chunked Processing For large datasets, embeddings are generated in chunks using the get_embedding_chunked function. This ensures: Scalability: Handles datasets of all sizes without performance issues. Parallel Processing: Processes text chunks simultaneously to speed up the workflow. Structured Outputs: Embeddings are returned in a structured DataFrame for storage or analysis. Storing Embeddings in the Vector Store The final step is storing these embeddings in the BigQuery vector store. This ensures that: Similarity searches are fast and efficient. Metadata is always accessible for chatbot interactions. ALT text: A screenshot of a data table displaying various columns such as "table_schema," "column_name," "data_type," "source_type," and several other attributes. The table shows sample values and metadata related to a database structure, organized in rows and columns. How the Chatbot Interface Works Now that the foundation for embeddings and metadata storage is set, let’s explore the chatbot interface in action. Imagine opening the chatbot in the Blox widget and asking a question about your dataset. Within moments, the chatbot responds in natural language, providing actionable insights. But what exactly happens under the hood to generate this seamless interaction? RAG Notebook and the Chatbot Workflow The chatbot operates using a pre-built RAG custom code transformation notebook, which orchestrates the end-to-end process. With this notebook, the entire pipeline—from understanding the query to generating the response—is automated. The notebook uses multiple specialized agents, each responsible for a specific task, ensuring precision and efficiency at every step. SQL Query Builder Agent BuildSQLAgent This agent specializes in constructing SQL queries for BigQuery. It uses the LLM to analyze the user’s natural language query and matches it with table schemas and column details from the vector store. It outputs a fully formed SQL query tailored to the user’s dataset and question. SQL Validation Agent ValidateSQLAgent The ValidateSQLAgent validates the SQL query before execution using a Large Language Model (LLM). Validation ensures the query adheres to essential rules, including: The presence of all referenced columns and tables. Proper table relationships and join conditions based on the schema. Formatting and compliance with BigQuery-specific SQL standards. Validation occurs during the debugging process, specifically within the DebugSQLAgent, to identify potential errors before attempting a dry run or execution. It provides a detailed JSON response: If valid, the process moves to the next step (dry run or execution). If invalid, the DebugSQLAgent uses the error details to refine the query iteratively. SQL Debugging Loop Agent DebugSQLAgent This agent runs the debugging loop to refine queries that fail validation or execution. The process includes: Validation: The query is passed to ValidateSQLAgent to check syntax, schema compliance, and structure. If valid, the query is ready for execution. Dry Run: If validation passes, the query is tested using a dry run via the test_sql_plan_execution function to confirm execution readiness. Execution: Once validation and dry runs succeed, the final query is executed using the retrieve_df function, which returns results as a DataFrame. Iterative Refinement: If the query fails either validation or the dry run, the DebugSQLAgent uses the LLM to troubleshoot and generate an alternative query. The loop repeats until a valid query is generated or the maximum debugging rounds are reached. This agent ensures the final query is: Correctly structured and semantically valid. Optimized for performance and aligns with the original user intent. Response Agent ResponseAgent This agent translates the SQL query results into natural language. It bridges the gap between technical SQL outputs and user-friendly communication. By combining the query results with the user’s original question, it crafts a clear and relevant response. How the Workflow Executes Here’s the step-by-step process for generating a response: User Query Embedding The EmbedderAgent converts the user’s natural language question into a numerical embedding. Using BigQuery native vector search, the system retrieves similar embeddings from the vector store created in the first phase. Schema and Content Retrieval Based on the retrieved embeddings, the system fetches relevant table and column schema details from the vector store. SQL Query Generation The BuildSQLAgent uses the retrieved schema details to construct an SQL query that aligns with the user’s question. SQL Validation and Execution The ValidateSQLAgent checks the generated SQL for accuracy and potential errors. If the SQL passes validation, it is executed against the BigQuery database. Debugging (if needed) If the query fails or generates an error, the DebugSQLAgent refines it iteratively until a valid query is produced. Response Generation The ResponseAgent uses the query results and the user’s original prompt to generate a natural language response. If the system fails to generate a valid response, it communicates the issue to the user. Conclusion By combining the foundational embedding process with this RAG-powered workflow, the chatbot transforms how users interact with their data. From seamless SQL query generation to delivering natural language responses, the system exemplifies the power of Sisense Fusion and advanced AI tools to simplify data-driven decision-making. As always, please reach out to your Customer Success Manager (CSM) if you would like to implement this in your own environment.
