Retrieving total count and paginating large datasets with Compose SDK

[herajapakse]

I'm currently working on implementing a custom table to handle a large dataset in Sisense, and I'm encountering some performance challenges related to pagination. Here's the scenario I'm facing:

Requirements:

1. Retrieve the total number of records that match a filter condition (to get the total count for display and pagination).
2. Efficiently paginate the results by fetching only the records for the current page, based on page size (e.g., 10, 25, 50 records per page), without retrieving all records at once.

The Challenge:

While I can use offset and count to paginate the records and fetch only the records for the current page, I don't have a way to determine the total number of records that match the filter condition without retrieving all of the records first.

For large datasets, fetching the entire dataset before pagination is causing performance issues, since Sisense retrieves all the records matching the filter condition before applying pagination, which is inefficient.

What I'm Looking For:

• Is there a way to retrieve the total count of records matching the filter condition without fetching the full dataset?
• How can I fetch only the records for the current page (using offset and count), while still being able to display the total number of matching records?

Does anyone has experience implementing efficient pagination in Sisense for large datasets or can share any best practices, examples, or solutions?

[DRay]

Hello @herajapakse,

Thank you for reaching out. I see your question hasn't gotten a response yet, so I'm asking internally to try and get you an answer.