This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Cookie Policy. Click Preferences to customize your cookie settings.
Hi all,
CONTEXT
------------------
We use GenericJDBC connector with custom queries to fetch data from our mongodb data source on Sisense. Our data model is similar to a star schema with one DIM table and several aggregated fact tables.
Till date we have been using replace all to sync with data source with every build. However, we are now trying to shift to an incremental build based model. We have an 'updatedTimestamp' field in our records for change data capturing.
------------------
SETUP
------------------
As I mentioned, our fact tables are aggregated and grouped by our primary key. For incremental build, since we need 'updatedTimestamp' column, I am adding `MAX(updatedTimestamp) as updatedTimestamp` part in the SELECT clause of the query. [Refer to Screenshot1 attached below]
Post this, I enable Incremental build behavior, selecting 'updatedTimestamp' as the column on the basis of which new data is added. [Refer to Screenshot 2 attached below]
------------------
ISSUE
------------------
Assume there was no change in the data source. Data source had 20 rows and so did the elasticube before I converted it's build behavior into incremental.
Now when I run accumulate build on this table,
Expected Output : No changes. 20 rows as before. (as there is no change in the data source)
Received Output : 40 rows. Each record is getting duplicated.
------------------
What am I missing? Why are the records getting duplicated?
Please let me know if I can provide any additional context or information.
Thank you for your time and help!
