Dear colleagues,
Let me explain our application first. We build and sell an ERP system. ERP systems are known for their large databases, containing thousands of tables with tables easily exceeding a million rows. Some of our customers need parts of our functionality in environments where wireless connection with the database is not feasible (e.g. inspections of a newly build ship hull or other boxes of Faraday, near machines that generate EMPs, etc.). For those cases we provide small parts of our functionality on a mobile device that is kept ‘in sync’ using Change Tracking.
Because mobile devices have more restrictions on the amount of available memory and because synchronizing your mobile device should not last more than a couple of minutes, a function that uses a frequently mutated table that contains over 10,000,000 rows in the database should not result in an implementation where you have an exact copy of this table on the mobile device. We only copy the columns and rows needed for the functionality on the mobile device. Below you see a (simplified) example of the incremental update statement for Change Tracking.
SELECT dbo.T_ProductionHeader.ProdHeaderDossierCode
, dbo.T_ProductionHeader.ProdStatusCode
, dbo.T_ProductionHeader.ProdHeaderCompletedInd
, dbo.T_ProductionHeader.Description
, dbo.T_ProductionHeader.PartCode
, dbo.T_ProductionHeader.Quantity
, dbo.T_ProductionHeader.StartDate
, dbo.T_ProductionHeader.EndDate
FROM dbo.T_ProductionHeader
INNER JOIN CHANGETABLE(CHANGES dbo.T_ProductionHeader, @sync_last_received_anchor) AS CT
ON CT.ProdHeaderDossierCode = dbo.T_ProductionHeader.ProdHeaderDossierCode
WHERE (CT.SYS_CHANGE_CONTEXT IS NULL OR CT.SYS_CHANGE_CONTEXT <> @sync_client_id_binary)
AND CT.SYS_CHANGE_OPERATION = 'U'
AND CT.SYS_CHANGE_VERSION <= @sync_new_received_anchor
-- In filter
AND dbo.T_ProductionHeader.ProdHeaderCompletedInd = 0
AND dbo.T_ProductionHeader.ProdStatusCode BETWEEN N'20' AND N'65'The problem is that filtering the rows results in the situation that an update on the table should be translated in an insert or delete for Change Tracking. Take the filter above. Now assume a record is inserted with ProdStatusCode = '00'. Change Tracking signals an insert, but the filter statement causes this record not to be synchronized. If the ProdStatusCode is updated from '00' to '30', Change Tracking signals an update. This update is synchronized, but since a record is not present on the mobile device, nothing happens
What I did is creating my own ‘Change Tracking’ tables having the same structure as the Change Tracking tables that Microsoft uses. I also added simple triggers on the filtered tables. Whenever an update causes a record to move from outside the filter to inside the filter, I record an insert for that record in my custom ‘Change Tracking’ table. When the record moves from inside the filter to outside the filter, I record a delete. When the record is deleted, I remove it from my custom CT table. Furthermore I wrote a scheduled job that retrieves the retention period from sys.change_tracking_databases and removes all recorded information from my custom CT tables that is older than this threshold. Next, I added a UNION clause using my custom CT tables to the standard incremental insert and incremental delete statements for the mobile device, adding the records that should be inserted or deleted because of an update.
My question is, is there a more standard way of solving this problem or do I have to do all this stuff myself. I cannot imagine that we are the first to encounter this problem, so maybe more elegant and better solutions are already available?