View Issue Details

IDProjectCategoryView StatusLast Update
0005827SymmetricDSImprovementpublic2023-05-10 12:10
Reporterelong Assigned Toelong  
Status resolvedResolutionfixed 
Product Version3.15.0 
Target Version3.15.0Fixed in Version3.15.0 
Summary0005827: Track expired data gaps and repair any data missed by routing
DescriptionImprove handling of data gaps by tracking expired data gaps and repairing any data that arrives late, no matter how late it arrives. Router service keeps track of data IDs that are missing, called data gaps, and continues to query them, waiting for data to arrive. Eventually they become so old that they are considered expired and they are removed, to keep routing running fast. The problem is that some systems have transactions open for hours and even days before committing, and this data is purged.

Keep track of expired data gaps with a flag is_expired on sym_data_gap. The router service will only use non-expired gaps. The purge service will watch for data to arrive in expired data gaps and be responsible for purging expired data gaps that are too old (based on a parameter setting). The purge can continue to use its fast range-based purge of sym_data, but it must step around any expired data gaps. (It would not be sufficient to check for expired data first and then blindly purge by ranges because they creates a race condition.) A new routine will query for data in the expired data gaps, and send the rows as reload events by their primary key values. For any deletes, it should query if the current row exists to determine if a reload or delete should be sent. The stranded data routine should be modified to also step around expired data gaps. Any data is does find should also be repaired, as it is likely data from an expired data gap that was purged. Track statistics for purging of expired data and stranded data.



There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2023-05-09 18:26 elong New Issue
2023-05-09 18:26 elong Status new => assigned
2023-05-09 18:26 elong Assigned To => elong
2023-05-09 18:26 elong Tag Attached: purge
2023-05-10 12:10 elong Status assigned => resolved
2023-05-10 12:10 elong Resolution open => fixed
2023-05-10 12:10 elong Fixed in Version => 3.15.0