Applies to: HSS/HVDS with ADE 5.1
Scope:
Post-release a problem was discovered that in particular circumstances an abnormal shutdown of ADE would lead to under-referencing blocks that in turn could lead to data loss.
Detailed description:
During the removal of streams (HSS/HVDS2) the garbage collector will be decrementing the reference counters of the blocks that compose the stream being removed. A small window of time exists where these updated counters get committed to the database but the mechanism to survive outages has not updated the commit. As a result, if an outage occurs during this small window the decrements will be applied again after restart hence effectively under-referencing the blocks.
In most cases this will appear in the logs at some time as “DataBaseROCKS::dec_records| Block reference decrement underflow detected|”.
To Fix/Prevent:
The issue occurs only on HSS or HVDS systems when an abnormal shutdown occurred with the garbage collector active.
If you have suffered from such a situation you should run a repair COUNTER operation to verify and repair the counters that could be affected.
Hence stop the ADE service and run ade_server –repair COUNTER
On HSS/HVDS only systems this will take a reasonable time to scan the active streams however on systems with Lina and HSS the operation might be a lot longer. Systems running Lina only are not affected by the issue.
ADE 5.1.11 will have a correction that will avoid this issue to occur so please upgrade as soon as possible.