Due to the intellectual property (IP) behavior, you might that the observe rx_latency_adj and tx_latency_adj value are changing for a certain amount of iterations before they settle to a fixed value.
No workaround is required. This is an expected behavior. The valid latency value is the fixed value after certain amount if iterations. The latency value changes after reset as it is statistically calculated, hence require certain amount of iterations before it settles to a fixed value.