Due to the IP behavior, you may observe rx_latency_adj and tx_latency_adj value changing for a certain amount of iterations before they settle to fixed value.
No workaround is required. This is an expected behavior. The valid latency value is the fixed value after certain amount if iterations. The latency value changes after reset as it is statistically calculated, hence require certain amount of iterations before it settles to a fixed value.