inconsistent part dimensions

I have this part that I'm trying to print in Orca. The central shaft is supposed to fit bearings, but it's also supposed to be consistent radius from top to bottom, and it isn't. The layers that are on the same layer as other extrusions (lower portion) end up .2mm in diameter larger than the upper portion. I can probably adjust for this in CAD, but it's a real pain that it's inconsistent in this way. Is there an alternative means for dealing with this problem? It was even worse when I had layer time targets set in orca, but even after removing that, it's still a problem. Any tips appreciated.