Overflow-x auto causes button to flicker

Hello, I got this div with a button, that when you click it you can add an image and the button moves to the right side. And there's a button on the image to remove it. And when there's enough pictures the div will scroll. All that works as intended. But when you remove the last image the button becomes basically hard to interact with. there's like a few pixels of good space where you can get it to work. Removing overflow-x auto solves this. But obviously it also removes the behaviour I want.