Closed modal keeps fetching

Alright this is a problem that I feel like I solved before, I have solved now, but my solution bothers me. Currently, I have a page that has 4 modals, out of those, 2 of them are supposed to fetch data right after they open, but here's the problem: they fetch all the time as long as I'm in that page. The obvious solution is to wrap the modal into a conditional render in the page that holds it, not allowing the modal to exist before isModalVisible == true. But this feels wrong, I don't remember having to wrap all my modals before, what am I doing wrong or what is the best solution for this nowadays?