What are the best practices for the number of srcset items for a dynamically sized image?

I have the ability using server side rendering to render as many different source images as necessary, given that the library (in this case, sharp) can render any size requested. When is the overhead of having too many possible permutations of sizes a performance hinderance? It seems like no matter what if you are selecting only a handful of sizes, your images are not being optimally sized for every possible screen size. I've Google searched for any kind of best practice information on this topic and have come up dry in finding any official recommendation from Google, Apple, Mozilla, or other browser makers, or any of the HTML W3C Draft docs and discussions. It's only more complicated given the various pixel densities across all devices... I have <meta http-equiv="accept-ch" content="DPR, Width, Viewport-Width"> also in place, just in case that client hints might assist in some way to select the best possible image for the user/screen size - but that also seems a bit of a black box of uncertainty.