Should I keep the XS or the S in this EG jacket?

Hey, I've got on both the XS and the S on this EG loiter jacket and I'm waffling between the two. I like the oversized-ness of the S but I'm afraid its a little too big and I'd be better off keeping the XS so its a little more versatile and I still get the drop shoulder. The XS fits more like a traditional sport coat type thing while the S fits more like an actual coat, going past my butt. The first 3 pics are of the XS, the last 3 are of the S.