something like this would to my knowledge require either extreme overhauls of how minecraft manages items or alternatively adding a 1 tick delay to item swaps in relation to attacking
oh or the other option is add checks anywhere that attributes apply to ensure it actually matches with the held items (but not too many places or it might lag stuff out)
an ideal ideal solution would be like 20x harder so oh well, it still overall performs fine and the perf isnt actually noticably worse anyways its a tiny difference really
unless im not feeling bored enough to continue maintaining it in which case if someone else is desperate to use this low quality fix they can maintain it themselves