Intermittent High Ping?

Hello! I've been struggling with this for a while. I have no clue how to go about diagnosing the issue either. Basically, it's intermittent and never consistent- a player, or sometimes multiple players will experience insane levels of ping that always result in timing out. The strange thing is that it never happens to more than one player at a time. The server's had at least 3 people on it that have not had it while 1-2 others will. Any ideas or suggestions as to where to start looking first?