While I've drifted from a good amount of my friends from high school, I don't really have much experience in straight up growing to resent people I hang out with all the time, so take my opinion with a grain of salt.
I don't think you need to break up with him or anything, but if it really bothers you to be around him, you maybe should stop "trying to mend" your friendship (if you haven't already). As in, I wouldn't like, go out of my way to spend time with him. Just do your own thing with people you enjoy being around, and if he ever shows up then whatever, just focus on the positive things, and try to have a mutual respect at least. You're still very young, and one day you'll be in very different places, and this shit will barely be a blip on the radar we call life.
tl;dr: You're in college, enjoy it. Don't let negative feels drag you down.