Bad memories and/or bad faith, I believe.
Cuz yeah, Yang honestly has the best motivation to want to get with Jaune, because her motivations for being attracted to Jaune are simply that she's thinks he's attractive, not because of what he can do or what he gave her.
Which by the way is an actually shallow motivation to want to get with someone. Wanting to be with someone because they're rich is far shallower than because you genuinely find them attractive.
Even Ruby likes Jaune primarily because of what he can do with weapons, not who he is.
Furthermore, I'm pretty sure Yang has done more to actually get to know him than the other girls on Team RWBY. Where she learned that he has some experience with girls that only want him for his Semblance.
So my vote for who he interacts with is definitely Yang and Pyrrha, who is similarly less shallow in her motivations for finding him attractive than the rest of the cast.