Stop lagging out of Xbox Live games?

This problem has been around for about two weeks now. 1 out of every 4 games i play, I lag out of. I thought it was just my internet connection but the weird thing is that I don't lag AT ALL during the game. I have a straight green bar connection and don't notice any lag, but randomly I'll lag out of the game and the xbox live party that I'm in at the same time. My NAT is open and my internet provider is comcast. I also have vonage which runs through my internet. This is really irritating. What could be the problem?