Internet disconnects when someone joins Xbox Live Party?

My internet is fine (TalkTalk) and I'm playing games on xbox 360 live - but when I'm in a party and mid-way through a game if someone else joins it will just disconnect me. Also whenever I try and join the party when I first log on it logs me out again straight away - every time..

Is there something I need to do to my internet settings or Talk Talk settings?

Thanks!