I find this more on rap albums than anything, but rock albums aren't exempt from this either...
To me it disrupts the flow of the album. I like to just listen to music. Not hear the artist try and be funny. Cause' here's the thing... You will listen to an album a 1000 times over sometimes if it's good, and if it's got stupid talking/skits in it you'll have heard them a 1000 times, care less about them, and be decreasingly amused with them each time you listen to them.
I love The Smashing Pumpkins... but if Billy Corgan told a joke between each song on an album... no matter how funny it is, it's going to get old eventually, and all I'm going to do is end up skipping the track.
I love Tenacious D. But I always skip over the talking tracks to get to the songs. They were funny the first time I heard them. But 9 years later I've heard it... it doesn't have the same replay value that the actual songs do.
I don't know... I just could do without them.