Thank you all for chiming in!
Your right. Every time I rubbed it out, it would overpack and draw too tight, and the pipe would get hot quickly! Leaving it be has a looser draw than I'm used to, but it smokes beautifully! Now this was before this morning when I re-learned how to sloooooooowly...