Sorry, I forgot to completely write my issue in my first post. :X
I'm running inspircd. The odd thing about this issue is that it has now stopped. I got 6 people in my irc net one night and it dropped once every 5 minutes 3 times. After that, it's been fine. If I remember correctly, it's been up for 6 days now.