Love thy worms.
I do agree though if your rocks come alive with them at feeding time, you're feeding too much.
They are doing you a service though, if there's that much food that they are flourishing on the leftovers, if they weren't there to consume it you'd have a lot of rotting food fouling the water.
They do poop too though, so if you're feeding that much, you want to keep a close eye on your water quality, especially for phosphate and nitrate.
The only real down side is if you get your fingers bristled, otherwise they are good to have in the tank. If you do get bristled, soak your fingers in vinegar to dissolve the bristles.
Jenn