Define "a lot". If you have a few worms here and there, that's a good thing, let them be.
If the rocks crawl with them when food is introduced to the tank then you have too many. That usually comes from overfeeding as their population responds to the amount of available food. That also means that even if all the food is eaten, there's enough waste (poop from the fish) for them to eat and have a population explosion.
The drawback to arrow crabs is that they can and will go after small fishes. If you don't plan to keep small fishes, it's no biggie. Same with coral banded shrimp - they can be really aggressive too. So know the risks if you go that route.
Many wrasses will eat bristleworms. They are a good natural control - but the down side is that some of those same wrasses will also dine on your clean up crew. Trade offs for everything.
If you feel the need to purge some worms, take the rock out and lay it out in the air for a little while, like an hour or so. The worms will come out and drop off, some will find pools of water that drip out of the rock onto whatever surface you have the rock on. You can also pick them off with tweezers as they emerge. Some will stay hunkered down in water in the cracks and holes in the rock, but you'll cull the population some.
Going forward, watch how much you feed, so you don't have another population explosion.
Jenn