The number one thing: DO NOT take the clam out of the water to do this. Air + clam == baaaaaad.
Second, you may find that after yanking, the xenia comes back. They're tenatious like that and can grow from stubs like leather corals. If that's the case, there's no harm in a brisk brushing of the clam's shell with a stiff brush to scour the spot. Clam shells are very hard calcium and it doesn't hurt the clam as long as you leave it under water and don't scrape the clam's mantle or byssal gland (where the 2 shell halves meet at the bottom).