It's a preference thing. It's harmless, a good filter feeder. Angels, if you have any, tend to nibble on them as part of their normal diet. I'd leave it be.
Occasionally some sponges can overgrow a bit - usually the ugly black stringy ones, I used to tear them out (and feed to angels in another tank) if they became unsightly.
If it bothers you that much, remove it, but if not, I'd let it be.