Bristol worms are doing their jobs by eating the dying or the dead. They are not predators that kill fish they simply clean up the aftermath. The reason you have so many and that they are so big is because you are over feeding. This allows the population to multiply at an alarming rate. I only feed my tank every 2-3 days and not a lot at that my worm population is at a stable level. If your tanks has as many pods
as you stated your fish will not starve without daily feedings.
Now am I correct in sayin that you only got these fish about two weeks ago? I do remember another thread that was started. Are you testing your water parameters? If not I would start. If I am correct you added the mandarin the goby and the Blenny all at once which is quite a heavy bioload for a 20gallon tank all at the same time. That is also a lot if bottom dwellers for that size tank. In mine I have 1-YWG and pistol combo and I wouldn't add anymore.
When adding fish to a tank on the smaller side I would suggest adding them one at a time. This will give your beneficial bacteria time to build up inbetween additions which creates less fluctuations. I would hold off on getting another mandarin fish as well. They are notoriously hard to keep and as you found out they die more often then not I won't even get one and my tank is almost a year old.
Feed less to bring down your worm population and hold off on getting another fish until your tank is a little more stable and then I would add one fish and see how that goes. Don't get the arrow crab they get fairly big and won't last in you tank. Not a fan of adding livestock to cure tanks problems it is only a cover up to the real issues that need to be dealt with first.
Man that was alot but I hope it helps any questions just ask.