Yes, definitely don't put them back into the wild.
If they pick up some disease from say a plant in your tank, or a piece of equipment that has been n touch with other fish that have come from who-knows-where-in-the-world, then you could introduce a new disease into your local ecosystem and the native fishes and animals might not have any resistance to it at all. It isn't enough to say "well my fish aren't sick", because there could be something that your fish have a resistance to but are still carrying, so you wouldn't even know that it was there in your tanks before letting it loose.
For the same reason, it is always important to make sure that any plant clippings you throw out don't get into local waterways or ponds, as they could give some disease or parasite to the local flora.