Plants need oxygen produced by photosynthesis to keep their root zones aerated and detoxified. Fermentation is horribly inefficient, so the plants lose energy fast if they can't photosynthesize.
The problem you had with the 10 gal is a little less clear based on what you've written. Possibly the low lighting, pebble substrate, and no water circulation. I'm also wondering about your water's water hardness.
Glad to hear that your new tank is doing better. With that much light (15 watts CFL over a 5 gal tank) and a soil substrate, those Amazon Swordplants should have filled up the tank within a few months. Again, I'm wondering if your water is a little soft (lacks calcium, magnesium, etc). Amazon Swordplants really thrive in hardwater.