It seems like a lot but they aren't rocks like the regular rocks that are all around us. $100 for 20lbs of the rocks I see everywhere I look where I live, yeah, that would be totally outrageous.
But if someone has to find a location that has particular stones that look good in aquariums, go collect them, ship them to someone (expensive) and then that person has to pick them out and ship them to you and in the end it costs you $100, that's not so bad.
I bought Ohko stone from AFA (Aqua Forest Aquarium) and was happy to pay the price because it was cheaper than a plane ticket to Japan and a guide to help me hike around mountains and then having to ship everything I collected back to the states.