I guess that depends on a number of things Shane. In nature, you could frequiently see a smooth stone right next to a rough, jagged stone however I believe that the reason you will notice it is because it stands out a bit. You eye picks up on a subtle change and then focuses on it. I think mixing smooth and rough "generally" would not work but you might be suprised. Give it a shot and see how it looks

Personally I believe that rough looks a bit better but that is meerly my opinion.
One thing to keep in mind is that no matter how natural an aquarum looks, it is not a direct scene from nature. Even biotope tanks are much nicer than a "slice" from the river. Also, everyone's goal in creating an aquascape is different. I prefer my tanks to look natural and serene. Lush and calmning. Amano's tanks tend to look a bit "fantasy" like to me. Same as James Hoftiezer's tanks. Remember his "Sleepy Hollow". Although, both of those are very calming too. I think these tanks are very dynamic and they certainly very impressive and I can only dream of creating something as nice as those. Now, lets continue to use Amano as an example and I will say the the reason is tanks all looks so "natural" flowing and harmonious is because EVERYTHING is in its place. THe rocks, wood, plants even the fish are in there exact place in the tank. His tanks all in many ways, are perfect. Atleast in respect to where everything is. The golden rule and all the proper proportions are utilized in his tanks and many forms of the visual arts that we westerners usually don't understand all lend themselves and influence his tanks. In an aquascape, as wiht any art form, a single thing slightly out of place can draw the eye and create an imbalance in the mind. A rock or even a stem pointing awry can be "confussing".
I guess what I am trying to say is you must try and try to find what works best. The only way to see if something looks ok is to actually see it first hand. Put them in the tank or at least play around with them on a towel or piece of cardboard with the same footprint as your tank. Can you post some pics? It would be nice to see your tanks

Be patient, step back and look at your layout then make small adjustments and try again until YOU like it. That is the important part to me. I don't care really what others think or how well it could do in a contest, I have to live wiht it and see it everyday. I enjoy it. It makes me happy
Hope that was atleast a bit useful. I know i have a tendency to rample occasoinally
