3mm is fine. If you start getting bigger than that it starts getting too big for the roots of the finer plants to grab on. I had that for years. It will do fine with the large root plants but no fine root plants.
The fine gravel will eventually drop to the bottom and you will be left with the larger gravel on the top. If you have small rooted plants they will have trouble staying planted.
I'm not saying you can't use them. I'm just telling you from my experience that you will not be able to use the finer rooted or stemmed plants because the larger rocks will injure their roots and/or you won't be able to keep them planted because there is too much space between the larger rocks.
I was told that you can get #3 blasting sand from places that sand blast for about $10 for 50 lbs. It's really cheap. You might want to think about that.
I'd suggest getting hold of a copy of Diana Walstads book (Ecology of the Planted Aquarium) and reading it a few times... Its a great read and full of a lot of facts and pointers...
On the subject of gravel size, its reccomended to go for 2mm-4mm size on top of the soil layer. Larger gravel will not be good for rooting as already said, but may also trap waste due to the large spaces between gravel where it may rot and foul the water...
I agree completely with TexGal & Kosh42, larger gravel is a big pain to work with and when you get tired of fighting with it changing out the substrate is a PITA. Large gravel is hard to plant in as stems and roots are easily bruised or broken, fragile plants or small plants are virtually impossible to plant and even if you get them planted they don't grow as well. There are some relatively inexpensive options for a capping substrate, the sand that TexGal mentioned and Soil Master Select or one of the similar products are both good to plant in. A more expensive option is Eco Complete - I love the grain size and color, it also has a good weight so it's easy to plant in. You can use the larger gravel and grow things in it, but if you think you'll want to grow a lot of different plants (foregrounds, various stem plants) do yourself a favor and get a capping substrate with a 2-4mm particle size, you'll thank yourself for it.