Any wood has the possibility of growing fungi.
As a very rough concept:
Fungi live by growing on dead matter such as fallen leaves, dead trees and bushes (still standing or fallen). It is their job in the circle of life to begin breaking down dead plants and start the recirculation of the nutrients in the leaves, branches, flowers, fruit and so on.
When we place (dead) wood in our aquariums one of the most common organisms that grows on it is a cloudy white gooey stuff. Occasionally referred to as 'wood snot'.
Some fish will eat it, especially Loricariads, and shrimp.
If you add such wood to your tank before the tank is cycled, then take a few weeks doing the fishless cycle, the fungi may grow too massive. If this happens, I would remove the wood and hose it off, brush it off with a soft brush, or other method of removal. Even leaving it in the tank and brushing it off with a siphon running nearby. Do not waste a lot of time trying to get 100%, though. It will come back. You are just trying to de-bulk it so it does not fill the tank.
In the end, it is not harmful to the fish (as I said, they can eat it) but it can be a bit too much.
I have heard several reports of grape wood being especially bad for this, but IME grape wood seems not to grow this material in my tanks.
Good to know about Rhody and Kalmia. I would sure be suspicious of fresh wood, though. Make sure it is well seasoned to break down the toxins.