Some people leave it on when they aren't putting in large amounts because they figure that CO2 doesn't shut off in the wild, so why in a tank? There really isn't any benefits to it. Just less work. They have to also watch their pH though, because unused CO2 can cause changes in the pH which wouldn't be good.
I would recommend turning it off because, even though it is a little bit more effort, you avoid risks and save more CO2 in the end, which could save you some money.
I would recommend turning it off because, even though it is a little bit more effort, you avoid risks and save more CO2 in the end, which could save you some money.