You don't need to be cleaning yoru gravel that much, especially with that few of fish. In most platned tanks gravel cleaning is not done much at all. Maybe in your case it is ok because you mostly have epiphytes??
I hope you get to the bottom of this.