Cats, dogs, foxes and birds, not to mention squirrels have all been known to attack plants. Chewing the leaves, digging out the soil, or knocking the pots over is common. And I think we can all agree that no plant deserves that.
To my mind, the biggest reason to avoid putting plants outside in the summer is what can get into their pots when you do. Every time I have tried to give my plants a little outdoors vacation, I have found all kinds of creepy crawlies emerging when I bring them back indoors, from ants to flies that hatch a few weeks later. It is difficult to prevent bugs from getting into the plant pots, and it can be almost impossible to get rid of them when they do.