I don't remember having an issue with this when I pushed carts, but it was a long time ago and I suspect if I saw someone do this I would have pouted about it.
I did carts for a little while, I wouldn't have cared at all. If one person leaves a cart out, everyone feels like they can. It's crazy how it just instantly becomes acceptable when everyone sees one cart out. But will walk then back when there isn't