I'm the same way, but I only move it when I'm in a large city or one a train/metro/subway/bus where I have to stand rather than sit.back right pocket.
on the rare occurrences I'm in a crowded area I'll move it to my coats zip up pockets.
i dont like it anywhere else, because i habitually check to make sure its there. if its not, there is usually a brief moment of "oh, pants"
I hate those moments when you move it, and then go for it and are like "oh shit, where did my wallet go?"