He will only poop if he's scared or really has to go. If he's pooping regularly on just one person, I would say it's down to how that person handles him.
Maybe your mom is a little nervous and he senses that so he doesn't feel secure when she handles him? Perhaps she is a bit erratic in her movements and just needs to slow it down a bit until he gets used to her?
I'm not saying that your mom is doing anything wrong on purpose, but it does sound like you need to look at what your mom does differently to you - then you will have your answer to why he poops on her more than you.