It's all about what suits you. My daughters 4 now and I've worked FT since she was 2 1/2. I worked PT since she was 5 months. In my situation my partner works back shift and in the over-lap between our shifts she's either at nursery school or her Nana's. Like me your child has loving family members, it's not like your abandoning her to a childminder or nursery. I still struggle with guilt, but she starts primary school in August and she's growing up, I work Mon to Fri and am off weekends.
It's hard to get a balance, I come in from work and she's now at an age where she likes to help make the dinner, so I let her help. It's important to me that we all sit round the dinner table and talk about our days. After dinner the dishes are left til she's in bed and between dinner and bedtime is our 1-2-1 time. Once she's in bed thats when the houseworks done, i try to keep on top of it as much as I can so it's just a scoot around. That normally takes an hour or so and the rest of the times me time, whether it be a bath, a good book or a glass of wine with the girls!
It's certainly never done my child harm, she's a happy, outgoing, confident little girl. She knows I go away to work, but she knows I always come back. Some of my friends with kids the same age have terrible problems now that they've just started back to work, there first wee while at nursery a lot of them were terribly clingy and tantruming.
As I said, the choice is yours, personally I don't think it's a bad thing and my partners supportive, he does his bit in the house. But remember the dishes can wait, spend the time with your son.
You're not bad for doing this, and your feelings are completely natural xoxox