There is nothing wrong with how you feel. Looking back, I don't know that I would have had children. We live in really bad times and everyday, I'm scared at what my kids will have learned in school.
If people butt into your business about kids, just tell them...
"After much deliberation and seeing how YOUR kids turned out, I decided that I didn't want an early death! I enjoy life too much!"
