Maybe once he thinks about it a little more, he will not hold anything against you, especially with it being the holiday.
I can share my experience, if it helps. I made the mistake of taking my grandchildren every weekend because I wanted to spend time with them. It started when my granddaughter was 3 weeks old. I took her so that my daughter could get some rest. Then, my grandson came along, and it was a little harder for me. It's not easy as you get older, and I'm not in the greatest of health. My daughter rarely asked me to babysit, but my son-in-law started getting the idea that he could just call me anytime, and I was supposed to drop everything. I was working full time then, and taking them on weekends was getting too much. I never had any time to rest, and I was getting exhausted.
When I started getting sicker, I just had a talk with him. I explained that although I LOVE spending time with my grandchildren, I just can't take them on weekends anymore, but that I would be willing to babysit in emergencies and I would take them on an occasional weekend when I was feeling good. He completely understood. We haven't had any problems at all, and I still take them once in awhile when I'm up to it. They just had a new baby in July, so taking care of 3 kids (ages 6, 4 and 4 months) isn't easy for my husband and me. We take the two older ones once in awhile, and it works out fine.
It's not that I don't love spending time with my grandkids. I'm absolutely crazy about them. But, I just can't do it all the time. Sometimes they have me come down for a weekend or for a day or so during the week, but I'm not babysitting. They do all the work, and I have all the fun.....just as it should be.
Don't feel guilty. I know that's easier said than done, but you can't. It took me awhile to be able not to feel guilty. Now I just enjoy them without the guilt.
