I would NEVER say anything like that to someone!
I would never say to a sanitation worker, "The only problem I have with garbage collection is that you stink at the end of the day."
I would never say to a banker, "The only problem I have with banking is that you have to lock all that lovely money back in the vault at the end of the day, and can't take any of it home."
I would never say to a postmaster, "The only problem I have with sorting mail is that you have to remember all the zip codes."
I would never say to the cashier at Wal-Mart, "The only problem I have with checking out people is having to stand here and interact politely to people you don't know, who might even be rude to you."
WHY would people feel the need to say, as soon as they hear you mention that you homeschool, "The only problem I have with homeschooling is ________?"
Everyone has their own opinion of homeschooling. Folks who don't personally know homeschooling families even have their opinion. It's all based on what they suppose homeschoolers do with their school time. What? Do they think we don't actually HAVE school? Do they think we hole up here in our quiet country home, never go anywhere except church, never interact with non-homeschoolers, and become completely socially inept? I'd put my children up against your average fourth and sixth graders any day of the week. Are they perfect? Heavens no! Are they always obedient? Uh, again, no. Do they sometimes drive me crazy? If today was any indication, heck, yeah! But for the most part, they are polite, well-mannered, pleasant children.
I would never, ever presume to tell any parent that I have a problem with them sending their children to public or private school. Every family should do what works for their own family, and not act like it's a personal affront if someone does differently.
Okay, done with my rant for the day.