I found a really great post over on the Bride's Guide blog entitled What not to say about your destination wedding. I think a lot of this is just common sense, but I guess some people need to be reminded about that every once in a while.
Personally, we're just telling people, No matter where we have the wedding, someone would be inconvenienced. His family is all in the Northwest. My family is all in Texas. Not all of our family members and friends are in a place to make the trip to the other side of the country--financially, due to health reasons, etc. So we're eloping so that no one feels slighted (or everyone feels slighted equally?), and we'll party with everyone (on either side of the country) when we get back. While I know that still doesn't satisfy everyone, I feel like we've done our best to include as many people as possible in celebrating the beginning of our marriage. And it's important to me that as few people as possible be put out or inconvenienced in doing that, because celebrating our marriage is our choice, not theirs.
I don't really know how people would justify it if they were expecting friends and family to join them in their Big Day Away. I certainly hope expense, size and "everyone can come!" aren't the reasons though, because they just aren't very honest reasons. If you are doing it because that's the way you've always imagined getting married and it means a lot to you to do it in that way, just tell people that. I'm assuming if you've made the decision to have a destination wedding, you've already acknowledged the fact that some people will be necessarily excluded, and you shouldn't try to make it seem like you're doing it so everyone can join in when you know that's not the case.