You could always say you're planning a trip for just you and your spouse for a special anniversary. Now, this could be the anniversary of the first date, the first time you ate pizza together, the 35th time you saw a movie together...

But they enjoy the company of the other couple. That's the issue. But yes if you cut out both of them, that would be clearer and possibly less hurt feelings, but not really what you were aiming for (being able to travel with the two you like)l.