It's not an exact science. "Cleared Direct", is mainly a way of getting you headed where you want to be going. It will normally be part of your flight plan. If "cleared direct" to the name of an airport, which is also the name of the vor, even if not co-located, I, as a controller, would expect you to fly directly to the airport. Besides, you're going to be vectored and sequenced to final before you reach the airport/vor. And if it's a busy area, you'll be put on an arrival/star. Unless you're flying in a non-radar environment. Then, the control instructions will be a lot more precise, for very different reasons.
When receiving a hand-off from the center, and the controller tells me, "so and so's direct [place name here], descending to 10,000, I expect the aircraft to be heading directly to the field, even though the vor/navaid is a couple of miles away.
Hope this helps un-muddy the waters