Going back many moons when I was at school I vaguely remember a teacher mentioning the fact of certain Kings riding on the backs of elephants in a frame made of ivory adorned with silks and jewells and I suppose if you were standing on the ground looking up at the sight of this it would appear that the person was infact in a 'Ivory tower' however, I have slept a couple or three times since then and may have got it a little wrong. But I suppose its a logical thought.
This was supposed to be in India, my brain is starting to hurt a little now and I feel a lie down coming on.