'World' seems to be the generally accepted term. But then there are those of us who come from the same world at different points in time. I come from Earth in the year two thousand and two, and I've already made a friend from the eighteen sixties.

But yes, 'world' seems to be the word unless you know for certain that the two of you are already from the same world. Then it's 'time', or 'place'.