I wouldn't say it's that complex, exactly, but magic does have plenty of nuances that the average person wouldn't have any hope of grasping and we do have an understanding of the existence of other worlds. Combine that with the fact that this place presents the idea that traveling to another world does not necessarily mean that you travel to or from an equivalent point on the world's timeline, and it doesn't seem that far-fetched to assume that another pair of universes that are linked together would behave similarly.