Most of the harem members will be from multiverses.
And very few will be from his world. He is not gonna make every woman he fuck, his harem members.
He can have a relationship for sex points or a lottery. And that is where a one-night stand comes from.
As for Taboo relations like married women or teachers like this, he will do with them from time to time, because they can constantly give him points unlike a relation that is not taboo. but they won't be his harem member.
But some will be added. Like Martha and Bony or Similar.