Remember not to hold anyone's hand after you invite them for sexy time. They will still follow you around while you go up to as many whores as you want and sex invite them. Once you have your desired crowd, just head to a bed and then choose sex like you usually would.
You do not have to be king, and you don't have to have built the brothel (though the brothel is very useful; packed with whores), just remember; don't hold anyone's hand!