Easiest way is with prostitutes, you go up to them and interact and there should be a 'Sex Invite' option using RB. I had a big problem with this at first until I found out the hookers have to be in a certain location for you to be able to invite them. When you find one (if you have the brothel it shouldn't be a problem) wait for them to do the little pose they do before you interact- seems like you can only get the invite option when they do that for some reason. After you've invited them, you don't need to hold their hand, they'll just follow you. Go around til you have as many people as you want and then go to a bed and hit A and then choose if you want protected or unprotected. I spent forever getting four hookers and then had one leave at the bed because I chose unprotected and they declined. Didn't know that could happen until that moment.
For the weapons upgrades you need four other people besides yourself so it's possible to do without the brothel, there's two hookers in the mercenary camp and two in Bowerstone Old Quarter.