My experience has been that men get far more upset at the idea than women do. All the ladies' rooms have private stalls, so what is the big deal?
The real issue isn't trans women using women's rooms but straight men trying to exploit the situation. Seems to me that trans women, if they didn't transition too early, are more of a protective presence than a menace. Honestly, I have never been molested by a gay person, man or woman, but high school and junior high were lousy with groping guys. My first apartment was in Hell's Kitchen and, being a woman, I was never harassed by the working ladies, either (although the same cannot be said for visiting guys). When I lived in the Village as a teen, I walked around a lot at night and I always felt safe in the gay neighborhood spots.
I have never met anyone I knew to be a trans man, but I've been acquainted with a couple trans women - a vanishingly small minority of the population even in the big cities - and I don't see a threat from them. This is a weird moral hysteria that is being exploited for political purposes to gin up the conservative vote.