Paul does say that marriage makes you torn between the ministry and your marriage. I don't think it's right to neglect your family. If you've chosen to have a family, then it is your responsibility to be a part of it and take care of it.
I think a lot of ministers treat their families like pets. They had children when they should have bought a CHihuahua.
On the other hand, if you marry a minister, you have to realize that you are going to be a juggled priority. It's the nature of the beast.
I don't think people give this issue enough thought when they jump into a marriage like this.