It's a bit complicated, but the answer is probably no. If your mother is competent, it's up to her who comes to visit her. Even though she's living in your house, as her residence she should have the right to have visitors. If she is not competent, then you have been appointed to make these decisions for her. It's a bit of a grey area, since having visitors is not strictly speaking a legal or financial decision under the power of attorney or a medical decision under a health care proxy. But the fact that your mother has appointed you to both roles should be sufficient. That said, it might make sense to arrange for your brother to visit with someone other than you supervising.