There's not much to go off here, but the only thing I can tell you is to talk to your mother. You said that she knows that he gets on your nerves, but what exactly does that mean? How do you show this to her, and what kinds of things does he do that bothers you? If you can speak to your mom privately and explain this to her in a way that she can understand better, then maybe things will get better.
I don't know how your mom feels, but if she is dating him, she probably cares for him. Remember, if you insult him, you will also hurt her. In your case, if your mom was to say bad things about a friend of yours, would you also not feel a bit defensive and hurt yourself? So be careful with the words you use, and just try to be mature about how you go about explain how you feel about this man.
Likewise, family nights are something that should be very important to the family. Talk to your mom and tell her that you aren't comfortable having him there yet. Let her know how you feel, other than 'my brother just likes to make me suffer'. Tell her why it bothers you, and let her know that maybe once in a while you would be accepting of it... but to make it a regular occurence is not a change that you're ready for.
All in all, just be honest. It's hard to say how your mom will response, I don't know how she feels about this whole manner. But in this case, the best thing to do is just talk to her about it. If you don't, then she'll never know exactly how you feel, and then the problem definitely won't get fixed. So it's worth a try, right?