It’s frustrating, huh? And in your case, it really sucks because even if you place the boundaries, it sounds like his mom and sister enable him right around them.
It sounds like he doesn’t know how to be a dad on his own (like my ex).
With the enablers, its hard to see the fix.
You’ve got my sympathy. My best thought is that, if he is scared of being a dad (scared of being incompetent or failing or being criticized — which sounds likely), how could you help him with that? I would reward any approximation of the behavior you seek. So then you have to figure out what it is that is rewarding to him and would make him want to do it again? Attention? A compliment? It really is like parenting a child.
I wouldn’t suggest any of this effort if it weren’t for the kids. They deserve better, you know? (I know you do. XOXO)
It’s easy to say that he’s not worth it. (And he may not be.)
But the kiddos are.