Your husband's disability does not take away from the fact he is abusing you.

If an affair is what allows you to continue with the day to day care of someone who no longer sees you as an autonomous human being with needs of your own, so be it.