There is no answer that someone else can give you with any assurance that it's the right one. Both of you have to weigh what you will gain against what you will lose and decide for yourself.
It sounds like you are aware that affairs don't always turn out to be eternal bliss over the long haul. If he cheats on his current wife, you have no assurance he will not do the same with you. I'd bet that if he put the same effort into repairing his marriage as he puts in this affair, things would be different at home.
You should consider that this doesn't just affect the two of you. There are others involved, especially his children, that had nothing to do with the situation getting where it is now. They have a great chance to lose and little chance to gain anything from this. To continue the ongoing affair without the children's knowledge would be a better choice than taking the children's father away from them in my mind.