Should I stay with him or just let him go?

He sounds like he really cares for you...just stay away for your safety. You know where you guys stand but there is nothing more you can do for him beside protect yourself BC he loves you. Hopefully things will turn around for him soon.