Am I a bad mom/person?

My daughter is 2 and a half and recently diagnosed with Autism. Her father and I aren't together and we live 2 hrs apart. He loves our daughter and I know he nor his family would do anything to deliberately harm her. When we met four years ago he was the hardest working, responsible man I knew. He picked up new "friends" who introduced him to marijuana and that all changed. THAT has become his life. He lost his job and his apartment. His mother welcomed him back into her home. Very small. Our daughter sleeps on the floor when she's there, which is fine I guess. They are clean for the most part. The mother's boyfriend is a married man on the run from a pricey divorce. The sister's boyfriend sleeps over and he doesn't rub me the right way. To make an even longer story short, 4 months ago, I let my daughter go with him by train to spend the night at the mother's house because they make me feel badly about keeping her away but they don't make me feel welcomed. At that time our daughter hadn't been diagnosed but she was non-verbal and unable to express her general needs. That worried me but I let her go and he wouldn't answer my calls and refused to tell me how she was. He never gave me a time that he was bringing her back. I ended up having to drive 2 hours and along with the police I picked her up. Now, again the family is making me feel badly because I won't let her go back until he gets his life together and they need to do their research on Autism because our daughter has gut issues and is on a special diet that they refuse to abide by. I don't know what to do. I don't want to be depicted as that cliche "baby's momma" who won't let the child's father be in their life. He loves her. I know that. He's just extremely irresponsible these days. He has threatened to take me to court. I hope he knows what he's doing.