Is it wrong to leave your church home?

I am a youth leader at my church, and have been for the last 3 years. It has been wonderful, my family and I have grown so much spiritually. Up until last November, things were great. Until, it came out that our pastor has been lying, cheating people out of money, and turns out he is violent! We ended up quitting the church, and very shortly after, he resigned. We prayed about it, and I knew I had a commitment to the youth group, so we went back, and I resumed my position as the youth leader. Since then, it has gone downhill! Almost all of the members have left, leaving around 20 people attending on Sunday mornings, and I have lost half of my youth group. The new pastor is very old school, traditional, and the music is now traditional hymns. We have been used to a band that plays contemporary music. Anyway, my entire family.... Mom, dad, sister, and husband have stopped going, leaving me to attend alone. I handle teen activities on Wednesday by myself as well. I feel like I am just going through the motions, and not growing at all. I know that we don't follow pastors, we follow God, but I need to connect with my pastor, right? I want to find a new church home so that I can worship with my family and so that my daughter will grow spiritually. (right now she is the only kid!) But, I feel obligated to the teen group. I feel like I need to be committed and fight through this struggle, but I also feel like I need to find a church where my family can thrive. Thanks for your answers in advance!