How can i tell my adoptive family the real reason i won't bring my daughter over?

My daugther is 8 1/2 months old and i've taken her to my adoptive familys house twice. Every time we go there it is really stressfull. I told them very early on that i would not be bringing her over due to their two rough grandsons. But now it is more than just that. The two teenage girls that are still living at home with them are always on and off drugs and always bringing this guy or that guy around. The house is very unclean and my adoptive mother is a person that yells alot which i always hated while i had to stay there. I ran away when i was old enough to do so, but i still have them somewhat in my life now that i am older. She tells me i am being to fussy, over protective, and that i shelter her to much. But my daughter goes out all the time with me to visit my real family, and she has a great relationship with my real mother. My adoptive family is always inviting me over and i always find a reason not to go. How can i tell them in a polite way why we won't come over?