Steve,
The sad thing is that my husband's family should get to see their granddaughter / niece/ great-granddaughter / cousin, etc. but the sad thing is that my father in law is an "authority" and he would do whatever he could to secure the happiness of his son. I know she'd not return and that is why I can't allow it to happen, not even once. So sad.
I know how it is there with powerful families. They will do anything to help their kids using "influence".
Do whatever you can so he cannot take her without your permission. Protect yourself and her. Still, make sure your husband is really planning on that.
It seems he has some bitterness towards the US and his life here, like he could be doing better in Brazil. Similar to Bruna's case it seems.
The other thing I did not like is how he left you to his own devices in Brazil. See, I was married to a Polish guy for 16 years, plus 4 years of dating. I went to Poland many times. I also did not understand the language at all. But I tried to like the place, and I enrolled myself in Polish courses.
When there is love, people overcome all these things. They decide jointly what is best for the FAMILY, not just for one. It seems that your marriage is on the rocks. And since you don't consider living in Brazil, then there is no hope if he wants to go back.
Protect yourself with documentation and the law as much as you can following everyone's advice here. Maybe you should ask him outright what what are his plans for the future. If he tells you adamatly he wants to go back to Brazil, tell him if that is the case you want to have a fair custody agreement where he can come and his your daughter. Tell him firmly her primary residence will be here in the US with you. If he shows ANY intention of makes any comments he does not like that idea, you know not to send her to Brazil for vacations until she completes 18 years of age.
Protect yourself the best you can, and talk to him and find out what his plan is. You may find out stuff from his body language, some people are ot good liars.