Your piece makes me a bit sad because it comes across as you absorbing the burden of other people's thoughtlessness and internalising it as your own failing. I don't think you've done anything wrong by putting boundaries in place to protect yourself and your family. People's actions don't have to be 'abusive' to be harmful.