@littledot OP. You are not being selfish. You are allowed to make decisions that benefit yourself and your existing family.
It sounds like this person is projecting their own decisions onto you.
I don’t think any of your reasons are selfish. Also - who cares what other people think. Do they...