@littledot You've gotten great replies already but I'll just add this.
When someone makes a remark like that to me, my default is ask myself whether it's true. "Am I selfish? I don't think so. Here's why I don't think so: First, Reason A; then, Reason B, then, Reason C..."
I've found (usually after the fact) that that's the wrong question. The right question is, "Why does this person have a sense of entitlement to make this judgement?"
Perhaps they've never been challenged before for hurtful statements and have low self-awareness. Perhaps they've dealt with inappropriate criticism growing up and never learned good boundaries. Perhaps they simply suck. We don't know.
I'm not suggesting that you fix this person, just that the real issue is their behavior, not your choice about more children.
If, within yourself, you are feeling conflicted, this can set off that inner conflict. (Sometimes people know we're conflicted and prey on that, but sometimes they say hurtful things out of sheer obliviousness, so I won't make any assumption on what was going on for the friend here.)
But, personally, I can only feel peace about a decision (and full disclosure in my case OAD was a decision that was made for me, but this applies to pretty much any major decision) when I step back from all the white noise of inappropriate comments and do not feel pressured to explain or justify myself.
edit for major typo