@monni No, you can have whatever boundaries you want with him and your child. He is being seriously disrespectful if you've told him this and he continues to ignore you.
I would also be uncomfortable as him acting like this after barely meeting him. It could lead to resentment on your son's...