Can they take denied visits to the non custodial if he doesn't see his child?

Hi I was pregnant when my boy-friend left me, after a my son was born i filed for child support, and his dad still didnt want to see him, he told me that he never wanted to see him, cause he was upset cause i was getting child support, now my son is 3 years old and his father only has seen him about 3-4 times in his 3 years of life, now one year ago his father got supervised visits trough the court but he has not visit him even one time, and my questions is if i could take awas his visits cause like i said he doesnt love my son and I really dont want him around, so can i take away his visits legaly and still getting child support?