Sorry to hear about what has been happening with your dad.
I agree with mbartonsmith; you absolutely do need to report this to the police. If there is some way you can catch the guy at the house when you're there you could even call the police then to make sure he's picked up.
Chances are your dad will be upset with you for this, but it may very well be that your dad thinks this guy is a friend, which he's not.
If you're not in the same city as your dad then maybe you could solicit help from some of your dad's neighbors to just pop in and keep an eye on him.
Ann Carson's advice to go slow is exactly on target. I have struggled with my mom's apprehension of letting me be involved in my parents' care. She once even told me she felt like I was trying to "take over", which I was and still am not. My only goals for trying to help my mom are to make sure she gets what she needs and no one takes advantage of her.
It seems to me as we age we all seem to need help but don't always see it. I think of aging as growing up in reverse; we need less help as we go from young to older, but as we get much older we need more help. That's a hard thing for most folks to realize and accept.
Good luck dealing with this situation.