Do not stop inviting people over. Despite the negativity and apathy, your husband does not need isolation. Moreover, YOU do not need to be separated from social interaction - that can be so dangerous for a caregiver.
It is very painful for many PwPs to follow conversation, to do the requsite social multi-tasking. Keep gatherings small but do not stop having them or attending them.
Don't expect or require him to contribute much but do try to bring up topics on which you know he has something to say. Don't complete his sentences but be prepared to prompt occasionally.
We have dogs and cats who interact well with each other but there is no question that the dogs really need the socialization of outside dogs to feel like vital dogs. They just bark until they have established the parameters. Why should it be any different for your husband.
When he objects to social gatherings or even drop-in visits, explain that he is important to these people, always has been and always will be. You can further explain that they like to be in his presence and they need his tolerance and acceptance.
From time to time remind the kids (perhaps a pre-set signal)to slow things down enough so that he can follow.
Consider PD counseling if possible. He gets tense, anxious and depressed by his situation. He may need breathing exercises to help him relax in public situations. Just the simple breathe in slowly through the nose, hold and feel the relaxation begin to flow throughout the body, slowy exhale. Repeat. That's one you can do anywhere.
Just remember that what he feels now is not what he felt before PD. In his case, his identity has changed as has the way he views the world around him. He may be having a problem with that imbalance.