I am so sorry to hear about your mother, and her declining mental health. While we always expect physicians and hospitals to make everything better, there are times when their
hands are tied. This seems to be one of those times.
Another time is when a patient has religious objections to treatment. When a patient is determined to have capacity, or to be mentally competent to make decisions about his/her own health care, then legally health care providers cannot override those decisions. The only way around this difficulty is to take your mother to court and have her declared legally incompetent and someone named her guardian. This is very expensive, lengthy, and unpleasant for all involved.
The one positive thing in all of this is that it is illegal for a hospital to discharge a patient into an unsafe environment. Clearly your home is not a safe place. Be very firm with the discharge planner and refuse to allow them to discharge your mother to your home. This forces the hospital to find an appropriate placement, probably in a locked unit. You are not responsible for the cost of her care. Medicare may pick up some of it but ultimately she will have to pay out of her assets or go onto Medicaid. Her method of payment may affect where she is placed at discharge.
Due to her multiple medical conditions, she probably needs to be in a nursing home, where she will have immediate access to medical care. If she is still suspicious, she may need a locked unit initially. Do not feel that you could have affected the outcome and avoided her going to a nursing home. You are not powerful enough to stop the progression of her diseases. It is up to you to continue to love her and to give her moments of joy, in whatever circumstances. If she believes that you will harm her, then send flowers or a card or candy without insisting that you see her.
To answer your original question as clearly as possible, you cannot force the hospital to treat your mother but you can force them to arrange appropriate care.