Caring for someone with dementia and another life threatening disease like cancer that has already metastasized can be very demanding. You try to give comfort and treatment but you are
not sure it is what your mother-in-law wants or needs.
Often metastasized cancer can affect the brain and at this point she may be exhibiting confusion or other dementia symptoms from its spreading. Perhaps a brain MRI or scan might be helpful to assess this for further treatment.
As you probably are aware, if she is exhibiting behavioral pain (anxiety, anger, restlessness), it may not matter whether it is from the cancer or the dementia. You cannot separate the two and you would treat it the same way- with tolerance, reassurance, and assistance. If she is in pain and unable to express this, her behavior might be her way of expression and you can try different methods of pain relief from a massage to a warm bath to a sweet treat.
It sounds like it is hard for the family to understand and accept your mother-in-law's dementia and this makes it more challenging for you. Since you are providing the day-to-day care, maybe you need some respite if they would be willing to share the caregiving.