First, as long as your mother sells her house to your sister, or to anyone else, for fair market value, the sale will not DIRECTLY affect her eligibility for Medicaid
nursing home coverage, which is what it seems you are concerned about, when you mention the "look-back" period, given your mother's early-stage dementia. However, the key word is "directly," because the money your mother receives for the sale of the house becomes part of her assets. Whenever she applies for Medicaid nursing home coverage, Medicaid will look at her income and assets to see if they are low enough for her to be eligible. If at the time she applies she still has money from the sale of the house, that money will be considered (along with any other assets she has) in determining whether she's eligible.
The second part of your question is whether there is a legal arrangement that can ensure that your mother has a right to continue living in the house after she sells it to your sister. The answer is yes, there's a legal arrangement called a "life estate," which makes the seller's right to stay in the house a condition of the sale of the house. This has to be arranged by a lawyer (or sometimes by a real estate broker/agent who's experienced with life estates). You need to consult with a lawyer who handles real estate transactions, or with a lawyer whose area of expertise is "elder law."