And if you own your home free and clear, how fortunate! –  housing expenses won't be as major a part of your financial outlay as for most of us. But that ownership might be up for grabs, too. In fact, free and clear ownership (or ownership, period) may or may not be the next best step for you. (My parents built a house in partnership with my husband and me, that we would live in full-time and they would visit in the summers; and they took out a mortgage on their free-and-clear home to fund this, which we paid on jointly. …Which illustrates an easy way to finance the building of your "downsizer", if building takes your fancy.)

Besides, you don’t, necessarily, have to burden yourself with any kind of monthly payments for housing. Many people are full-time RVers, or live on boats all or part of the year; many more are "professional" homesitters; others are caretakers of ranches, motels, apartment houses, construction sites and the like in exchange for rent, so that their housing expenses are very minimal. But not everyone is cut out for these more imaginative, maybe more energetic, and possibly less dependable housing/lifestyle choices.

And, of course, there is always the option of resizing your mortgage payments through refinancing (assuming that, with any points added in, you do truly save by going to a lower monthly payment - i.e., by taking advantage of a lower interest rate).

So certainly, at the least, reducing the financial outlay for your abode can be a sure way to downsize – while "upsizing" your bank account.

(You do know about the possibility of making bi-monthly mortgage payments, don't you? - if your loan allows it - which pays off your principal sooner, saving you money in the long run... even if it doesn't lower your monthly payment.)