StuSEL suggestion is good, but I would advise being a little more direct. E.g. "approach I have the XXXX weather and going to use their local altimeter setting." Also I would say this as soon as possible. If you're IFR or even VFR on a code that is setup for MSAW processing and near the MVA the, MSAW alarm may start going off. If the controller isn't prepared he could be unnecessary distracted. At the very least it will save a few unnecessary transmissions. The 1st step for the controller when noticing you're mode C being off is to give you their closest altimeter they have.
This is indeed a good idea. I haven't had an experience where my destination altimeter is that far off of the approach controller's primary airport to make any noticeable difference in my reported altitude, but if I were to ever encounter such a situation, I'd definitely let them know I was switching settings.