We have 18" RFT shod wheels for Summer and 17" RFT shod wheels for the Winter. Never noticed any excess noise or odd handling. The ride is harder but it's not that bad and I've never experienced any twitching caused by potholes that was in any way unusual. Most poor road surfaces will upset just about any car.
Try taking an X6 with 21" RFT shod 325 tread width wheels out on a rough road......that's a bad ride!
Current: 2011 X6 40d & 2012 Countryman SD All4
Project: 2005 E46 330Cd M Sport convertible