We don't have many neighbors out my way, but I do check the houses out everytime I go by.  Silly really, it's not like I don't already know what's there, LOL.
But when I go into town I do check out all the houses and stuff.  I don't think it's rude.  It's either thoughts of admiration or boye! they need to mow their yard, etc.  LOL
 
My favorite is passing farm houses w/ horses.  Not only do I check out their homes, landscaping, and colors, but their horses and any other living thing they might have.