Occasionally yes, and after enough grimaced stares you get used to not doing that.
Most can be flipped downward (and are often best used at a slight down angle) so when talking with people you make sure it's pointed down or switched to the red low light mode.