privacy and geolocation

google released latitude and privacy international warns us about a major security flaw.

…second party can gain physical access to a user’s phone and enables Latitude without the owner’s knowledge…

if someone has physical access to your phone, they can do whatever they want without your knowledge. I am just not sure how this is any different than me taking your phone, installing a key sniffer and handing it back to you. Is there really a “Major Security Flaw” here?