If Umno does not stand by what is right then the people would desert us, warned Umno Youth chief Khairy Jamaluddin today.