Malaysia and its tiny oil-rich neighbor Brunei today agreed to allow frequent travellers from both sides to cross borders without the need for passports.