Former Chief Judge of Malaya Haidar Mohd Noor should decline or withdraw as chairperson of the three-man panel set up to look into the VK Lingam video scandal, said Opposition Leader Lim Kit Siang.