Some of Penang Chief Minister Lim Guan Eng's female colleagues believe that his "racist grandmother" jibe at opposition leader Jahara Hamid was "inappropriate", but stopped short of criticising him.