That is a slang. When communisam at peak, it was very difficult to get workes at normal wages. Also a migration happenend to gulf and there was a shortfall. People from Madurai, Theni went to Kottayam, Munar and other areas as laboureres in 80s. The common name is Pandian with variations like Pal Pandi, Muthu Pandi etc etc.
Hence they started called them Pandi. But it is a slang. I have travelled extensively in early 90s and this slang was in sue.