It is believed that the ITV companies Carlton and Granada are behind moves to commit a boardroom coup and push Phillips out.
However, rival broadcasters are at loggerheads with ITV, determined that Phillips keep his position despite the debacle. Barb's shareholders are the IPA, the BBC, BSkyB, Channel 5 as well as Channel 4 and ITV.
ITV and Channel 4 have lost the most in terms of audience and revenue because of the new panel but, in public at least, all shareholders have rallied around the beleaguered Barb and its chief executive, Caroline McDevitt, who used to work within ITV.
But at the Granada interim report earlier this month, its chief executive, Steve Morrison, became the first to break ranks when he acknowledged that there was "concern about the new Barb audience measurement system".
Phillips, a former director-general of the IPA, said that he had heard nothing of the rumour and that he hoped to continue doing the job for some time to come.
The Barb panel is not expected to be up to its full strength until December, some 12 months after it was originally due to be running. In its six months of reporting Barb has produced audience figures that have surprised both media owners and agencies, particularly for younger audiences.