Brook was unveiled as the fourth judge on the talent show last week, alongside regulars Simon Cowell, Piers Morgan and Amanda Holden.
Her arrival was rumoured to have upset Morgan and Holden and surprised hosts Ant and Dec, who were only told about Brook joining the panel minutes before auditions started.
Cowell said in a statement: "I have genuinely enjoyed working with Kelly, she is absolutely lovely and a complete professional. But it has become clear the format doesn't support another judge and we will never add a fourth judge to the panel."
A spokesman for production company TalkbackThames, said: "The format of the show is cleaner and less complicated with just three judges, which was not fully appreciated ahead of filming.
"Producers reviewed the tapes over the weekend and although Kelly brings a huge amount of warmth to the show and we've loved having her on board, the complications it adds to the filming process and format of the show means that unfortunately, and with huge regret, we have agreed with ITV not to proceed with a fourth judge."
Brook, who signed a reported £200,000 contract to become the fourth judge, is said to be "stunned" by the decision.
Her first major role presenting 'The Big Breakfast' on Channel 4 10 years ago was torn apart by the press and audiences and she quit after only five months in the job.
In 2005 she hosted 'Celebrity Love Island' with Patrick Kielty but was replaced in the second series by Fearne Cotton.
.jpg)
.jpg)