According to the company, the transmission area for the licence is smaller than originally thought, providing a potential audience of 1.3m.
The group, which owns The Guardian and Observer newspapers, said that after carrying out "considerable" research it concluded it was not possible to launch a commercially viable quality product.
GMG Radio chief executive John Myers said: "We've done all the numbers and do not believe it is possible to provide a quality talk station for Manchester while still making a commercial return, when the potential audience is smaller than we had hoped for."
He added: "At GMG, we pride ourselves on our programming output and would not want to submit a proposal that cut corners in order to make money and thus did a disservice to the people of Manchester."
GMG's exit leaves Capital Radio, Emap and The Wireless Group in competition for the licence, for which applications have to be submitted by tomorrow. Ofcom expects to announce the winner in May or June.
Despite pulling out of the Manchester race, GMG Radio will still apply for the forthcoming Solent licence. The Wireless Group has also announced it is going for the licence.
GMG Radio was recently the subject of a £100m bid from Chrysalis Radio, which GMG rejected.
In other news, GMG chief executive Bob Phillis has been appointed as a non-executive director of ITV, along with Baroness Usha Prashar.
Phillis is also chairman of All3 Media, president of the Royal Television Society and a life fellow, previously international vice-chairman, of the US National Academy of Television Arts and Sciences.
Baroness Prashar is First Civil Service Commissioner, chancellor of De Montfort University, a trustee of a number of charitable organisations and chairman of the Royal Commonwealth Society.
ITV chairman Sir Peter Burt said: "We're delighted to have attracted two non-executive directors of such high calibre, who together bring a wealth of relevant media and government experience. They will make a significant contribution as we grow our business."
If you have an opinion on this or any other issue raised on Brand Republic, join the debate in the .