Carter is understood to have lost out in battles with key Brown advisers, according to reports in the Independent on Sunday and today's Daily Telegraph.
The paper said that Carter, the former Ofcom and JWT chief executive, is to move to a lower-key role after a terrible summer for the prime minister, who has been assailed by a string of gaffes, a worsening economy and a possible leadership challenge from foreign secretary David Miliband.
Carter has only been in his Downing Street role since January when he joined from City PR firm Brunswick, which was the role he took up after leaving Ofcom.
Carter was appointed as "chief strategy adviser and head of communications" in an attempt to revive Brown's Number 10 operation. As part of his efforts he brought in new blood including WPP Group's David Muir and Which? communications chief Nick Stace.
However, his arrival ruffled a number of feathers and sparked the exit of long-term advisers, including speech writer Spencer Livermore among others. There was also consternation that Carter was earning as much as Brown with his £180,000 a year package, not including perks.
It is understood that Carter had planned to serve only until the next election before returning to the private sector, but his tenure could now be cut short.
According to Carter is moving out of frustration with infighting at Downing Street and was close to resigning last week.
Carter's move comes ahead of a talked of economic relaunch by Brown next week, which will look at tackling the dire housing market and helping those struggling to meet soaring energy bills.
Brown's plans for a relaunch were hit when chancellor Alistair Darling issued a doom and gloom statement on the economy at the weekend, saying the slump will be "more profound and long lasting than people thought".