Johnson, who was in charge of Microsoft's Windows and web operations, is expected to step into the role of chief executive officer at Juniper Networks, a networking hardware maker, following a transition period.
Since 2005, Johnson has worked as president of Microsoft's platforms and services division, which included the Windows operating system and Windows Live programs such as web email and instant messaging. The division also included online advertising, search and Microsoft's MSN sites.
He joined Microsoft in 1992 and has been the public face for the company's search and online advertising strategy, which is struggling to catch market leader Google.
In November, Johnson laid out aggressive goals, including capturing 30% of US search queries.
However, it became clear that Microsoft's internal search and advertising efforts were not making enough headway, spurring Microsoft's bid for Yahoo!, in which Johnson was instrumental alongside chief executive Steve Ballmer.
According to recent research by comScore, Google commands around 60% market share for search engines worldwide, while Yahoo! is second with around 14%.
Third place is occupied by Chinese search engine Baidu with about 5%, while Microsoft is fourth with around 3.5% share worldwide.
Johnson also oversaw the $6bn acquisition of online advertising company aQuantive in 2007, which has boosted Microsoft's web ad revenue.
Microsoft said it will search for a new sole leader for its online services business, while senior vice-presidents Satya Nadella and Brian McAndrews take charge of its engineering and advertiser and publisher solutions divisions during the interim.
Last week, Microsoft said its online business lost $488m in the quarter, more than double its loss last year.