The 44-year-old only took charge in July 2008, but recent poor results have seen Bayern crash out of the Champions League after a 4-0 hammering at Barcelona while Saturday's 1-0 defeat by Schalke leaves the team third in the German league.
"Of course, I am very disappointed at the moment," said Klinsmann in a statement on the Bayern Web site. "Nevertheless, I would like to extend my heartfelt thanks to Bayern Munich, the fans, the trainers, the players and my colleagues for an exciting time. We have laid the foundations for the future. I still believe that the team can be German champion this season."
Former Bayern trainer Jupp Heynckes will take over as head coach for the remaining five league games.
Klinsmann, who along with assistant coaches Martin Vasquez and Nick Theslof, will leave with immediate effect, was told of the decision in a meeting with Bayern team manager Uli Hoeness and chairman Karl-Heinz Rummenigge yesterday.
"We did not come to this decision lightly," Rummenigge said in a statement. "But the results of the past weeks, the way in which they came about and above all, the situation we find ourselves in five games before the end of the season forced us to act out of a sense of responsibility to the club."
Having guided Germany to third at the 2006 World Cup, Klinsmann's Bayern contract had been until June 30, 2010, but he only lasted ten months in his first role in charge of a domestic team and lost seven of his 29 league games.
The alarm bells rang in earnest when Bayern was hammered 5-1 at Wolfsburg on April 4 in the Bundesliga before being routed 4-0 by Barcelona at the Nou Camp four days later in the Champions League.