The far-right, anti-immigrant party was founded just four years ago as an anti-euro force.
Chancellor Merkel, who has held the post since she was first elected in 2005, will face a Parliament where the far-right AfD wants to launch a tough debate on the influx of migrants fleeing war, poverty, and persecution.
After 12 years in power and running on a promise of stability and economic strength, Merkel's Christian Democratic Union/Christian Social Union (CSU) bloc scored 33 percent, according to the final results, against 20.5 percent for the Social Democrats under challenger Martin Schulz, who pledged to go into the opposition.
Lindner said his party would only enter government "if there are major changes in German politics - that's our mandate from our voters". Timo Lochocki, a political scientist at the German Marshall Fund in Berlin, told Science before the election that AfD's appeal comes from nationalistic rhetoric rather than its policies on science and higher education. Maybe the AfD is less Nazi than we might fear, and it has the usual complement of misfits and the confused: but there should be no room for complacency.
Germany has had MMP for much longer, but a quarter of a century after its adoption here we should not be waiting for one party to give us a decisive result. Populist and nationalist parties in countries that have taken in far fewer refugees have garnered far greater support than the AfD.
The spokesman said Pretzell and another AfD lawmaker in NRW's regional assembly who is also leaving the party had made the decision for reasons of "personal integrity".
Even in eastern Germany, with more than 21 percent of the vote, it has become the second strongest party in the region, and is among the most popular for the male voters in East Germany. The SPD's collapse is consistent with a deep crisis that has engulfed many of Europe's social democratic parties in recent elections from the Netherlands to France.
Still, government coalitions and alliances between different parties are the rule in Germany.
The Social Democrats are likely to remain the chief opposition party, weakening the political impact of the AfD despite its third-place showing, said Sergey Lagodinsky, a political activist with the Green Party and member of the Berlin Jewish Community Council.