Official results are expected to be announced by the end of Monday, 48 hours after polling stations around the country closed. With 95 percent of the votes from 10 of Iraq's 18 provinces tallied - more than half of the total votes - the firebrand nationalist Shiite cleric Muqtada al-Sadr was leading, in what would be an unexpected political comeback.
Leaks from the count indicated that Victory, the new party led by the incumbent prime minister, was on course to win the most MPs in the 328-seat parliament.
Abadi, a Shi'a who has sought to balance the competing influence of Washington and Tehran, finished third in six provinces but came in fifth in the capital.
The record-low turnout - 44.52 percent - has been attributed to problems with the country's new electronic voting system and voter disenchantment.
The complete official answers are all due to be introduced later on Monday.
Sadr, an anti-American firebrand with a large following among Baghdad's urban poor, was once leader of the Mahdi Army, which battled USA forces in 2003.
The Shia-led administration of Mr Abadi has won praise because of the struggle against IS militants, also security has vastly improved throughout the country.
He was viewed as a frontrunner before the election.
Sadr derives much of his authority from his family. Sadr's father, highly respected Grand Ayatollah Mohammed Sadeq al-Sadr, was murdered in 1999 for defying Saddam Hussein.
The alliance is headed by Hadi al-Amiri, a former minister of transport with close ties to Iran who became a senior commander of paramilitary fighters in the fight against the Islamic State extremist group. His father's cousin, Mohammed Baqir, was killed by Saddam in 1980. His rivals were seen as Maliki and Amiri, both closer than Abadi to Iran, which has wide sway in Iraq as the primary Shi'ite power in the region.
The vote came just days after US President Donald Trump pulled out of the Iran nuclear deal. Al-Sadr commanded fighters in the war against the IS group and headed a powerful militia that fought US forces in Iraq prior to 2011, but his 2018 campaign focused on social issues and eliminating government corruption. All three posts are chosen by parliament.
Abadi, a British-educated engineer, was seen by some Iraqis as lacking charisma and ineffective.
Amiri's Badr organization played a key role in the battle against Islamic State. The dissident-turned-militia leader spent more than two decades fighting Saddam from exile in Iran.