Netanyahu said on Tuesday that he had met with Israeli residents in Tel Aviv and decided to cancel the agreement.
The Israeli prime minister, who suspended the deal on Monday, spoke to disgruntled Israelis living in and near "Little Africa" - the part of Tel Aviv that has attracted the largest migrant community in Israel.
The decision followed complaints from nationalist members of Netanyahu's government, who had said the deal was "bad for Israel."
"I have listened carefully to the many comments on the agreement. As a result, and after I again weighed the advantages and disadvantages, I decided to cancel the deal," a statement from the prime minister's office quoted Netanyahu as saying at the session.
The fate of some 37,000 Africans in Israel has presented a moral dilemma for Israel, a state that was founded to provide a haven for Jews after the Second World War. Israel's right-wing government has found itself under pressure from the right to expel the migrants.
The deal struck with United Nations' refugee agency UNHCR would have seen about half of the migrants — mostly from Eritrea and Sudan — be relocated to countries in the West. The rest would have stayed in Israel, to be granted residency. Among the countries mentioned as possible locations by Israel were Germany, Italy and Canada.
Officials in both Germany and Italy said they had been taken by surprise by Netanyahu's announcement. Similarly, UNHCR spokesman William Spindler told DW TV that his organization had not received advanced notification from Israel.
"No, we have just learned about this decision a few minutes ago. But we are still hopeful that a solution will be found for these asylum seekers, who are in a very precarious situation in Israel," Spindler told DW. "It's estimated that there's about 39,000 of them, most of them from Eritrea and Sudan, and they are not able to return to these countries, because they fear persecution."
Netanyahu put the deal on hold shortly after its agreement had been announced on Monday.
Naftali Bennett, who is the leader of the nationalist Jewish Home party, tweeted shortly before Netanyahu's statement to say that the agreement as it stood was "bad for Israel."
"Its approval would cause generations of crying and determine a precedent in Israel granting residency for illegal infiltrators," Bennett said.
rc/msh (AFP, AP, dpa)