Papua New Guinea's Immigration department did not respond to a request for comment.
One man tweeting from inside the camp on Friday morning, Ezatullah Kakar, indicated numerous men were complying with police demands to leave.
Papua New Guinea's Post-Courier newspaper reported on Tuesday that immigration officials would begin evicting the men on Wednesday, the fourth such deadline imposed on the refugees to leave since the camp's closure on October 31. One refugee, a journalist, was detained and released.
"PNG authorities have cut food, water and electricity and have told the remaining men they are squatters on defence force property", the ABC says.
Men inside the centre said local police who entered the centre today were aggressive and were destroying their belongings.
"This morning police attacked the prison camp and the refugees are saying that they beat them", Iranian journalist and refugee Behrouz Boochani, who has acted as a unofficial spokesman for the men, tweeted early Friday.
PNG Police Commissioner Gari Baki on Tuesday said his officers would not use force to move the men.
"We would expect the number which, up until this morning had been about 370 people within that centre, would drop well below that now", Mr Dutton told Sky News on Thursday. "We are under attack", he said.
Manus Island was one of two processing centres used by the Australian government to detain asylum seekers and refugees who arrived in Australia by boat.
Amnesty International's Pacific Researcher Kate Schuetze, who has just returned from Manus, said the country where the men on Manus sought refuge - Australia - has violated their rights at every turn.
"Immigration officials went in and explained to them that they really had no legal standing. voluntarily they packed their bags and moved across", he said.
New Zealand has offered to take some of the refugees from Manus, but the Australian government refused the offer, with Dutton arguing it would encourage smugglers.
So far, the United States has agreed to resettle 54 refugees, but those on Manus Island are yet to be relocated. So far, 54 have been resettled while another 500 await assessment of their status.
"We are all crying leaving the camp to another camp", he wrote.
"The Australian taxpayers have paid about $10m for a new facility and we want people to move", he said.
The 12 former Australians of the Year called on Australia to restore basic services to the camp on Manus Island and allow doctors to provide care there as reports emerged of an operation on Thursday to evict the remaining asylum seekers.
The letter's signatories include Australians of the Year Rosie Batty, Ita Buttrose, Simone McKeon, Patrick McGorry, Mick Dodson, Tim Flannery, Fiona Wood, Fiona Stanley, Gustav Nossal, Peter Doherty, John Yu and Robert de Castella.
While the detainees occupying the center had been active on social media posting photos and video, several hours on, Zazai claims the group was surrounded and anyone caught filming was being beaten.