A Canadian man who was held hostage with his family for five years has said that the Taliban-linked militants who abducted him and his wife in Afghanistan raped her and killed an infant daughter born in captivity.

Giving new details of the family’s ordeal after arriving at Toronto airport following a rescue operation mounted on Wednesday by the Pakistani military, Joshua Boyle said they had been kidnapped while trying to deliver aid to villagers in a part of a Taliban-controlled region that “no NGO, no aid worker and no government” had been able to reach.

There has, however, been some confusion and questions about events following his release along with Caitlan Coleman and their three children, and Coleman’s father decried Boyle’s decision to visit Afghanistan.

“What I can say is taking your pregnant wife to a very dangerous place is to me and the kind of person I am, is unconscionable,” Jim Coleman told ABC News during an interview in which he also expressed puzzlement at reports that Boyle had refused to board a US military plane after the release.

US authorities have said Boyle was not wanted, but a CNN report quoted a senior US official as saying he had balked at boarding the plane because he feared possible detention on US soil. Reports have also focused on his previous marriage to the sister of Omar Khadr, a Canadian who spent 10 years at Guantánamo Bay after being captured at an al-Qaida compound in Afghanistan.

Boyle denied that he had refused to make the return trip aboard a US military aircraft and had chosen to fly back from Islamabad to Canada on commercial airlines via London. “Obviously, it will be of incredible importance to my family that we are able to build a secure sanctuary for our three surviving children to call a home,” Boyle told reporters after arriving at Toronto’s Pearson international airport, wearing a black sweatshirt and sporting a beard.

Reading out a statement to journalists from a small notebook, he used much of it to hit out at the family’s abductors, the Haqqani network, a group deemed a terrorist organisation by the US.

“The stupidity and the evil of the Haqqani network in the kidnapping of a pilgrim … was eclipsed only by the stupidity and evil of authorising the murder of my infant daughter,” Boyle said, in a calm voice which cracked at the mention of the child.

“And the stupidity and evil of the subsequent rape of my wife, not as a lone action, but by one guard, but assisted by the captain of the guard and supervised by the commandant.”

He did not elaborate on what he meant by “pilgrim”, or on the murder or rape. Coleman, who was not at the news conference, was preparing to travel to Boyle’s family home in Smiths Falls, 50 miles (80km) south-west of Ottawa, with their three children, all of whom were born in captivity.

He said the Taliban, whom he referred to by their official name – the Islamic Emirate of Afghanistan – had carried out an investigation last year and conceded that the crimes against his family were perpetrated by the Haqqani network.

He called on the Taliban “to provide my family with the justice we are owed”. “God willing, this litany of stupidity will be the epitaph of the Haqqani network,” said Boyle.

Boyle also revealed that one of their children – called Jonah, Noah and Grace – was in poor health and had to be force-fed by the Pakistani military after their liberation in an operation that was carried out on the back of a tipoff by US intelligence.

“In the last three days I have actually only seen one US soldier and we had to speak very briefly and very cordially about the medical attention that the Pakistani medical team was providing to the injured child,” he said.

Boyle had reportedly told his parents on Thursday that he had been in the boot of the car with his wife and children when shooting began and that he was hit by shrapnel. The last words he said he heard his captors shout were: “Kill the hostages.”

‘There are good Muslims, there are bad Muslims and then there are pagans,’ says Joshua Boyle – video

Fresh footage of the family following their release emerged on Saturday after it was released at a press conference held by the Pakistani military’s media wing

“The truth was that the car was riddled with bullets the ISI [Pakistani intelligence] and the army got between the criminals and the car to make sure that the prisoners were safe and that my family was safe,” he said.

Of the kidnappers, he said: “There are good Muslims and there are bad Muslims and there are those who are not Muslim … they are pagan. The criminals who held us, they were not good Muslims. They were not even bad Muslims. They were pagan.”

The Pakistan military’s spokesperson, Major Gen Asif Ghafoor, said that the rescue operation was launched after information provided by US intelligence indicated that the family was being moved to Pakistan from Afghanistan. Three armed men and a driver who were in the vehicle fled to a nearby Afghan refugee camp, he said.

In a separate interview with the Toronto Star on Thursday, Boyle said his family looked forward to rebuilding their lives even though they were “psychologically and physically shattered by the betrayals and the criminality of what has happened over the past five years”.

“But we’re looking forward to a new lease on life, to use an overused idiom, and restarting and being able to build a sanctuary for our children and our family in north America,” he said.

“I have discovered there is little that cannot be overcome by enough Sufi patience, Irish irreverence and Canadian sanctimony.”

Their release took place nearly five years to the day after the couple lost touch with their families while traveling in a mountainous region near the Afghan capital of Kabul after embarking in 2012 on a trip that took them to Russia and the former Soviet states of central Asia.

Separately, in a statement and brief words to the Associated Press news agency, Boyle appeared to express disagreement with US foreign policy.

“God has given me and my family unparalleled resilience and determination, and to allow that to stagnate, to pursue personal pleasure or comfort while there is still deliberate and organised injustice in the world would be a betrayal of all I believe, and tantamount to sacrilege,” he wrote.

Nodding to one of two US state department officials on the flight from London he added. “Their interests are not my interests.”