Cali, 53, was shot several times in the Todt Hill district of Staten Island on Wednesday evening and died later in hospital.
Pandrella, 59, is accusing of shooting Zito in his own home and stealing money from him - before returning to the scene of the crime hours later to grieve with the older man's family.
Witnesses reported hearing at least a half-dozen shots, police said.
Federal prosecutors in Brooklyn had referred to Cali in court filings in recent years as the underboss of the Gambino organization, related through marriage to the Inzerillo clan in the Sicilian Mafia, according to The Associated Press.
Gotti died of cancer in prison in 2002. He was succeeded in 2011 by Domenico Cefalu, only to be replaced four years later by Cali.
The Gambino family had previously gained a high profile under the leadership of the late John Gotti - who orchestrated the last public killing of a Gambino crime boss in 1985, when Paul Castellano was gunned down outside of Sparks Steak House in Manhattan.
Sources said the getaway vehicle was a blue pickup truck, reported NBC New York.
Citing a source in NY law enforcement, the outlet reports that Cali was respected for his personal family ties to Italy, his traditional Mafia values and his low-profile style - a contrast to the flamboyant decades under John "The Teflon Don" Gotti, who irked New York's other crime families by attracting scrutiny, and column inches, with murders and a series of splashy trials marked by witness intimidation and other tactics. He was serving a 139-year prison term.
Gotti then ran the Gambino family until he was convicted in 1992 of racketeering and five counts of murder.
"There were like six shots, and then there were three more", one witness, who spoke on the condition of anonymity, told the Daily News.
Among law enforcement officials, Cali was known as a "real quiet old-school boss" - one police source told the New York Post. Cali served a 16-month prison sentence for the crime.
Later that year, he pleaded guilty to extortion conspiracy related to the planned construction of a NASCAR speedway on Staten Island - a plan that eventually was scrapped.
The Gambino family was once considered the biggest organised crime group in the U.S., the New York Times reports, but began to decline after Gotti and other senior figures were jailed.
Last October, Bonanno crime family associate Sylvester Zottola, 71, was gunned down at a Bronx McDonald's drive-thru as he waited for a coffee.