NY media reported that 53-year-old Cali was shot multiple times outside his auto on Staten Island on Wednesday evening, local time, by an unidentified man who then fled the scene in a blue vehicle.
Cali, 53, was shot six times, the official said.
Police say there are no arrests and the investigation is ongoing.
He described the suspect as an adult male who sped off in a pickup truck.
He was pronounced dead at a hospital soon after.
A 58-year-old neighbor who gave only his first name, Salvatore, said that he heard a burst of gunshots shortly after 9 p.m. Family members were seen to rush into the street and sit crying next to his body.
"He has a conversation with an individual in front of that residence and that individual at some point in time, it's only about a minute into it, pulls out a firearm and shots are fired", said NYPD Chief of Detectives Dermot Shea.
AFP 2018 OLIVIER MORIN'Mafia Can't Become Extinct Because It's a Type of Behaviour- Criminologist
Two other neighbors who declined to give their names corroborated Salvatore's descriptions.
Cali, who was born and raised in NY, was the nephew-in-law of John Gambino and the great-nephew of Bonanno mobsters Vito Bonventre and Giovanni Bonventre. Castellano was the last New York City crime boss to be gunned down, and Gotti later died in prison in 2002. In 2013 his son, former Gambino boss John A. Gotti, was stabbed in the stomach in Syosset, New York.
The Gambino crime boss was assassinated outside Sparks Steakhouse in Manhattan in 1985.
The Gambino family is one of New York's five historic Italian Mafia families - along with the Genovese, Lucchese, Colombo and Bonanno families. His son Salvatore had been shot three months earlier outside the family home, but survived.
Cali shunned the limelight even after becoming don, ruling from the top while lying low in his affluent Todt Hill neighborhood in Staten Island, where Castellano also held court.
"Frank Cali was a pretty likeable guy, and also this is not a way the mob would kill one of their own", he said. A home on nearby Longfellow Road served as the filming location of the Corleone compound in "The Godfather".
Since Gotti's downfall, the Gambinos - and other mafia families - have lost much of the power they once had over politicians, judges and labor unions.