Rajoy accepted his fate ahead of the vote, telling ministers it had been "an honor to leave a better Spain than I found".
"I will accept as a democrat the result of the vote as it is well-known", Mr Rajoy said.
The 46-year-old leader of the Spanish Socialist Workers' Party came to power at a ceremony at the Zarzuela Palace on the outskirts of Madrid, presided over by King Felipe VI.
Socialist leader Pedro Sanchez on Thursday secured the backing of six smaller parties for the motion, giving him the absolute majority he needs to take over from Rajoy as premier. He heads a bloc of only 84 in the 350-seat parliament. His triumph was a dramatic return to the spotlight after being fired as party leader less than two years ago.
Less than two hours after Sanchez had taken his oath to uphold the Spanish Constitution, Catalan chief Quim Torra demanded to meet with Sanchez and speak "government to government" regarding the future of the wealthy yet restive northeastern region.
Sanchez said he meant to call elections before the end of this parliamentary term in 2020, but he didn't say when, and he probably will want to make his mark first with some headline policies before going to the polls. In the end, 180 members of Parliament backed the motion, 169 voted against it and one abstained. Spain's stock market rose after the parliamentary vote, to trade almost 2 percent higher on Thursday's close, while the country's borrowing costs fell, soothed by Sanchez's commitment to respecting a fiscally conservative budget passed by Rajoy.
Meanwhile, Catalan authorities announced their new cabinet on Friday, paving the way for Madrid to end direct rule over the region by excluding four nominees under investigation for their role in its independence drive. PNV's decision on Thursday to support Sanchez's motion to oust Rajoy sent Spain into a frenzy, as pundits and politicians realised the government was doomed to collapse.
In Spain, a no-confidence vote not only removes the present prime minister but requires to choose a replacement as well.
He has also pledged to abide by a national budget recently negotiated by Rajoy, assuaging fears of economic instability in the EU.
Aitor Esteban of the Basque PNV nationalist party, whose support proved decisive for the motion's success, said Thursday such a minority government would be "weak and hard, complicated".
The court sentenced 29 people with links to the PP, including a former treasurer, to jail and ordered the party to pay back 245,000 euros ($290,000) which it received from the scheme to help finance election campaigns.
New Spanish Prime Minister Pedro Sanchez had been barely sworn in Saturday before one of the country's most critical issues facing his fragile government was pressed upon him: ending the Catalan secession crisis.
Mr Sanchez has said he wants to "build bridges" with Catalonia's new separatist government, but opposes an official independence referendum.
Also, the centre-right Ciudadanos (Citizens) party - which has been leading recent opinion polls - is demanding a snap election and is vowing fierce opposition to Sanchez.