The 53-year-old justice told the White House gathering that he would not let the "bitter" confirmation process affect his work on the highest court in the land.
He tried to distinguish between President Donald Trump's nomination of Kavanaugh this year and his own decision not to have the GOP-run Senate consider President Barack Obama's high court nominee, Merrick Garland, in 2016.
Challenges to the law, which enhances penalties in illegal firearms cases based on previous convictions, have returned over and over to the Supreme Court as the justices are asked which types of previous convictions should count.
There were no disruptions in the courtroom, and the justices laughed at one another's jokes.
Trump has now put his stamp on the court with his second justice in as many years.
As the newest member of the court, Kavanaugh will take on a few special jobs. "We wish you a long and happy career in our common calling".
On Tuesday, the court is scheduled to hear two hours of arguments in cases involving long sentences for repeat offenders. "The justices do not sit on opposite sides of an aisle".
He could also be seen having a lighthearted exchange with liberal Justice Elena Kagan, seated next to him. "So I don't know how people are going to react to it".
The cases concerned a Florida robbery conviction and burglary convictions in Tennessee and Arkansas.
Kavanaugh's confirmation gave the Republican president a major political victory ahead of crucial November 6 congressional elections, with Trump's second selection for the nation's highest judicial body.
"You, sir, under historic scrutiny, were proven innocent", Trump said to Kavanaugh. Two other women also accused Kavanaugh of sexual misconduct dating to the 1980s.
Kavanaugh's critics, unhappy with the Senate process, Kavanaugh's record and the way sexual assault allegations were handled, still plan to continue their protests against him.
He later wrote in a newspaper opinion piece that he regretted some of his comments. The US Supreme Court is supposed to be a sacred institution impervious to partisan politics but the newly confirmed justice Brett Kavanaugh gives the court its staunchest conservative majority in decades. The result is a tacit endorsement of Kavanaugh's ruling on the limit of executive power in agency law, one that sets a precedent for future court cases unless the Supreme Court chooses to revisit it. He's already hired four women to serve as his Supreme Court clerks, something that he noted Monday evening was a first, and studied the legal briefs for the cases on deck. Kavanaugh had clerked for the justice in 1993-94 and was sworn in by him in separate oath ceremonies.
The appeals had been brought by an environmental group and companies that supported the 2015 rule that had limited hydrofluorocarbons, which are used in a variety of products including spray cans and air conditioners.
Speaking after his swearing in, Judge Kavanaugh agreed that the process had been "contentious and emotional", adding: "That process is over". Some Democrats have talked of impeaching Kavanaugh, though the process is hard with a two-thirds Senate majority required to remove a federal judge.