For security reasons, the two leaders held the talks at a private location -- the historic Filoli Estate in Woodside, California, outside San Francisco -- near where the Asia-Pacific Economic Cooperation (APEC) summit is being held this week.
The two men smiled and shook hands after Biden walked out to greet his Chinese counterpart and then the leaders immediately walked inside.
With the delegations seated around a large conference table, Biden said, "I value our conversation because I think it's paramount that you understand each other clearly, leader-to-leader, with no misconceptions or miscommunication. We have to ensure that competition does not veer into conflict."
Noting their personal history, Biden expressed his appreciation for their "straightforward and frank" discussions in the past.
"We haven't always agreed, which was not a surprise to anyone, but our meetings have always been candid, straightforward and useful. I've never doubted what you've told me in terms of your candid nature in which you speak," Biden said.
He ended his remarks by telling Xi, "Welcome back."
"For two large countries like China and United States, turning our back on each other is not an option," Xi, speaking in Mandarin, told Biden in return, according to a translation.
"It is unrealistic for one side to remodel the other, and conflict and confrontation has unbearable consequences for both sides," Xi continued, according to translation.
"It is an objective fact that China and the United States are different in history, culture, social system and development paths," he said. "However, as long as they respect each other, coexist in peace and pursue win-win cooperation, they will be fully capable of rising above differences and find the right way for the two major countries to get along with each other. I firmly believe in a promising future of the bilateral relationship."
"I look forward to having an in-depth exchange of views," Xi said.
Just before cameras and reporters were then led out of the room, ABC News Senior White House Correspondent Selina Wang asked Xi several times in Mandarin: "President Xi, do you trust Biden?"
Xi took out his translation earpiece to hear her question, looked at her and smiled slightly, but did not respond. She then asked both leaders in English: "Do you trust each other?"
Neither answered and Xi does not normally engage with the press.
Later, he and Biden took a walk around the estate after their working lunch.
Biden told Selina Wang that his meeting with Xi was going "well" when she asked.
Details of four-hour meeting
Fentanyl, artificial intelligence, climate change, the Middle East and Taiwan were among the topics discussed in Biden's four-hour meeting with Xi, according to a senior Biden administration official who briefed reporters afterward.
On fentanyl, an ultra-deadly synthetic opioid that has become pervasive in parts of the U.S., "The president basically said, 'Look, this is one of the worst drug problems the United States has ever faced,'" the official said.
A plan has been drafted between the U.S. and China in order to curb Chinese companies who produce "precursors" to fentanyl and to restrict access to necessary supplies, the official said.
The official noted that the U.S. will continue to monitor China's follow-through on these measures but touted them as important.
The official also said China has agreed to a series of steps that would allow for better communication about each country's armed forces, including what is likely to be contact between senior commanders on both sides.
Biden was very direct with Xi, according to the administration official, and Xi responded with his own concerns, such as what he called undue negativity surrounding China's ruling Communist Party.
The official described the meeting as more personal than the two leaders' previous sit-down, with time for each of them to talk about their spouses -- and a moment where Biden reminded Xi about Xi's wife having an upcoming birthday, since she and Biden have the same one.
Taiwan, the administration official said, is seen by Xi as potentially the biggest and most dangerous issue that could undermine China's relationship with the U.S.
China regards Taiwan, a self-governing island, as a breakaway province.
The official said that Biden told Xi the U.S. is committed to the status quo and asked China to respect Taiwan's election process, but Xi signaled that some larger resolution was needed.
As the administration official described it, Xi indicated that China was not preparing to invade Taiwan despite its ongoing military preparations. The U.S. is focused on deterrence, the official said.
Chinese officials, in their own readout of the Xi-Biden meeting, said that Xi stressed that the two countries could either pursue cooperation in order to promote global order -- or could focus on their own interests over the other.
Xi also detailed how he sees China's ongoing development and modernization. According to the Chinese readout, the two countries should focus on cooperation and managing their differences rather than descending into conflict.
Xi criticized what he called U.S. efforts to suppress Chinese development, including through economic restrictions.
In a news conference after the meeting, Biden said, "I welcome the positive steps we've taken today."
"I believe they're some of the most constructive and productive discussions we've had," he said.
Biden stressed how the U.S. and China want to find areas in which to cooperate while noting that major differences remain -- but a goal for both countries continues to be to avoid outright antagonism.
At the top of his remarks, he highlighted agreements on dealing with fentanyl and resuming military contact.
"We're taking action to significantly reduce the flow of precursor chemicals and pill presses from China to the Western Hemisphere. It's going to save lives," Biden said, "and I appreciate president Xi's commitment on this issue."
"Secondly -- and this is critically important -- we're resuming military-to-military contact, direct contacts. As a lot of you press know who follow this, it's been cut off," Biden continued. "That's been worrisome. That's how accidents happen, misunderstandings."
Going forward, Biden and Xi will "preserve and pursue high-level diplomacy ... in both directions," Biden said.
"I think I know the man, I know his modus operandi," Biden said of Xi. "He has been -- we have disagreements, he has a different view than I have on a lot of things. But he's been straight. I don't mean that as good, bad or indifferent. He's just been straight."
Pressed by reporters about whether he trusted Xi in light of China's commitment to take steps to limit fentanyl production, Biden said: "'Trust but verify,' as the old saying goes. That's where I am." (He was quoting what's known as a Russian proverb made popular in the U.S. by President Ronald Reagan.)
"We weren't going to solve all of the military tension between their army and ours here in a four-hour meeting," National Security Council spokesperson John Kirby said on "Good Morning America" on Thursday.
But the re-opening of military communications could have "a very significant" effect on relations between the two countries, he said.
"Now we'll have a chance to pick up the phone, talk to them about what we're doing, hear what they're doing, and try to deconflict any tensions," he said.
Biden also said he does not expect any Chinese interference in Taiwan's upcoming presidential election, saying the pair discussed the matter as Xi was leaving the summit.
Asked as well about the unfolding Israel-Hamas war in the wake of Hamas' Oct. 7 terror attack, he said he was committed to seeking to free hostages that are thought to be held by Hamas and said he doesn't want Israel to re-occupy the Gaza, the Palestinian territory currently controlled by Hamas and under siege by Israel.
Before leaving Washington, Biden had said his goal was "to get back on a normal course of corresponding, being able to pick up the phone and talk to one another when there's another crisis, being able to make sure our militaries still have contact with one another."
Officials have said a key objective was to restore military-to-military communication that the Pentagon said essentially stopped amid incidents involving the two nations' ships and planes, with tensions around Taiwan especially concerning.
Other items on the summit agenda, U.S. officials had said, included Biden being expected to press China to use its leverage with Iran to persuade Tehran to stop its proxies from attacking U.S. forces in the Middle East and escalating the conflict between Israel and Hamas.
Biden has also called on China not to supply weapons to Russia to aid in its invasion of Ukraine -- and to press North Korea not to do so.
ABC News' Sarah Kolinovsky, Molly Nagle and Karson Yiu contributed to this report.