Donald Soffer Remembers James Caan as a Longtime 'Loyal' Friend: 'Always a Straight-Up Guy'

Donald Soffer; James Caan

Donald Soffer is remembering James Caan as a tough guy who would do anything for those he loved.

In an exclusive chat with PEOPLE, the 89-year-old Miami businessman and philanthropist recalls first meeting Caan, who died Wednesday at age 82, "in the early 1980s, when I was fishing off of Bimini in the Bahamas."

"He was a rodeo rider and wearing cowboy boots. We hit it off and became friends," says Soffer.

He says they loved to hang out in Europe, California and Miami: "We were two single guys for many of those years, and had a great time in Europe."

Soffer is known for having developed the city of Aventura, Florida, including Turnberry Isle, and tells PEOPLE Caan had his own place there.

"We hung out and did things together all the time," he says.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human-interest stories.

Courtesy of Donald Soffer James Caan, Tom Brady and Donald Soffer

RELATED: James Caan Once Lived at the Playboy Mansion, Joked It Was to "Get Over My Divorce"

According to Soffer, Caan "loved to meet girls and take them out on dates," and sometimes "he would take one out and tell me he would be right back. Then I didn't see him until three weeks later!"

Soffer also says the actor got into "a lot of trouble in Miami by overdoing everything" — but at the same time, he felt deeply.

"I remember how hard he took his sister Barbara's death from cancer in 1981," Soffer tells PEOPLE. "It tore him apart."

Soffer says what he admired most about Caan was his loyalty to his friends, often to a fault. He acted like a tough guy, especially if someone bad-mouthed a pal.

"One time, someone made a wisecrack about me, and Jimmy took him to the ground," says Soffer.

RELATED VIDEO: James Caan Dead at 82

But while Caan "played a criminal" in The Godfather and "liked to act that way in real life," Soffer says "he was a good guy who was a friend when he said he was a friend."

"In fact, Jimmy was always a straight-up guy," he adds.

Case in point? Caan once appeared on The Tonight Show and said during the broadcast, " 'I owe all I have to Don Soffer, who flew me here on his plane,' " remembers Soffer.

"Before [host Johnny] Carson stopped him, he tried to sell apartments in my Turnberry Isle development to members of the audience because he was my friend," he adds of Caan. "He never stopped being loyal."

Matt Carr/Getty James Caan

RELATED: James Caan Remembered by Adam Sandler, Rob Reiner and More: "Best of the Best"

A tweet from Caan's official account Thursday read, "It is with great sadness that we inform you of the passing of Jimmy on the evening of July 6."

"The family appreciates the outpouring of love and heartfelt condolences and asks that you continue to respect their privacy during this difficult time," the tweet added.

A cause of death was not made immediately available.

Caan — a prolific actor with 60 years of filmography under his belt, including memorable turns in movies like The Godfather (1972), Misery (1990) and Elf (2003) — is survived by sons Scott, James, Jacob and Alexander, as well as daughter Tara.