The Untold Story of Judy Garland’s Tragic Love Affair with Sid Luft: The Affair, the Abuse Claims and Her Downward Spiral

Judy Garland’s tumultuous 15-year romance with Sid Lift, a former boxer and test pilot, was marked by both career highs and a dark descent into drugs and rumors of abuse.

Only four short years after ending her marriage to Luft, the Wizard of Oz icon died tragically from an accidental barbiturate overdose in 1969 at age 47.

In a new memoir, Judy and I: My Life with Judy Garland – crafted from notes left unfinished before his own 2005 death, and excerpted in this week’s issue of PEOPLE – Luft details Garland’s painful decline of self destruction.

“My dad loved her,” their son Joey Luft, 61, tells PEOPLE . “He wanted to protect her.”

Luft and Garland first met in a Manhattan club in 1950, and the attraction between them was magnetic and instant. They began an affair while the actress was still married to director Vincente Minnelli (with whom she had daughter Liza Minnelli).

The early days were categorized by “secret night rendezvous” in Los Angeles, Luft writes.

“I did not want to fall in love with a married woman; it seemed chancy,” he writes in the book.

Though he knew she was troubled, Luft felt protective – and fell hard. He became her manager by the next year, and as Garland began her divorce proceedings – Luft writes, “I knew I was in love with Judy.”

Soon after, Garland was pregnant with his child while they worked on a show together in New York.

“I found myself saying, ‘Of course I want your baby, but we’ve got a show to do,’ ” Luft writes of the unplanned pregnancy in the book. “Because of my negative reaction, Judy didn’t confide in me where and when she was going to have the abortion.”

Despite serious warning signs of Garland’s difficulties – such as unexplained scars on the inside of her wrists – the couple married in 1952. In the years that followed, her dependency on various pills quickly began to dominate their relationship.

“She had been encouraged to take pills by the studio bosses and then she began to rely upon them,” Luft writes. “Mickey Rooney and Elizabeth Taylor, among others, were similarly caught in the studio dope traps. This feeding of narcotics to children was a dark secret known only to those connected to the studio.”

Garland would spend “nights in the den with the door closed,” Luft details, explaining that his wife became “more and more remote,” once attempting suicide by cutting her throat with a razor blade.

Though Luft says his wife found it “virtually impossible” to work without being medicated, she found some sustenance in their children – Lorna, born in 1952, and Joey, who arrived three years later.

For more about Judy Garland and Sid Luft’s marriage, pick up this week’s issue of PEOPLE — on newsstands Friday

Still, Luft tried to intervene, once taking her to an AA meeting, but he writes that “If I were to show concern, she’d abruptly tell me to ‘f— off.’ ”

Eventually, the pair began to live “virtually separate lives” in the same home, with the children residing in one wing that Garland occasionally visited. “ wouldn’t realize she was stoned,” Luft writes.

They finally divorced in 1965, with Garland telling a judge at their hearing that Luft had been abusive, according to numerous reports. The New York Times said Garland told Edward R. Brand of the Superior Court, “He struck me many times. He did a lot of drinking.”

Though Luft denied the claims, he drew ire again once decades later, when he attempted to sell her Academy Award, only to be blocked by the Academy of Motion Pictures Arts and Sciences.

In Judy and I, Luft writes that Garland’s claims to her attorney Jerry Giesler that “Sid hit me” or “Sid threatened me” were “all untruths.”

He further details an incident where Garland knocked on his hotel door at the Stanhope, only to begin screaming, “He’s hitting me, he’s hitting me!” as he let her in.

“Just then, a private detective and a cop busted in. The two men had me by the neck, and the other by the arms,” he writes, claiming that the whole scene was orchestrated by Garland so she could jet off to England with their children.

Luft writes of her shocking death in London in 1969, “Emotionally, none of us ever overcame it.”

“Judy Garland was a very rare mix of shattered nerves and insecurities, self-destructiveness and suicidal tendencies but also a true genius,” he adds. “She was to me the greatest talent who ever lived.”