Jimmy Fallon and Robert Downey Jr. Recall Feeling 'Dejected' Over Failed Audition for 'The Holiday'

"[Kate] Winslet said, 'That is the worst British accent I've ever heard in my life,' " Robert Downey Jr. remembered

Robert Downey Jr. and Jimmy Fallon are recalling their unsuccessful dual audition for Nancy Meyers' The Holiday.

As Downey Jr., 58, and Fallon, 48, appeared on SiriusXM's The Howard Stern Show Sunday among a group of celebrities that included Drew Barrymore and Jon Bon Jovi, the actors recalled meeting at a joint audition for Meyers' classic romantic comedy. The pair read for roles eventually taken on by Jude Law and Jack Black in the 2006 film. The Holiday also starred Kate Winslet and Cameron Diaz.

"We both got called in just as seat-fillers and we saw each other," Downey Jr. said, after Fallon asked whether he remembered the audition. "Jack Black is getting his part and Jude Law is definitely getting my part, but [Meyers] needed someone to read with the gals, and we’re sitting there going, 'It’s about to happen for us.’ "

“I was in a hotel room with Robert Downey Jr., Cameron Diaz and Kate Winslet, and we read this script... I said, 'If Jack Black says no to this movie, I’m so in. I’ll do it for half his price,' " Fallon added.

Although Downey Jr. said that he and Fallon "seemed confident" about their chances to win the roles, he jokingly noted that Winslet did not approve of his accent work.

Related: 'The Holiday' Cast: Where Are They Now?

“I was like, I’ve got to have a better English accent than Jude Law at this point, and Winslet said, 'That is the worst British accent I’ve ever heard in my life,' " he told the group. "I was like, I’ll check out now but I’m taking the gummy bears from the minibar.’ "

Fallon, who shifted from working as an actor to a late night talk show host in 2009 when he took on Late Night with Jimmy Fallon at NBC, jokingly suggested that the failed audition "was the last time I auditioned for a role."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

"Without a doubt, I’m sitting across from Robert Downey Jr., and I go, 'This is the best actor I’ve sat across and did a scene with in my entire life,' " he recalled. "It was mind-blowing for me. I quit the business. I never went back."

Related: Robert Downey Jr. Declares 'Oppenheimer' the 'Best Film I've Ever Been In'

The Holiday ultimately starred Diaz and Winslet as two women who swap homes during the winter holiday season and meet new love interests (Black and Law) in new countries. During their appearance on Stern's show, Fallon referred to the film's premise as "like Airbnb before it existed."

“You remembered the logline, I blocked it out," Downey Jr. joked, as Fallon quipped that he has a tattoo of “us arm in arm, leaving dejected not being cast in this movie.”

“Nancy said to both of us at the same time, ‘This was great. It’s just not a perfect fit. It’s not a perfect fit, ' " Downey Jr. added of how the famous director let the two men down.

“That movie would have been so much better with the two of you guys,” Stern asserted. 

For more People news, make sure to sign up for our newsletter!

Read the original article on People.