Freddie Prinze Jr. Says He Wanted to 'Fight' His I Know What You Did Last Summer Director Over Clashes on Set

Courtesy Everett Collection Sarah Michelle Gellar, Ryan Phillippe, Jennifer Love Hewitt and Freddie Prinze Jr. in I Know What You Did Last Summer (1997)

Freddie Prinze Jr. is recalling a tumultuous time making I Know What You Did Last Summer.

The actor, now 47, played Ray Bronson in the 1997 slasher movie, which also starred Jennifer Love Hewitt, Ryan Phillippe, Anne Heche, Johnny Galecki and Prinze Jr.'s now-wife Sarah Michelle Gellar.

In an interview with TooFab this week, Prinze Jr. — who is launching his own horror movie podcast That Was Pretty Scary — called the project a "difficult job" and admitted he butted heads with director Jim Gillespie because he thought the filmmaker wanted a different actor for his part.

"It's not that we weren't on the same page, I knew what the correct choices were for the Ray character. He wanted a different actor, a really good actor named Jeremy Sisto, who I know and I like and respect very very much," Prinze Jr. said.

He added, "I'll give the man this, I think his name is Jim, he made no bones about it. There was no passive aggressiveness, which I hate. He was very direct in the fact that, 'I don't want you in this movie.' So when that's your first job and you hear those words, it just wrecks you, man. It just wrecks you."

Prinze Jr., who returned for the 1998 sequel I Still Know What You Did Last Summer with new director Danny Cannon, said Gillespie gave him "psychotic notes" during production of the first film, "like, 'Don't leave your mouth open. You look stupid when you do that.' That was the exact note, word for word; I'll never forget it."

RELATED: Why Sarah Michelle Gellar Won't Be in I Know What You Did Last Summer Sequel: 'I Don't Have a Head!'

Jesse Grant/Getty Sarah Michelle Gellar and Freddie Prinze Jr.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

"I'm like, I'm either gonna break down or I have to beat this guy's ass," the actor remembered thinking. "Like, those were the only two options in my head. I remember Ryan came up to me and was like, 'Screw that guy, man. How many times did you audition for this movie?' and I go, 'Five times,' he goes, 'Yeah, you earned it. You didn't get offered the role, you earned it. There were less people every single time you went and then it was just you. Remember what booked you this role. Screw his notes. Any note he gives you just say, "OK" and do what you want to do.' He was the first person to say that to me."

Prinze Jr. said it got to the point where it "was very difficult waking up ... and go to work with the right attitude." He claimed Gillespie "made it a point to single me out every time." At one point, Prinze Jr. says he almost packed his bags and "caught a flight and went home." He explained, "I was just gonna quit the business."

"I wanted to fight that guy two or three different times. Once I felt was a legitimate reason, and the other two I was just pissed off, which, that's not right. I'm glad everybody talked me down. In hindsight, I'm not upset, because that movie launched my whole career. I wouldn't have any of the things I have without that movie, I wouldn't have my wife, I wouldn't have all the other movies I've done, I wouldn't have this podcast," he said. "... I'm here because of that struggle and because of that pain and it was those things."

Despite feeling it was a "struggle to finish work every day" and feeling like he was "in pain every single day" making I Know What You Did Last Summer, Prinze Jr. told TooFab the experience "prepared me for this business in a way."

RELATED VIDEO: Sarah Michelle Gellar's Kids Were 'Not Interested' in Watching I Know What You Did Last Summer

"It sounds weird to say this — I'm forever grateful for Jim for being such an assh--- because I've never met one like that since. I've been prepared for every lesser A-hole in the business," he said. "And I'm sure he's a hero in someone else's story. I'm sure he helped someone else out and they loved him. But for me, he took a lot of frustration out on me. He was a first-time director, he didn't have a lot of time, he didn't have the budget he wanted, he didn't have the actor he wanted, and he didn't know how to deal with that frustration."

A rep for Gillespie did not immediately respond to PEOPLE's request for comment.

Back in 2017, for the movie's 20th anniversary, Gillespie told Digital Spy that he was the one who advocated for Prinze Jr. to be cast in the film.

"Nobody wanted Freddie; they thought he was too soft, he wasn't muscular enough, so Freddie probably screen-tested four or five times. He got to the point where he was saying, 'I'm done', and I really had to plead with him to stick with it because I wanted him," Gillespie said at the time. "I thought he was going to be great with it. He went to the gym and worked out, changed his diet and his hair cut. I stuck to my guns and eventually they went, 'Yes.' "

Last month, multiple outlets reported that another I Know What You Did Last Summer sequel is in the works with Prinze Jr. and Hewitt, now 44, both said to return for the new entry. Sony Pictures had no comment when reached by PEOPLE, and reps for Hewitt and Prinze didn't respond to requests for comment about the project.