Jeff Garlin Speaks Out About the HR Investigations Over His 'Behavior' on The Goldbergs Set

Jon Kopaloff/FilmMagic

Jeff Garlin is speaking out about his alleged inappropriate behavior on the set of The Goldbergs.

The 59-year-old comedian contributed to a demeaning work culture, particularly for the women who worked on the show, according to Vanity Fair, which reported that Garlin "engaged in a pattern of verbal and physical conduct on set that made people uncomfortable."

Garlin responded to the allegation in a lengthy interview with the outlet, admitting to "silly" behavior but insisting he meant no harm.

"I'm on a comedy show. I am always a kind and thoughtful person. I make mistakes, sure. But my comedy is about easing people's pain. Why would I ever want to cause pain in anybody for a laugh? That's bullying. That's just uncalled for," he said.

Garlin clarified that he has not been fired from the ABC sitcom, which is currently filming its ninth season. "There has been an HR investigation on me the past three years. HR has come to me three years in a row for my behavior on set," he said.

In a statement to PEOPLE, a Sony Pictures Television spokesperson said, "The well-being of our cast and crew is of utmost importance to us. This is an employment matter and it is being addressed by HR and production."

A rep for Garlin did not immediately respond to PEOPLE's request for comment.

Ron Tom/Disney General Entertainment Content via Getty Jeff Garlin in The Goldbergs

RELATED: The Goldbergs' Wendi McLendon-Covey Reflects on Costar George Segal's Death: It 'Didn't Seem Real'

Garlin then responded to specific incidents that reporter Maureen Ryan said she heard occurred on the set, including a "physical and verbal" altercation he had with a stand-in.

"I will say the joke was missed and I apologize to him. If someone misses a joke, that's unfortunate to me. I don't like it," he said on the matter, adding that the joke was "completely silly, it was — and it was misconstrued, but there was nothing physical."

Then, Garlin addressed reports that he would touch and hug cast and crew members who did not want such contact.

"I gotta be honest with you. I am a person who hugs for sure," he said. "And it's quick hugs, but I didn't realize that anyone was feeling—so if you want to write that story, that's a true story. It puts the onus on them to say something, and I would gladly respect anyone who said, 'Please don't hug me. I don't feel comfortable.'"

Earlier in the interview, he said he would "never physically come at anyone, for any reason."

The Goldbergs

Garlin was also asked about an allegation that he "sent a text to a Goldbergs colleague saying that they should go to the next table-read wearing only panties." Garlin denied the allegation.

"I didn't write those words. And I'd like to see the text and who I sent it to. But everything that I write like that is about joking. I am respectful of women and women's bodies. And I don't make jokes about that, because I don't find it funny," he said.

The actor added, "I'm telling you that I don't talk that way. That's all, I'm just telling you. That's not in my M.O. Really. It's not."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

As for his future on the show, Garlin said he's unsure if The Goldbergs will progress beyond the current season, admitting that he nearly left before the ninth season because he was "bored" and "frustrated with the parameters that were being put on me by Sony," but ultimately decided to stay.

For the remaining episodes, he said his goal will be "to make it go smooth," adding, "I will not be doing any of my silly stuff or anything, out of respect. I'm just going to keep it on the QT, because you know why? They're [Sony] my bosses. And they have every right to have the set be the way they want."

Elsewhere in the interview, Garlin said, "I have a platform. I will continue to have a platform. So I'm not feeling like I'm silenced, and I'm also wanting to cooperate with the way Sony wants things done, because they're the boss. I get that. And I'm sorry, I truly am sorry, to have hurt anyone's feelings, or made anyone feel greatly uncomfortable. That's for sure."