Richard Simmons’ Former Client Recounts His Troubling Behavior: ‘We All Just Saw Him Deteriorate’

While Richard Simmons‘ disappearance from the public eye may be a surprise to some people just tuning into the Missing Richard Simmons podcast — now the most popular on iTunes — his longtime studio clients say they saw troubling signs years ago.

Although his longtime rep Tom Estey says the fitness guru is “fine” and simply taking a much-needed break, Marie Garofalo, who attended classes for three years at the 68-year-old fitness icon’s recently closed-down Slimmons Studio in Beverly Hills, says that Simmons changed dramatically in the years she attended his classes.

“When I first started going it was amazing,” she tells PEOPLE exclusively. “He was motivational, he had this AOL email address that he gave to people. I remember emailing him once and he got back to me within a day.”

Garofalo continued, “If you’re listening to what people are saying , like that one woman he would call every Sunday — he was so invested in people. That is completely true. And he really seemed to care.”

She says his demeanor began to change, however, after Simmons put out a series of DVDs for Project Hope in 2013.

“It would be a whole big deal for him to come in , and he would take like 20 minutes to change and get ready,” Garofalo says. “So there was that part of it. Then he would be doing the cool down and would just burst out crying.”

She added, “There were lots of regulars who were there every week, and we all just saw him deteriorate.”

RELATED VIDEO: Missing Richard Simmons

Garofalo explains that while the fitness legend regularly shared his personal struggles with food and weight to bond with clients, his random outbursts became more and more concerning.

“ it was like, ‘Oh, ha ha, this is just Richard being funny,’ but then it just started to be like, ‘Is this funny anymore?’ ” she says.

Physically, Simmons appeared to be worsening as well. Garofalo says he began doing workouts with a chair due to knee problems, then inexplicably — and without warning — stopped teaching classes in early 2014.

“I’m in a Slimmons Facebook group, and I remember someone posted and said, ‘It’s so weird Richard isn’t here,’ ” she recalls. “Then it was like, ‘He’s not going to be teaching for a month,’ then it was like, ‘Okay, now it’s two months.’ He completely disappeared, doesn’t email anybody. When people are saying he fell off the face of the earth, he really stopped communicating with anyone. It just completely stopped.”

Estey tells PEOPLE that Simmons didn’t disappear — he is just relaxing at home after dedicating his life to fitness.

“For 40 years, he took care of everyone else but himself,” Estey argues. “And so it’s not that he’s being selfish, he’s just being a person, a regular person, taking care of himself. At the end of the day, that’s really what it is. He’s just a regular guy who’s taking care of, who’s kind of taking his life back a little.”

Garofalo says she isn’t shocked by rumors on the podcast, such as the odd claim that Simmons is being held captive by his housekeeper, because she’s heard them all before. She hopes he is just taking some quiet time off.

“I was a dedicated member of his classes, but there are people who are devastated and just so completely heartbroken,” she says. “I try to just believe that maybe he is just taking time away from the spotlight. He used to teach Tuesday night, Thursday night, Saturday morning, and there would be weeks where he would teach the Tuesday night class, he would go off to New York for some event, be there for three days and be back for the Saturday morning class. He was always on the road, and he’s in his late 60s now, so he deserves to take a nap!”

Estey vehemently denies the podcast’s claims, saying that Simmons’ housekeeper, Teresa Reveles, is “a godsend.”

“Teresa is the housekeeper, she’s the caretaker, she is extraordinary, she is amazing, she takes impeccable care of Richard and she has for as long as I have been working with Richard, so that is a complete load of crap.”

Dan Taberski created the podcast as one of Simmons’ many former friends who are now concerned about his disappearance, which started in Feb. 2014. In the most recent episode, Taberski spoke to Mauro Oliveira – a longtime friend and former masseuse – about the last time he saw Simmons, and why he believes Reveles is holding Simmons against his will.

It is now the most popular podcast on iTunes and the first three episodes are among the most downloaded in the country.