Inside the Survivor Finale After Dan Spilo's Removal: Security, Protests and Contestant No-Shows

Survivor will air its 39th season finale and reunion show on Wednesday night — but things will be very different.

While the show’s finales traditionally include a live reunion show, the final episode of Survivor: Island of the Idols will be pre-taped several hours before it airs. The change comes in the wake of contestant Dan Spilo’s removal from the game after an incident in which he touched a female producer’s thigh as he was getting into a transport boat.

PEOPLE has learned that the format of the final show has been in flux because it was unclear how many contestants would attend.

“There’s a lot that goes into a reunion show,” a production source says. “It’s not just, ‘Let’s put 20 contestants on the stage and let them talk.’ We have to get clips from the show together. We have to be prepared for any contingency. We have unmovable commercial breaks. All of that changes if we only have a handful of contestants there to talk about the game.”

While contestants are paid $10,000 to appear in the final broadcast, some players initially decided to forego the money.

“It was chaos there for awhile,” the production source says. “There were about eight contestants were on the fence, because there would be so much negativity.”

One contestant also tells PEOPLE that many players discussed their options via a group text. “It was like, ‘I’m going, are you? If you don’t go, I might not go, either,” one contestant says. “There was kind of an ‘all for one and one for all’ mentality there, but then it went away.”

Eventually, high-ranking producers — including show host Jeff Probst — personally reached out to the contestants to convince them to appear. One by one, most of the contestants decided to show up for the reunion.

As of Wednesday afternoon, only three of the 20 contestants are not planning to appear at the reunion show, multiple sources tell PEOPLE. Spilo was uninvited from the finale by producers; two other post-merge contestants are currently planning to skip it.

A source told Entertainment Weekly that the decision to pre-tape the finale was made in order to help the players “feel as comfortable and safe as possible while discussing what happened out on the island,” in addition to concerns about security and the sensitive material covered on the show this season.

“Security will be tight,” a second production source tells PEOPLE. “The last thing anyone needs is for a protest to break out in that studio. People are upset. Everyone has an opinion. This is about real-life situations that strike a chord with a lot of people.”

The incident that resulted in Spilo’s ouster involved a member of the production team. After an immunity challenge, Spilo and other contestants were getting into a boat to transport them back to the camp.

At one point, Spilo allegedly touched the female crew member’s leg. He insisted that the contact was inadvertent and accidental as he lost his balance. At least one of the remaining contestants witnessed the incident.

But the show’s production team wasn’t convinced that the contact was merely incidental. After consulting with the show’s legal team, producers eventually decided to remove him from the show.

The incident came just weeks after Spilo, a Hollywood agent, was accused of inappropriately touching a female contestant earlier this season. During the controversial episode, contestant Kellee Kim expressed concern that Spilo had violated her personal space, even after she made multiple requests for him to stop.

On Tuesday, Spilo, 48, apologized for his actions.

“I am deeply sorry for how my actions affected Kellee during the taping of this season of Survivor,” he wrote in an exclusive statement to PEOPLE. “After apologizing at the tribal council when I first learned that Kellee still felt uncomfortable, I want to make sure I do so again, clearly and unambiguously.”

“I truly regret that anyone was made to feel uncomfortable by my behavior,” Spilo continued. “In my life, I have always tried to treat others with decency, integrity and kindness. I can only hope that my actions in the future can help me to make amends and show me to be the kind of father, husband, colleague and friend that I always aim to be.”

CBS is not providing additional comment at this time.

The season finale of Survivor airs Wednesday at 8 p.m. ET on CBS.