Will Ryan Lochte and James Feigen Be Banned from Olympic Swimming? USOC Source Speaks Out

If the United States Olympic Committee has its way, U.S. swimmers Ryan Lochte and James Feigen may not be allowed to compete at the 2020 Tokyo Games.

After Lochte claimed that an alleged robbery incident involving himself, Feigen and teammates Jack Conger and Gunnar Bentz took place in Rio on Sunday, a USOC source tells PEOPLE that if the organization had their way, the swimmers wouldn't return for another Olympic Games.

"The thinking is that no amount of gold medals is worth this," the source discloses to PEOPLE. "They totally took over the entire Olympics with foolishness."

"Do you know anyone who won a medal yesterday? No, you don't. All you know is that Ryan peed outside a gas station and then told a story about being robbed to Billy Bush. It's disrespectful to the USOC and every other athlete that trained for years to get to this moment."

The source says there will be severe consequences and the USOC officials aren't going to let the incident slide – they want to send a message to future athletes that the kind of behavior displayed by the swimmers in Rio is unacceptable.

Lochte originally claimed that the foursome was robbed at gunpoint during a taxi ride back to the Olympic Village after a night out. His account was disputed by Brazilian authorities who said their investigation revealed that an altercation had occurred between the swimmers and a security guard after the swimmers allegedly vandalized a bathroom. When the guard reportedly asked the men to pay for the damages, they resisted – leading security to draw a gun and demand payment. It's unclear as to what damage the swimmers actually caused, as there was no visible sign of damage found at the gas station on Friday.

But according to Sérgio Riera, a Brazilian lawyer who represented swimmers Bentz and Conger, two of the swimmers urinated at the back of the gas station and Lochte punched and damaged an advertisement in a metal frame, reports The Washington Post.

On Wednesday morning, Lochte returned home from Rio, but Bentz and Conger were pulled off of their flight for further questioning following the incident and arrived back on U.S. soil early Friday morning.

Feigen is reportedly still in Brazil, as authorities have confiscated his passport.

The swimmer's lawyer said Feigen will donate money to have his passport returned, according to reports.

On Friday morning, Lochte apologized on social media in response to his claim that he and his teammates were held at gunpoint:

"I want to apologize for my behavior last weekend – for not being more careful and candid in how I described the events of that early morning and for my role in taking the focus away from the many athletes fulfilling their dreams of participating in the Olympics. I waited to share these thoughts until it was confirmed that the legal situation was addressed and it was clear that my teammates would be arriving home safely.

"It's traumatic to be out late with your friends in a foreign country – with a language barrier – and have a stranger point a gun at you and demand money to let you leave, but regardless of the behavior of anyone else that night, I should have been much more responsible in how I handled myself and for that I am sorry to my teammates, my fans, my fellow competitors, my sponsors, and the hosts of this great event. I am very proud to represent my country in Olympic competition and this was a situation that could and should have been avoided. I accept responsibility for my role in this happening and have learned some valuable lessons.

Road to Rio: How Did Team USA Train for the Olympics?

"I am grateful for my USA Swimming teammates and the USOC, and appreciate all of the efforts of the IOC, the Rio '16 Host Committee, and the people of Brazil who welcomed us to Rio and worked so hard to make sure that these Olympic Games provided a lifetime of great new memories. There has already been too much said and too many valuable resources dedicated to what happened last weekend, so I hope we spend our time celebrating the great stories and performances of these Games and look ahead to celebrating future successes."

Michael Phelps also spoke out about the incident on Friday. According to E! News, when paparazzi outside a Scottsdale, Arizona, Starbucks asked whether or not he believes Lochte will emerge from the controversy unscathed, he responded: "We have good people taking care of it, so hopefully."

And in a statement on Friday, USA Swimming executive director Chuck Wielgus said: "The last five days have been difficult for our USA Swimming and United States Olympic families. While we are thankful our athletes are safe, we do not condone the lapse in judgement and conduct that led us to this point. It is not representative of what is expected as Olympians, as Americans, as swimmers and as individuals.

"That this is drawing attention away from Team USA’s incredible accomplishments in the water and by other athletes across the Olympic Games is upsetting. The athletes and their remarkable stories should be the focus.

"We’re extremely thankful of the support and efforts from the USOC, Department of State and U.S. Consulate General throughout this process. USA Swimming will undergo a thorough review of the incident and determine any further actions, per our Code of Conduct."

• With reporting by Steve Helling