The latest episodes of the Netflix series tracks the Royal Family drama from 1992 to 1995, a period that included the breakdown of Charles's marriage to Princess Diana, and the subsequent reveal of his long-held secret relationship with the married Parker-Bowles.
However, rather than their acting skills, viewers have instead deemed Williams and West's portrayal of the secret lovers inaccurate, as the pair are "too good-looking" compared to the real couple.
When quizzed by Variety about the fan reaction, West said: "Well, I think there's something in that, you know. I’ve had worse!"
Williams added: "[The producers] will just have to try and suppress how very deeply hot we are for season six."
West and Williams will continue to portray the now-King and Queen Consort through to the show's conclusion in season six.
Season five follows the tradition of the show with a complete overhaul of the cast as it jumped forward in time.
Charles was originally portrayed by Josh O'Connor in seasons three and four, with Emerald Fennell playing Parker-Bowles.
Emma Corrin was widely praised when, in season four, they joined the cast as the young and naive Diana Spencer.
Australian actress Elizabeth Debicki took over from Corrin during the latest time-jump.
But while Debicki is said to look eerily like the late princess – who died in a car crash in August 1997 – West and Williams have instead been criticised for being "too hot" to play their counterparts.
"They really make the British royals look HOT in the crown like Charles has never looked this sexy a day in his LIFE," wrote one fan on Twitter.
"I need to know is Prince Charles like incredibly flattered by the casting for this season of The Crown? Because he should be this guy has no right being this hot," added another.
While a third said: "Yes Dominic West is hot. But you got to give it to the man. His acting is so on point. The man speaks like Charles. Complains like Charles. Walks like Charles. He knew how to deliver the role of roles perfectly."
Watch: The real dramas that didn’t make it to the final cut of The Crown