Prince William Invited Harry and Meghan to Join Him and Kate Outside Castle in Major Show of Unity

In a surprise moment on Saturday, Prince Harry and Meghan Markle joined Prince William and Kate Middleton for a walkaround at Windsor Castle following the death of William and Harry's grandmother, Queen Elizabeth.

Dressed in black, the foursome were seen greeting onlookers and emotionally taking in the flowers and other memorial items left for the late monarch, who died on Thursday at age 96.

Kensington Palace says William, 40, invited the Duke and Duchess of Sussex to join him and Kate, 40, who was recently named Princess of Wales alongside her husband's new Prince of Wales title.

Another source adds that security was not expecting both couples at the walkabout.

For more on Queen Elizabeth, listen below to our daily podcast PEOPLE Every Day.

Prince William thought the walkabout to greet the crowds "was an important show of unity at an incredibly difficult time for the family," a royal source tells PEOPLE.

James Whatling / MEGA From left: Kate Middleton, Prince William, Prince Harry and Meghan Markle

A palace insider tells PEOPLE: "It's such an extraordinary historical moment and also a deeply personal one for the family that you'd hope and think that all members of the family would unite and support [the King] especially. And perhaps some of those wounds can be healed in the process."

It was the second outing for William on Saturday. Earlier in the day, he witnessed the solemn historical moment that saw his father King Charles III officially proclaimed monarch at St. James's Palace in London.

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

Alberto Pezzali/AP/Shutterstock

RELATED: Surprise Moment Unfolds as Meghan and Prince Harry Join Kate and Prince William After Queen's Death

The reunion among William, Kate, Harry and Meghan is poignant, as relations have been strained between the Duke and Duchess of Sussex and members of the royal family in recent years, including an ongoing estrangement from William and Kate.

Before the Queen's death — which occurred while the U.S.-based Harry, 37, and Meghan, 41, were already in the U.K. for charity events — the couple were last in the nation in June to help celebrate Queen Elizabeth's Platinum Jubilee, but there was no public interaction between the couple and William, Kate and the rest of Harry's immediate family members.

They did, however, privately introduce Charles to his granddaughter Lilibet, 1, for the first time, which also gave him the chance to see his 3-year-old grandson Archie, PEOPLE learned at the time.

"It was a fantastic visit. The prince was delighted to see his grandson and meet his granddaughter for the first time," a royal source said, adding that it was "wonderful" to have Harry and Meghan back in Britain.

VICTORIA JONES/POOL/AFP via Getty King Charles III

RELATED: Prince William at 40: Settled as a Dad, Thriving as a Royal — but in "Pain" Over Prince Harry Rift

In a historic first speech since ascending to the British throne, King Charles, 73, on Friday expressed his "profound sorrow" and gave an emotional tribute to his mother, the late Queen Elizabeth.

The new King also acknowledged the "personal grief that all my family are feeling" as they join their country in mourning the death of a monarch who was also a mother and grandmother to his sons, Prince William and Harry.

"This is a time of change for my family," Charles continued in his inaugural address to the kingdom, in which he named William and Kate as Prince and Princess of Wales for the first time.

King Charles also acknowledged his younger son, the Duke of Sussex, who is now fifth in line to the throne. Harry's children with Meghan, Archie and Lilibet, are now sixth and seventh in line, respectively.

"I want also to express my love for Harry and Meghan as they continue to build their lives overseas," the King said.

RELATED: Queen Elizabeth's Funeral Date and Details Announced by Buckingham Palace

William said in an official statement released Saturday, "On Thursday, the world lost an extraordinary leader, whose commitment to the country, the Realms and the Commonwealth was absolute. So much will be said in the days ahead about the meaning of her historic reign."

"I, however, have lost a grandmother," he continued, reflecting on what the Queen meant to his family. "And while I will grieve her loss, I also feel incredibly grateful. I have had the benefit of The Queen's wisdom and reassurance into my fifth decade. My wife has had twenty years of her guidance and support. My three children have got to spend holidays with her and create memories that will last their whole lives."

"She was by my side at my happiest moments. And she was by my side during the saddest days of my life. I knew this day would come, but it will be some time before the reality of life without Grannie will truly feel real," said William.

He went on to thank the late Queen "for the kindness she showed my family and me. And I thank her on behalf of my generation for providing an example of service and dignity in public life that was from a different age, but always relevant to us all."

William concluded, "My grandmother famously said that grief was the price we pay for love. All of the sadness we will feel in the coming weeks will be testament to the love we felt for our extraordinary Queen. I will honour her memory by supporting my father, The King, in every way I can."