Prince Harry Says Windsor Castle Is a 'Lonely Place' Without Queen Elizabeth

Chris Jackson - WPA Pool/Getty

Prince Harry is missing his grandmother.

The Duke of Sussex, 37, connected with the crowd during a surprise walkabout at Windsor Castle Saturday following the death of his grandmother Queen Elizabeth on Thursday. Prince Harry stepped out hand-in-hand with wife Meghan Markle beside brother Prince William and sister-in-law Kate Middleton, where he made an emotional admission about how things already felt different inside the royal residence.

"It's a lonely place up there now without her," Harry told well-wishers, pointing to the castle, in a video shared by The Sun.

"Whichever room she was in, you felt her presence throughout," he added with a smile.

RELATED: Queen Elizabeth's Coffin Seen for First Time as It Travels from Balmoral Castle to Edinburgh

Chris Jackson - WPA Pool/Getty Kate Middleton, Prince William, Prince Harry and Meghan Markle

The Queen relocated her home base from Buckingham Palace in London to Windsor Castle when the coronavirus pandemic began in March 2020 and spent most of her final years there.

For more on Queen Elizabeth, listen below to our daily podcast PEOPLE Every Day.

Prince Harry, Meghan, Prince William and Kate left the gates of Windsor together Saturday to tour the many tributes honoring the late monarch, who died "peacefully" died at age 96 on Thursday.

The outing for the group once known as the "Fab Four" was especially poignant as relations have been strained between the California couple and members of the royal family in recent years.

Kensington Palace says William, 40, invited the Duke and Duchess of Sussex to join him and Kate, 40, who was recently named Princess of Wales, alongside her husband's new Prince of Wales title.

Chris Jackson - WPA Pool/Getty

RELATED: Prince William Invited Harry and Meghan to Join Him and Kate Outside Castle in Major Show of Unity

Another source adds that security was not expecting both couples at the walkabout.

Prince William thought the step out to greet the crowds "was an important show of unity at an incredibly difficult time for the family," a royal source tells PEOPLE.

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more! 

A palace insider tells PEOPLE: "It's such an extraordinary historical moment and also a deeply personal one for the family that you'd hope and think that all members of the family would unite and support [the King] especially. And perhaps some of those wounds can be healed in the process."