Prince Harry Says Windsor Castle Is a 'Lonely Place' Without Queen Elizabeth

·2 min read
WINDSOR, ENGLAND - SEPTEMBER 10: Prince Harry, Duke of Sussex and Meghan, Duchess of Sussex speak with members of the public at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)
WINDSOR, ENGLAND - SEPTEMBER 10: Prince Harry, Duke of Sussex and Meghan, Duchess of Sussex speak with members of the public at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)

Chris Jackson - WPA Pool/Getty

Prince Harry is missing his grandmother.

The Duke of Sussex, 37, connected with the crowd during a surprise walkabout at Windsor Castle Saturday following the death of his grandmother Queen Elizabeth on Thursday. Prince Harry stepped out hand-in-hand with wife Meghan Markle beside brother Prince William and sister-in-law Kate Middleton, where he made an emotional admission about how things already felt different inside the royal residence.

"It's a lonely place up there now without her," Harry told well-wishers, pointing to the castle, in a video shared by The Sun.

"Whichever room she was in, you felt her presence throughout," he added with a smile.

RELATED: Queen Elizabeth's Coffin Seen for First Time as It Travels from Balmoral Castle to Edinburgh

WINDSOR, ENGLAND - SEPTEMBER 10: Catherine, Princess of Wales, Prince William, Prince of Wales, Prince Harry, Duke of Sussex, and Meghan, Duchess of Sussex on the long Walk at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)
WINDSOR, ENGLAND - SEPTEMBER 10: Catherine, Princess of Wales, Prince William, Prince of Wales, Prince Harry, Duke of Sussex, and Meghan, Duchess of Sussex on the long Walk at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)

Chris Jackson - WPA Pool/Getty Kate Middleton, Prince William, Prince Harry and Meghan Markle

The Queen relocated her home base from Buckingham Palace in London to Windsor Castle when the coronavirus pandemic began in March 2020 and spent most of her final years there.

For more on Queen Elizabeth, listen below to our daily podcast PEOPLE Every Day.

Prince Harry, Meghan, Prince William and Kate left the gates of Windsor together Saturday to tour the many tributes honoring the late monarch, who died "peacefully" died at age 96 on Thursday.

The outing for the group once known as the "Fab Four" was especially poignant as relations have been strained between the California couple and members of the royal family in recent years.

Kensington Palace says William, 40, invited the Duke and Duchess of Sussex to join him and Kate, 40, who was recently named Princess of Wales, alongside her husband's new Prince of Wales title.

WINDSOR, ENGLAND - SEPTEMBER 10: Prince Harry, Duke of Sussex, Meghan, Duchess of Sussex, Prince William, Prince of Wales and Catherine, Princess of Wales look at floral tributes laid by members of the public on the Long walk at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)
WINDSOR, ENGLAND - SEPTEMBER 10: Prince Harry, Duke of Sussex, Meghan, Duchess of Sussex, Prince William, Prince of Wales and Catherine, Princess of Wales look at floral tributes laid by members of the public on the Long walk at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)

Chris Jackson - WPA Pool/Getty

RELATED: Prince William Invited Harry and Meghan to Join Him and Kate Outside Castle in Major Show of Unity

Another source adds that security was not expecting both couples at the walkabout.

Prince William thought the step out to greet the crowds "was an important show of unity at an incredibly difficult time for the family," a royal source tells PEOPLE.

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate MiddletonMeghan Markle and more! 

A palace insider tells PEOPLE: "It's such an extraordinary historical moment and also a deeply personal one for the family that you'd hope and think that all members of the family would unite and support [the King] especially. And perhaps some of those wounds can be healed in the process."