'Like a more exotic version of Scotland': Charles Spencer on the country that changed his life

South Africa was a place I first heard about when I was a little boy. My mother had her car, a Rover 3500, stolen in Knightsbridge, and we later found out that it had been found in South Africa; I remember thinking that it sounded incredibly exotic, that it must be the place where Tarzan lived. Then I saw the film Zulu and thought: “My goodness, it’s the most beautiful place on earth.”

I was lucky enough to go there when I was a correspondent for NBC in my 20s; this was in the late 1980s, during apartheid, and I was based in Johannesburg where I saw the bad side of life in the old South Africa. But despite the horror of what was happening, I was also struck by the warmth of the people and the natural beauty of the countryside. In the rather fortress-like bureau of NBC, I spoke to my colleagues about my love of their country, and they said: “If you like Jo’burg, you must visit Cape Town.”

I didn’t have the opportunity to do so then, but years later, in 1993, I decided to take my wife Victoria out there for a two-week holiday with our daughters Kitty, Amelia and Eliza.  It rained nearly every day, which was quite unusual at Easter in Cape Town, but it was still beautiful. Like a more exotic version of Scotland, it was hot, hilly and lush. We also visited Bantry Bay and Cape Agulhas, on the southernmost tip of Africa.

In Cape Agulhas, we stayed with friends who lived on an old farmstead. There was no electricity, just paraffin lamps and a small stove in our cottage on the grounds. Waking up in the morning, I realised how far I had taken my children out of their comfort zone; between us and the water was a herd of springbok, grazing, and in the bay were two whales.  One day we took a road trip along the coast to the villages of Arniston and Mossel Bay, and I was struck by the vastness of the landscape; there were little settlements dotted here and there, an ostrich farm or two, but very little else.

Back then, South Africa wasn’t a big holiday destination like it is now. People had not got used to the idea of a post-apartheid world, and it had a reputation for poverty and crime and violence. But to me, it felt undiscovered. The children had a wonderful time, and Victoria and I both started to wonder whether a childhood spent here would be better for them. We saw kids running around barefoot at one of the local schools, and even the boys played rugby without boots because it’s the perfect climate. The lifestyle was focused on the outdoors and everyone seemed sun-kissed and happy. When we left, we agreed it was somewhere incredibly special.

A few years passed and a lot happened in that time: my son Louis was born and my wife and I separated. I went back to South Africa to produce a series called The Great Houses of the World in 1995 and my love for the country grew. My ex-wife and I ended up meeting new partners who both happened to be South African, and suddenly the idea of making a permanent move there didn’t seem such a huge leap.

One miserable winter’s day, we said, “Shall we give the children a blast of something different?” and so in January 1996, all six of us moved out there, the children living with me in a lovely little fishing village called Hout Bay. I remember the dramatic change in them happening very quickly. We took with us four white-faced, snotty-nosed English kids and within weeks they just bloomed. It was incredible to watch their health improve as a result of the climate. And they had private, happy, carefree upbringings.

I’m not saying we were in any way celebrities, but there is no celebrity culture in South Africa, and so there is a big respect for privacy. There was one very persistent paparazzo who would reduce the children to tears, so I had to get a restraining order against him, but apart from that we were left totally alone.

That first family holiday to South Africa inspired a flight of fantasy that I was somehow able to follow through, and I will always be grateful for it. But I knew it couldn’t last forever. I was running my Northamptonshire estate, Althorp, by fax machine and there is only so long you can do that, so we moved back in 2001. But I’m so glad we had those five years out there. It was the ultimate privilege and it transformed all of our lives.

As told to Kirsty Nutkins

Charles Spencer will be participating in Stratford Literary Festival’s Online Winter Weekend (Nov 20-25). Book at visit stratlitfest.co.uk. His new book, The White Ship: Conquest, Anarchy and the Wrecking of Henry I’s Dream (£25, William Collins), is out now. Buy for £19.99 at books.telegraph.co.uk (0844 871 1514).