The Business Secretary faced a hostile reception as he crossed Victoria Square in the city centre, where hundreds of demonstrators had gathered to vent their anger at the Government.
Escorted by several police officers, Mr Rees-Mogg was pursued by a loudly booing and jeering crowd, some of whom shouted “Tory scum”, as he headed towards his party’s annual gathering at the nearby International Convention Centre.
Mr Rees-Mogg played down the protests as a “fact of democracy”.
Speaking to Sky News while being led through the crowd by police, he said: “There have been protests at Tory conferences since time immemorial, it’s nothing new.
“It’s a fact of democracy.
“They’re shouting but it’s perfectly peaceful.
“And the right to peaceful expression of your view is fundamental to our constitution.”
His appointment has been criticised by environmental groups due to his criticism of “climate alarmism” and support for fracking.
Mick, who followed the minister with a placard that read “Tory lies kill”, said he joined the rally because “I hate the Tories”.
The 58-year-old, from Birmingham, who did not want to give his surname, described Prime Minister Liz Truss’s mini-budget as “disastrous for normal people”.
He told the PA news agency: “It’s just the start though – the next step is, to balance the books again, they’re going to cut public services even further. They disgust me.”
He said: “We pay tax to support our people not to subsidise the rich.
“The rich should be subsidising us”.
He also said the Government was “acting in the interests of their people”.
To loud cheers, he said: “We’re going to change this country, we’re going to change society.”
Other demonstrators also expressed their anger over the Government’s move to axe the top rate of income tax for the nation’s highest earners during the cost-of-living crisis.
Jane Elledge, 53, an IT trainer from Bromsgrove, told PA: “Enough is enough really. We’ve had Brexit, we’ve had falling standards, we’ve had people having to work two jobs, people starving, people with no heating, and just the kind of final straw is the announcement of the richest people getting a tax cut.
“Trickle-down economics doesn’t work. We get nothing – nothing for the working people.
“It’s got to stop. Tories out.”
As speakers took to the stage, demonstrators chanted “Tories are not welcome here” and “Tory scum out of Brum”.
Organised by the People’s Assembly campaign group, it was part of a wave of protests sweeping the country amid rising energy costs and falling living standards.
Speaking ahead of the conference, Ms Truss said she is standing by her tax-cutting plan as she refused to rule out public spending cuts.
“I do want to say to people I understand their worries about what has happened this week,” she told the BBC’s Sunday With Laura Kuenssberg programme.
“I do stand by the package we announced and I stand by the fact we announced it quickly, because we had to act”.
She admitted she should have “laid the ground better” for Kwasi Kwarteng’s financial statement, and said the Chancellor had made the decision to cut income tax for the richest.
The crowd later set off to march through central Birmingham.