Two in five of those asked by Which? said they felt intimidated and the same number said they were distressed by unsolicited marketing or sales calls.
The consumer watchdog is urging the Government to ensure bosses of the companies behind such calls are held accountable.
Under new proposals, the Information Commissioner’s Office (ICO) could be given stronger powers to hold company directors directly responsible for breaking the law with further fines of up to £500,000.
Currently only businesses are liable for such fines, and some directors try to avoid paying the penalty by declaring bankruptcy, only to open up again under a different name, the department said.
A three-month consultation on the issue launched by the Department for Digital, Culture, Media and Sport was due to close on Tuesday.
MOST POPULAR ON YAHOO NEWS UK
Britain’s most popular and least popular cities have been revealed
NHS warns ‘no deal’ Brexit could lead to hospital drug shortages and spread of disease
Mother denies murdering four-year-old daughter Amelia Brooke Harris
‘Strictly Come Dancing’ 2018 line-up: Lauren Steadman announced
British air hostess credits yoga and singing for surviving in the sea after ‘falling off cruise liner’
Alex Neill, from Which?, said nuisance calls can be a ‘daily torment’ for some people.
She said: ‘The Government must seize this opportunity and act swiftly to deliver on its promise to stamp out dodgy practices and make sure that those responsible for making nuisance calls can no longer evade justice and skip fines.’
Earlier this month telemarketing firm AMS Marketing Ltd was fined £100,000 after making more than 75,000 nuisance calls about compensation for road traffic accidents.
A DCMS spokesman said: ‘For too long a minority of company directors have escaped justice by liquidating their firms and opening up again under a different name, which is why we’re taking action to hold rogue bosses to account and put an end to these unwanted calls.’
It said it will publish its response to the consultation shortly.
Which? said Populus surveyed 2,093 UK residents in August.