The 84-year-old did add, however, that she was indebted to the roles of Violet Crawley and Professor Minerva McGonagall.
"I am deeply grateful for the work in Potter and indeed Downton, but it wasn’t what you’d call satisfying," she told ES Magazine. "I didn't really feel I was acting in those things."
Smith played Violet Crawley, the Dowager Countess of Grantham for the entirety of Downton Abbey’s six-season run and resumed the role for the 2019 feature film.
Meanwhile, she starred as Hogwart's transfiguration teacher for seven of the Harry Potter films from 2001 until 2011.
The actress told the publication she wished to "get back to the stage" as theater is her "favorite medium."
Smith returned to theater work this year for the first time since 2007 as Joseph Goebbels's secretary in Christopher Hampton's one-woman play, A German Life.
Speaking about learning her lines, Smith said: "It was actually easier to learn than Downton Abbey, because it wasn’t fragmented. I wasn’t just ordering tea or something."
As for another Downton Abbey film, producer Gareth Neame has previously divulged that a sequel is in the works.
He said, “We're having those conversations. We're working on what the story is, and when we might be able to make it. But it's the same as the first time around. We have to try to get everyone back together again. And that was very challenging.”
Read more from Yahoo Entertainment:
Want daily pop culture news delivered to your inbox? Sign up here for Yahoo Entertainment & Lifestyle’s newsletter.