One State Department official said that Mr Tillerson was caught unaware by the announcement, according to CNN, with it clear that the US's top diplomat wanted to stay in his job.
The tweet that notified Mr Tillerson that he lost his job read: "Mike Pompeo, Director of the CIA, will become our new Secretary of State. He will do a fantastic job! Thank you to Rex Tillerson for his service! Gina Haspel will become the new Director of the CIA, and the first woman so chosen. Congratulations to all!"
In an official statement, Steve Goldstein, undersecretary for public diplomacy at the State Department, said that Mr Tillerson "had every intention of staying... The secretary did not speak to the President and is unaware of the reason" for his firing. Mr Tilleron will be replaced by CIA Director Mike Pompeo.
Mr Tillerson's departure caps months of friction between the Republican president and the 65-year-old former Exxon Mobil chief executive. The tensions peaked near the end of last year amid reports Mr Tillerson had called Trump a “moron” and considered resigning. Mr Tillerson had repeatedly denied those reports.
Showing the apparent lack of coordination between the White House and the State Department Mr Trump claimed to reporters that he and Mr Tillerson had “been talking about this for a long time.”
The changes come as Mr Trump considers whether to withdraw the US from the 2015 deal over Iran's nuclear programme that Mr Pompeo has called “disastrous.” Mr Tillerson had long pushed the president to remain in the agreement and had been pursuing a delicate strategy with European allies and others to try to improve or augment it to Mr Trump's liking.
"We got along, actually, quite well but we disagreed on things, Mr Trump said of Mr Tillerson. "When you look at the Iran deal; I think it's terrible, I guess he thought it was OK. I wanted to either break it or do something and he felt a little bit differently," Mr Trump added.
The president said he was nominating the CIA's deputy director, Gina Haspel, to take over for Mr Pompeo at the intelligence agency. If confirmed, Ms Haspel would be the CIA's first female director
Mr Pompeo, a former Republican congressman from Kansas, has already been confirmed by the Senate for his current role at the CIA, making it extremely likely that he will be confirmed for the State Department role. Mr Trump tweeted, “He will do a fantastic job!”
On Monday, Mr Tillerson blamed Russia for the poisonings of former Russian spy Sergei Skripal and his daughter on UK soil despite denials from the Kremlin. Earlier, White House press secretary Sarah Sanders had refrained from saying Moscow was responsible.
A senior White House official said Mr Trump had asked Mr Tillerson to step down on Friday but did not want to make it public while he was on a trip to Africa, hence making it public today. Mr Trump's Twitter announcement came only a few hours after Tillerson landed in Washington after the trip, which had been cut short.
But, officials in Mr Trump's administration finding out they lost their job through untraditional means is anything but unusual. Last year, the President fired his former FBI director, James Comey, but the former intelligence chief was not properly notified before news outlets could confirm the decision.
Mr Comey ended up finding out about his firing while speaking to FBI agents in California, because CNN happened to be in the background with a chyron announcing his ouster.