Timothy Warren was driving his truck through a Portland, Oregon, neighbourhood when Joseph Magnuson began to shout at him to tell him that he was going too fast.
Mr Warren stopped his truck and tried to explain to Mr Magnuson that he was exhausted and just wanted to finish his work.
But Mr Magunson did not stop shouting and began hurling racist insults at him.
Mr Warren, who is black, stepped out of the truck and both men shouted at one another.
Mr Magnuson went to punch Mr Warren, who then punched back and hit Mr Magnuson's left eye.
The 55-year-old man then fell to the ground and briefly lost consciousness. He died later that evening.
Prosecutors have decided not to charge Mr Warren with a crime.
He did not intend to kill Mr Magnuson and acted in self-defence during the incident on 26 September, the Multnomah County District Attorney's Office said in a memo.
Mr Magnuson's death from the fall was precipitated by “extremely poor health,” a medical examiner concluded and the punch itself was not fatal, Senior Deputy District Attorney Adam Gibbs wrote.
Mr Warren was within his legal right to challenge Mr Magnuson's “racist vitriol,” Mr Gibbs noted. He said Warren's decision to confront Mr Magnuson - rather than ignore him - was not legally significant.
Oregon law does not carry a “duty to retreat” provision that would have required Warren to reasonably remove himself from danger, Mr Gibbs wrote.
Prosecutors and police pieced together events using accounts from six eyewitnesses, three of whom saw the entire incident. They all maintained that Mr Warren had driven at an acceptable speed and that Mr Magnuson started the incident, and then escalated it.
“Mr Magnuson's actions, as reported by all three witnesses and Mr Warren, gave rise to a reasonable belief on Mr Warren's part that a limited use of force was necessary to prevent injury to himself,” Mr Gibbs wrote.
Mr Magnuson was living in a van near the park where the deadly encounter occurred, district attorney spokesman Brent Weisberg told The Washington Post.
Mr Warren, whose age wasn't released, could not be reached for comment.
The announcement by the prosecutor's office was the first public mention of Mr Magnuson's death and the ensuing investigation, the Oregonian newspaper reported. It was unclear why prosecutors had only recently released information about the incident.
It is also unclear whether Mr Warren still works for FedEx. The company declined to comment on his status.
The company, which said it cooperated fully during the investigation, extends “condolences to those affected”, FedEx Ground spokeswoman Nikki Mendicino said in a statement.