Talley, who died on Tuesday at age 73 after reportedly battling an unknown illness, was a trailblazing fashion journalist and editor, having worked with Wintour, 72, for years at the magazine.
"We didn't have to speak," he told PEOPLE in May 2020 of his close bond with the legendary fashion editor, while promoting his memoir, The Chiffon Trenches, in which he wrote honestly and candidly about the ups and downs of their friendship. (Wintour has been editor-in-chief of Vogue since 1988 and earlier this year took on two new job titles: worldwide chief content officer and global editorial director of Vogue.)
Continued Talley: "I knew what she was thinking, without words. She doesn't say many words."
Talley first joined Vogue in 1983 as the magazine's fashion news director. He quickly rose to creative director and Wintour's right-hand and can be seen in documentaries about the magazine, including The September Issue and The Gospel According to André.
"When I first went to Vogue, I met her, I said hello politely, I took the subway home to 14th St. and Vogue was at 44th. I took two stops on the express train. By the time I got home, under my door on engraved stationary from Anna Wintour was a note: Welcome to Vogue," Talley told PEOPLE.
"Ms. Wintour and I made waves. She was always my biggest supporter. She was very inspiring, and she is inspiring," he said. "Going to work every day was amazing. Everything about it was amazing."
However, the pair's decades-long friendship became strained after a red-carpet shakeup rocked Talley's usual Met Gala duties, when a YouTube personality was named host of the 2018 event, in lieu of Talley.
Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.
While their relationship hit "an iceberg," as Talley told Gayle King on CBS This Morning, Talley insisted that his memoir was, in a sense, a "love letter" to Wintour.
"And so this is a painful thing for me, but it is a love letter about the joys as well as the lows of my life. And the joys of my life have been with Anna Wintour," he explained. "But this book will help unpeel the onion about her. This is about is a Black man's experience in a very insulated world."
RELATED VIDEO: We Finally Know What Anna Wintour Thought of Meghan Markle's Wedding Dress
"I owe to her the pioneering role that I had of a creative director of Vogue," he continued. "I was the first Black man to ever be named such. I owe that to Anna Wintour. I owe her much. And I think, in turn, I think she owes me."
At the time, a source close to Wintour told PEOPLE exclusively: "Anna considered André a friend for over 30 years and naturally was saddened by the way he chose to portray many aspects of their friendship, but he is of course entitled to tell it as he remembers it. She wishes him the best."
"I love her," Talley said. "People see my book as a vengeful, bitchy tell-all. It is not. My book is in many ways a love letter to Anna Wintour."