Inside Bill Clinton’s Pandemic Year: Bingeing Bridgerton , Dusting Off His Sax & Writing Thriller

Slaven Vlasic/Getty James Patterson and former President Bill Clinton

The first time Bill Clinton and James Patterson co-authored a book (2018's The President Is Missing), America's 42nd president told reporters he was an "old dog" learning the "new trick" of fiction-writing.

So when the best-selling duo got on the phone with PEOPLE this week to talk about their pandemic-year joint venture — the upcoming thriller The President's Daughter, to be released on June 7 — they, of course, chatted about the book, which follows ex-President Matthew Keating as he plots to save his kidnapped daughter from a terrorist. But the pair, both 74 (and so chummy by now that they finish each other's sentences), also revealed how they used their months in separate quarantine to try some other new things.

PEOPLE: Michelle Obama taught herself knitting, lots of people figured out sourdough starter. Any new hobby you took up in all those days stuck at home?

Patterson: I did a couple of Audible podcasts and that was really fun and different, because those are five or six hours, which if you think about it, is a long time to keep people involved. I hadn't done a podcast and there's tricks to them. For starters, you have to keep repeating people's names so people will remember who they're hearing. The first one was The Coldest Case.

Clinton: He's got a lot of side-hustles. I was writing a non-fiction book about my life after the White House. Hillary and I started using streaming more. We binge-watched Bridgerton all night long — started on Saturday afternoon and quit early Sunday morning. I started playing my saxophone a little more, which I hadn't done in years, because I was working too hard. I went through my whole phone list — talk about people I cared about that I hadn't heard from or contacted in years! — and I just called people out of the blue, just to see how they were doing. But I didn't pick up painting or anything else, and I haven't played golf in almost a year now. I don't know if I can remember how to do it!

RELATED: Bill Clinton & James Patterson Share Excerpt from Thriller, The President's Daughter

Diana Walker//The LIFE Images Collection via Getty The former president

PEOPLE: What did each of you learn, in writing your first book together, that made the collaboration this time easier or more seamless?

Patterson: The first one was pretty seamless, and this one was too. Right from the get go, I think we worked well together and it's continued. Sometimes, the second time around, people get bored of it but that didn't happen.

Clinton: Yep. We agreed on the idea, then we had an outline and then my first job was to answer questions related to the presidency and the Secret Service, and recheck the facts, and not rely on my memory too much. I had a block of stuff that was related to my life and experience I had to fill in, and he would do an early draft and —

Patterson: — we went back and forth. I'd do a version, he'd do a version, which was kind of fun. What separates both of the books from anything that I've written or anything that I read is just the authenticity about it. People are not great at creating fiction of presidential characters. They have trouble getting it right, believable and real. The wonderful thing about working with President Clinton is that it's real.

PEOPLE: Tell us a specific detail that President Clinton either corrected or inserted.

Patterson: Not just details, a lot of stuff, but president, the thing in the embassy —

Clinton: There's a Chinese guy in the mix here, who hates the president and hates America, because his father was one of the three Chinese citizens killed (in the U.S. bombing of a Chinese Embassy). That's the sort of thing that I wanted to make sure we got right — that people understood it could actually happen.

Patterson: Matthew Keating, he's a retired U.S. president, so the president would lay in what the powers and frustrations of retired U.S. president are. Pretty much everything in the book, the president will fill in, 'Okay, here's if this happened in real life, here's what the Secret Service would do, what the CIA would do' —

PEOPLE: In the book, Melanie "Mel" Keating, when she was first daughter, was the subject of a Saturday Night Live skit mocking her thick glasses. So I caught a whiff of (President Clinton's daughter) Chelsea in Mel's character.

Patterson: We tried not to do that. There's very little. A little bit, but not much. Mel is probably the hero of the book. She is not a victim.

Courtesy of Knopf and Little, Brown.

PEOPLE: President Keating, at one point, talks about his post-White House anxiety dream about the military aide who carries the nuclear codes. Did you have post-White House anxiety dreams?

Clinton: No.

Patterson: He has them now. (Laughter.)

Clinton: The only anxiety I felt was, for about three weeks, I was anxious because they didn't play a song when I walked in the room anymore.

Patterson: That should be a relief, actually.

Clinton: I promised myself I wouldn't waste a day of my life wishing I could still do something I couldn't do anymore. I like being able to walk the streets now and go up to people and shake hands with them and listen to them. I've liked it.

PEOPLE: You mentioned missing golf.

Patterson: We played golf a couple times.

Clinton: Well, Jim gave me a putter which says, "Guaranteed two putt" on it. I'm going to take it out. He's a better golfer than I am, so I need to guarantee two putts. I'm going to carry it every time we play together.

Patterson: It was for Christmas, and he gave me —

Clinton: Socialist Monopoly. Monopoly for Socialists. This was our Christmas presents to each other this year.

RELATED VIDEO: Bill Clinton and James Patterson Speak About New Novel

PEOPLE: Are you already talking about your third book together or are you thinking you might cheat on him and maybe do a romance novel with Stacey Abrams next?

Patterson: Oh, he better not! We'll get to that (third book).

Clinton: I hope so. I'm ready. I'll tell you what, if he left me for Stacey Abrams, I wouldn't blame him. She has a luminous mind. She is so special.