Paul Shaffer Reveals How David Letterman Knew He Was the Skeleton on Masked Singer

The Skeleton shook his bones during another performance on Wednesday’s episode of The Masked Singer, but his rendition of Jet’s “Are You Gonna Be My Girl” landed him in the graveyard of eliminated contestants.

After the studio audience voted the Skeleton as their least favorite of the six performers, he was unmasked and revealed to be David Letterman’s former bandleader and Blues Brothers co-creator Paul Shaffer.

“I’m a whole lot lighter,” Shaffer, 69, jokes to PEOPLE of being out of his costume. “The skeleton head is heavy. It had its own agenda sometimes. I would want to go one way, it would go the other.”

While judge Jenny McCarthy correctly guessed Shaffer’s identity, her fellow judge Ken Jeong insisted Shaffer’s close friend, comedian Martin Short, was underneath the Skeleton disguise. “I was surprised that she got it and she was so sure of it,” Shaffer says of McCarthy’s accurate prediction.

Speaking with PEOPLE after his elimination, Shaffer shares the costume he almost wore instead of the Skeleton, how it felt being in the spotlight, and what Letterman thought of his appearance on the singing competition.

RELATED: The Masked Singer: Panda Is Unmasked and Revealed to Be Laila Ali

You talked in your clue packages on the show about feeling like you were playing second fiddle throughout your career. Is that why you wanted to do The Masked Singer?
I saw the show in the first season and it just blew me away because it’s just so crazy and the costumes are outlandish. When I got the call — “do you want to do it?” — I didn’t have to think twice. What a great opportunity to be a part of this thing that was absolutely mystifying and blowing me away the first season.

You have a musical background, so did you have to do much prep for the show?
I would never think of warming up normally like a real vocalist does, but on this show they have vocal coaches, so I would absolutely take advantage of that and work with the vocal coach and try to open up, warm up my throat in the classical way. Otherwise, I just had to think of songs that I knew the lyrics to. I learned later there was a teleprompter somewhere with the lyrics on it, but I couldn’t even see two feet in front of me. Who could see a prompter? So I was just trying to make sure I knew the lyrics.

Was it hard to move in the costume?
Just a little bit. It wasn’t too unwieldy compared to some of the other ones I saw. It’s hard enough to sing anytime but they make it even harder and they make it hard for everybody. That’s part of the fun. For the actual singing, the mic that you’re using, that’s a real microphone. That’s strange because you’re putting it up not to your lips but to the lips of a big plastic head. You don’t know if it’s close to your mouth or not. That’s another interesting part of it.

RELATED VIDEO: ‘Masked Singer’ Host Nick Cannon Admits He Would DM Lizzo ‘For Sure’

What made you pick the Skeleton costume?
They showed me a number of options. One of them was the Ice Cream Cone. Somebody else became that. But when I saw the Skeleton, to me he reminded me of Fred Astaire, so suave and debonair a skeleton was he. From an old ’40s movie. I thought, let me try and be as cool as Fred Astaire as the Skeleton. So that’s why I picked it.

Was it disappointing to be unmasked before Halloween?
The merchandising, I could’ve put out those suits! Paul Shaffer Skeleton Halloween suits, I was all ready to go.

Did Letterman know you were doing the show?
Letterman did not know but called me. He does know about things like nondisclosure agreements. He called me up and [said], “Paul, you don’t have to say anything, but I know it’s you. I can tell by your walk. I recognized it right away.” I couldn’t confirm or deny, I just had to be cagey about it. But this guy saw me every day for 33 years. He wasn’t going to miss me. The other funny one was my friend Martin Short, he’s one of my best friends coincidentally from Canada and here’s Ken Jeong. He went, “I have studied Martin Short. I know that’s him. I’m positive!” Marty did call me and he said, “Aren’t you the Skeleton? I think you’re the Skeleton!” And I said, “I think you’re the Skeleton!” That was all I could say.

RELATED: Dr. Drew Pinsky Reveals He Suffered a Vocal Cord Injury While Prepping for The Masked Singer

How did it feel to be the one in the spotlight after all these years?
The truth is I get to do it all the time. I’m always appearing in a club, doing a symphony show, I’ve got a show on AXS TV, Paul Shaffer Plus One, where I play and sing with guests. This though, I was in a singing competition. I sure never thought I would be there. And you really have to have confidence. It sure is true when you’re second banana, there’s a lot of difference. I remember when I got to host the show for Dave a couple times — I thought my job was hard until I did his. When your face is on a camera in a closeup, that trumps everything else. That’s some pressure for sure, but very enjoyable and a lot of fun.

The Masked Singer airs Tuesdays (8 p.m. ET) on Fox.