Todd Chrisley Says Daughter Chloe's Biological Mother Has 'No Rights' to Her amid His Conviction

Todd Chrisley is making his feelings known about daughter Chloe Chrisley's future.

On the latest episode of his Chrisley Confessions podcast, the reality star, 53, opened up about the news that Chloe's biological mother is planning to regain custody of the 10-year-old following his recent conviction.

Todd and wife Julie Chrisley — who were convicted of conspiracy to commit bank fraud, conspiracy to defraud the United States and tax fraud — adopted their granddaughter Chloe, the child of Todd's estranged son Kyle and Angela Johnson, in 2016.

"It's ironic that her biological mother would come forward now, when she has not been in her life since 2015. Angela Johnson surrendered her parental rights on March of 2017, and we never heard from her again," Todd said. "Chloe was legally adopted by myself and by Julie, and she is legally our child. We will be the ones who decide, if and when the time comes, as to who will be her caregiver."

Paul Archuleta/Getty; Dennis Leupold/USA Network/NBCU Photo Bank/Getty

He added: "We're not going to discuss who that's going to be, because we don't owe that to the world and that's our private matters. At the end of the day, Chloe's family is who Chloe has been raised with."

RELATED: Todd and Julie Chrisley 'Saddened' by 'Misleading Narrative' Regarding 10-Year-Old Chloe's Custody

Calling Chloe "a very bright, intelligent, charismatic, funny, beautiful child, who has a heart of compassion," Todd explained on his podcast that "her life is going to be as unaltered as it possibly can be, based on what Julie and I have the ability to control."

Todd Chrisley/Instagram Todd Chrisley and granddaughter Chloe

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Turning his attention back to Johnson, Todd said that "she literally surrendered her rights to her child. She did this in front of a judge. The judge explained to her, 'You understand what you're doing? You understand that you're surrendering the rights to your child?' And then explained that she had x amount of days to come back in and rescind that."

"[The judge] asked her in the hearing that day, 'Has anyone promised you anything financial? Has anyone threatened you?' The answer to all of that is no, and that's in writing. It's in a court document. We're not posting court documents because they are supposed to be private," Todd continued. "... Just so that the record is very, very clear, Angela Johnson has no rights to Chloe. Her rights were terminated. [Chloe] has been legally adopted."

PEOPLE reached out to Johnson for comment.

On his podcast, Todd also said that he and Julie, 49, attempted to have a relationship with Angela, and even provided her with money, a home and more to help her out amid tough times.

USA Network/NBCU Photo Bank via Getty

"We tried to do the right thing with Angela. We provided her housing for her to get on her feet at some point when she was homeless. I did not want to have to say to Chloe when she was 18 years old that I allowed her mother to be in the streets. I did not owe this to her mother. I did not owe this to go out here and provide housing for her," he said. "All the woman had to do was pay the $258 a month in an HOA fee for the condo we bought for her to live in until Chloe was 18 years old. At that point, the condo would've reverted back to Chloe, and Chloe could've done whatever she wanted to with it."

"She could've allowed Angela to live there, or she could've sold it and done what she wanted to with the money. The woman would not even pay the $258 a month. ... I've given this woman money to go to school for nursing, only to find out that she never completed any of it," Todd added. "She was just using us for the money."

Want to get the biggest stories from PEOPLE every weekday? Subscribe to our new podcast, PEOPLE Every Day, to get the essential celebrity, entertainment and human interest news stories Monday through Friday.

Tommy Garcia/USA Network/NBCU Photo Bank

Todd also claimed that Johnson wasn't around during the time period in 2016 when she had joint custody of Chloe, which made him and his wife move forward with the legal adoption of the now preteen.

"When she didn't have a car in the initial beginning, when we got joint custody of Chloe with Angela, Angela did not have a car that would get her back and forth. We bought her a car, a Toyota Prius, and gave it to her free and clear for her to be able to meet Julie every other week in Commerce, which was halfway — Commerce, Georgia," he explained. "She wouldn't show up half the time. She would never even call and tell us that she wasn't showing up."

"Julie would drive that hour and a half there, sit and wait in the parking lot for 45 minutes. She wouldn't answer the phone. We didn't know anything," Todd continued. "And then I would tell Julie, 'Bring her back home.' That's what prompted us to move forward with the adoption."

USA Network/NBCU Photo Bank via Getty

Todd then said that Johnson "terminated her rights voluntarily" and "she wasn't threatened, she wasn't coerced," adding: "What she needs to do is spend more time trying to find a better wig and less time trying to go out here and get more time on the camera."

Addressing Johnson directly, Todd told her and podcast listeners that he has documentation of their communication from over the years and can release it should he need to. "I know you listen to this, Angela. What you need to do is understand that I have retained every text message, and so has Julie, with any communication that we've ever had with you. We have not released those, but if we need to we will," he said.

"That's just drawing our line in the sand. You have had no relationship with Chloe. She does not have any relationship with you. You've never reached out to talk to her, you've never sent her a Christmas gift or a birthday gift, you've never done anything for her," Todd added.

RELATED VIDEO: Todd and Julie Chrisley Are Sentenced in Bank Fraud and Tax Evasion Case

Earlier this month, Johnson spoke with TMZ about her plans to "go back to court and get Chloe back home."

In a statement exclusively shared with PEOPLE, Todd and Julie said they were "saddened by the unfortunate and misleading narrative currently being portrayed in the media regarding Chloe."

Jessica Doyle, the family law attorney who Todd and Chrisley used during their adoption process for Chloe, exclusively told PEOPLE that Johnson's public statements are "simply not legally correct or even valid."

"I handled the formal adoption of Chloe by Todd and Julie Chrisley. Angela Johnson's parental rights were terminated when she voluntarily surrendered her parental rights to Todd and Julie Chrisley on March 24, 2017," said Doyle. "She has had no contact with the minor child since 2015."

"Todd and Julie Chrisley don't just have custody of Chloe. They are her legal parents through adoption," Doyle added. "I have reviewed the public statements made by Angela Johnson. They are simply not legally correct or even valid."