Simon Cowell Knew From 'Day One' Britney Spears 'X Factor' Panel Didn't Work

Cowell addresses season two of his Fox show while dancing around rumors he's expecting a child with a female friend.

Simon Cowell didn't hold back on Thursday when asked how he felt about the second season of the "X Factor," which featured (now-departed) newcomer Britney Spears in the mentor's seat.

While Cowell didn't name names, he did speak his mind with comments that could be interpreted to be about Spears, considering the pop star left the singing show after just one season and weak reviews.

"It's like having a dinner party. You invite people for dinner and sometimes it's a fun night, and other times it's not as fun as you hoped it would be," Cowell said at the Television Critics Association panel.

"This [season] is a fun dinner party," he continued before adding that he knew the season-two panel — made up of Spears, Cowell, Demi Lovato and L.A. Reid — wasn't gelling from "day one." "I felt we were doing what everybody else was doing and I had this impatience to get on to [upcoming] season three so we could do what we wanted to do," he explained.

Both Spears and L.A. Reid left the show at the end of season two, paving the way for Paulina Rubio and Kelly Rowland to join Cowell and returning mentor Lovato on the panel. Cowell said the new group feels "like family." "You have to be competitive, and you have to make changes," he said. "I work hard. It makes it fun. I'd love to be #1."

Season three will be split up into boys, girls, over-25s and groups; the prize has been shaved down from $5 million to $1 million, plus a Sony Recording contract.

In other singing competition show news, Fox also confirmed that Keith Urban will return to "American Idol" next season. Ex-"Idol" judge Jennifer Lopez, as well as "AI" alums Jennifer Hudson and Adam Lambert are also being considered for the judges' table.

Cowell, though, shot down any prospect of him ever returning to the show he helped make a hit Stateside. "I just can't do that," he said. "No. Somebody well-known once said to me, 'Know when to leave the fair.' "

While Cowell tried to keep the focus on "X Factor," he briefly responded to a question about tabloid reports that he's expecting his first child with the wife of a friend.

"I haven't read the newspapers," he laughed. Later he commented, "There are a lot of things I will eventually clear up when the time is right, but I really have to be sensitive because there's a lot of people's feelings involved here."