Simon Cowell has known wild success. From his run on "American Idol" when that show ruled the ratings to the global phenomenons of Susan Boyle and One Direction, the music man with the Midas touch is used to coming out on top.

But with the second, ratings-challenged season of "X Factor" in the books, the judge's panel once again up in the air and a newly crowned victor who's also the oldest winner in major U.S. singing show history (and whose commercial prospects are unknown), the question becomes, "Why hasn't Cowell been able to make 'Factor' click with American audiences?"

"One problem for Cowell is that 'Idol' launched 10 years ago and it's a different environment now," said Michael Slezak, senior editor at TVLine.com. "What worked 10 years ago might not work now."

Slezak believes that Cowell and the "Factor" team have fundamentally misjudged the audience for these types of shows, which he said have become more sophisticated and know that there are lots of things going on behind the curtain that we don't see that have a big effect on the show. "We know the 'contestant's choice' is not really the contestant's choice," he said. "I don't think he understands that and likes to think he's somehow the Wizard of Oz and we're not looking behind the curtain."

The decision to hire Britney Spears and Demi Lovato to lure eyeballs also seems to have fizzled. With uncertainty about whether Spears will be back and a hard choice facing producers who paid her a reported $15 million in an effort to juice ratings, as well as the departure of mentor [article id="1698925"]L.A. Reid,[/article] 
the judging panel appears likely to get blown up again for the third time in as many seasons.

That has been the M.O. of the original British "Factor," where change has kept the show fresh. But so far Cowell has struggled to find the right mix of personalities for the U.S. version.

"Britney had the hint of stunt casting and ultimately putting a big name on these panels does more for the names than the show," said Hollywood Reporter music editor Shirley Halperin. "For Britney it keeps her out there for a year without touring or making a video ... but without a meltdown or her being really awesome, was it worth the [reported] $15 million? Probably not."

To be sure, the reality singing show playing field is crowded these days, with "American Idol" still chugging along in the winter and "The Voice" grabbing headlines, and ratings, twice a year with a formula that audiences seem to think is fresher and more exciting. Halperin said ratings are down around 20 percent across the format, so the "Factor" struggles are not unique.

But add in the schedule shuffling that took place because of the baseball playoffs on Fox and other jumps around the schedule due to pre-empting (not to mention cheeky counter-programming from "The Voice") and it was sometimes hard to know when "Factor" was on this year.

Even Cowell [article id="1699087"]admitted this week[/article]
 that there's just too many shows out there, saying, "We were expecting there was going to be a problem with the ratings this year, not just with us but with everything, because it's overkill, there are way too many talent shows on," offering up some choice words for his rivals at "The Voice."

He said he's not concerned about numbers-- the show has been renewed for a third season-- but rather the "Factor" legacy. So far, though, season one winner Melanie Amaro has floated a handful of singles that have not caught on, and not yet finished work on her debut, and a trio of top finishers from last year have failed to break through in their respective genres.

And what about this year's top finishers? From Halperin's perspective, they all seemed like B-team talent at best. "Something was missing in all of them," she said of winner [article id="1699306"]Tate Stevens[/article]
 ("capable, but a wet rag"), Carly Rose Sonenclar ("a Broadway girl, which is her calling") and Fifth Harmony ("just okay, but don't have the personalities to pull off a Spice Girls or chops to be a Destiny's Child").

On Thursday night, the act with the most commercial potential, Fifth Harmony, was sent home first. Then Sonenclar, whose extensive Broadway background was omitted from her backstory by producers, was sent packing, leaving 37-year-old Stevens standing.

In the annals of modern popular music history, country or otherwise, the record of launching a multi-platinum career at that age is, well, thin, if not non-existent. In fact, one of the only recent examples is "Britain's Got Talent" star Boyle, who was 47 when she rose to fame, mostly because of the YouTube factor of her mismatched voice/image.

"One thing that could save 'X Factor' is if it produced a legitimate pop star," said Slezak, who saw Harmony as one of the few acts with a shot at doing that. "If the winner of this dubious second season produces a great single very soon and has a viable career and you hear them on radio it could re-energize the show ... Stevens could sell some albums, but I don't see him being a cultural phenomenon."

Halperin also wondered if after a decade of searches by "Idol" whether the talent pool has been drained a bit and if "Factor" is having to go with second- or third-string singers, with the A-listers saving their shot for a spot on "Idol."

What do our experts suggest for season three?

"They need to hit control-alt-delete and reboot," said Slezak. "Rethink every aspect of the show. You can't be a cut-rate 'Idol' or 'Voice' and be successful. The whole approach should be to create the next big pop star, in which case, let us see the machinery that goes into that, because that's a story that's never been told on one of these shows."

Halperin agreed, saying there needs to be less staged and more natural, spontaneous interactions between the mentors and contestants. "And the auditions rounds are too long," she said. "While the live shows are too short. They need to rethink how they're hooking people into the show."