What matters more at the earliest of early stages? Team? Or Idea?

I have attended over 45 hackathons in the last year. Some as a mentor, some as a participant and most as a judge. There are many biases I have, including a “developer bias” – I prefer developer heavy teams to MBA teams or sales teams etc. I know some of these biases and many I am not aware of.

So, it is always fascinating when I get a chance to meet the individuals first before they form teams and then see them selling their ideas to get quality hackers, hustlers (presenters, more than sales guys are needed for a hackathon) and hipsters (designers) on board.

I have been doing an experiment to help me understand how to evaluate teams and ideasbetter.

The experiment is fairly simple. On a white sheet of paper, I put the names of 5 people I think are the likely to win at the beginning of the hackathon. This may be based on just a 2-3 minute conversation with them. I dont know what goes in my mind, but I want to document it. I think I judge how they introduce themselves, what the first few words are, the background and education, etc. I think I have the same judgment biases as most investors and entrepreneurs have, so it is very likely that the top 10 on my list will be the same as any of the other folks who are judging or mentoring. I have shared these notes and the top 10 list with other mentors and asked them to share theirs as well.

Then when the ideas are being pitched I write down the top 5 ideas. I am sure many biases are being played in my mind again. I generally dislike the education and ideas aimed atsolving problems that college students have, for example looking for mentors or finding internships. I am very partial to solutions aimed at developers or marketing audiences.

Usually at the end of this process I have my top 5 or 6. The reason is that I have found that in 4 out of 10 cases the “good person” on the list does not choose a “good idea” or the “good idea” has some “average teams”.

I have tried to then map my initial “picks” against those that win the hackathon eventually. In the 11 hackathons that I have consistently done this over the last 2 months, there has been only 1 case when a team from the “other list” has won the hackathon. In 6 of the remaining 10, my first pick has gone on to win and in the remaining 4, my first pick has made it either to 2nd or 3rd place.

I thought I was doing pretty good. Over the last 5 hacakthons I have let other judges and mentors in on this experiment and asked them to rate before and after so they can get their before and after picks.

Turns out most of the judges were in the same ballpark. So, my entire basis for being able to judge good teams was as good or as bad as other judges. There is no proprietary information to leverage since the meetings and information exchanges are so short.

The second part of the question is what mattered more – a good team or a good idea – this is only to win the hackathon.

Turns out good ideas win more hackathons than good teams in 5 of my 11 cases. I have insufficient information to find out how many of the good ideas will eventually turn out to be good startups, given how nascent some of the ideas and teams were.

What I also think is that even if I took someone reasonably tech savvy regardless of experience, they would have a similar “wining picks” ratio.

This also calls into question the “experience” judges and mentors have gathered over the years and the exposure they have to all other competing ideas and companies. Turns out all that experience is possibly useful at certain other things, but at the earliest of early stages, it does not require a rocket scientist to figure out who wins the hackathon, within a low margin of error.

What do you think? Or am I taking limited data and generalizing to a trend?