This Number Crunching Life

Congratulations to the Machine March Madness Winner

2013-04-09T12:13:00.000-07:00

Well, after another exciting March Madness tournament, Louisville emerged as the winner of March Madness, and Ryan Boesch emerged as the winner of Machine March Madness, with his algorithm beating out the field of 22 other machine competitors and all the human baselines. Congratulations, Ryan!

I asked him a few questions, which he answers below:

1. What inspired you to compete in the Machine March Madness competition?

Last year I finished a class on Convex Optimization during the winter quarter and was planning to take a Machine Learning class in the spring quarter. I was looking for a project to apply what I had learned. I saw this competition and submitted a last minute bracket.

2. What do you attribute your win to? What is your model best at?

The win was of course very lucky. Basketball games are random in nature so to find which model is actually the best it would require many years of tournaments. One tournament is not statistically significant.

There is nothing particularly special about my model. I used Danny's model, only I fit the parameters using convex optimization instead of batched gradient decent.

3. What do you think the most promising direction(s) towards improving your model would be?

Most Promising: My current model simply matches teams and sees which has the higher predicted score. It doesn't account for difficult of previously played games in the tournament. For example, say team 1 has a 51% chance to win the first round and also 51% chance to win the second round against team 2. If team 2 has a 95% chance of winning the first round then they are more likely to make it to round 3 even though they only have a 49% chance to beat team 1 in the second round. This is taken into account in Nate Silver's picks for example.

Second Most Promising: When in a pool with other competitors the goal is no longer to maximize your expected score, but instead to maximize your expected chance of winning. These two optimizations do not always result in the same picks. I may consider taking this into account in future years. I found this paper on Nate Silver's blog which analyzes this idea.

4. What advice would you give to future competitors?

Be wary of over fitting your model.

5. What would you change about the competition in future years?

We should try to get out and advertise for the competition earlier and to a broader audience to maximize participation.

Machine March Madness Final Four Outlook

2013-04-05T13:30:00.000-07:00

Once again, we have a guest post by Scott Turner, our local Machine March Madness competitor and analyst, who also runs http://netprophetblog.blogspot.com/. Thanks for another great post, Scott!

The Machine March Madness Contest finds itself in peculiar waters this year -- no doubt because the Tournament is in strange waters itself. There is only a single team left alive from the twelve #1, #2, and #3 seeds. Going into the Final Four, only four brackets have their champion prediction left alive -- and all four have Louisville. In fact, only one bracket has anyone left alive other than Louisville -- and that's "Tim J's Nets for Nets", who has Syracuse to the final game (but losing to Ohio State). But despite the craziness in the Tournament this year, the top four predictors in the Machine March Madness Contest are in the top 5% of all brackets.

If Louisville manages to win out, the champion will be "Ryan's Rank 1 Approximation" with 121 points. He will beat out all the human competitors as well.

If Louisville loses in the final game, the champion will be "Predict the Madness" (tied with "Danny's Dangerous Picks"), with "Ryan's Rank 1 Approximation" and my own Prediction Machine both a single point behind.

If Louisville somehow loses to Wichita State and Syracuse beats Michigan, then "Tim J's Nets for Nets" will vault all the way from fifteenth into a tie for first with "Predict the Madness". If Syracuse loses to Michigan, then "Predict the Madness" will win outright.

It has certainly been a crazy year for the Tournament!

Upset Analysis by Scott Turner

2013-03-25T15:07:00.002-07:00

This is a guest post by Scott Turner, who is a perennial Machine March Madness competitor, and who runs http://netprophetblog.blogspot.com/.

There are 22 entries in the Machine Madness contest this year, so analyzing them is a much bigger task than in past years. Nonetheless I dug through all the brackets and looked at all the first round upset predictions to see how well the machines did.

Correct Upset Predictions

Interestingly enough, every first round upset was picked by at least two of the predictors except for Harvard -- which no one picked -- and Florida Gulf Coast, which only "Larry's Upsetting Picks" predicted. The only consensus upset pick was Minnesota over UCLA, which was predicted by exactly half of the predictors. Iowa State also got broad support (40%) but none of the rest of the picks had more than 4 predictors in support. Here's the full table of the upsets that occurred and who predicted them (stretch your window!):

Entry	Sum	Minnesota	Iowa St	Oregon	Wichita St.	Mississippi	Temple	California	La Salle	Fla GC	Harvard
*Danny's Dangerous Picks	2	1	1
Andy's Astounding Bracket	2		1					1
Ask me about my T-Rex	3	1	1		1
Curtis Lehmann's Crazy Bracket	0
Dan Tran's Dazzling Bracket	3		1	1				1
Guess O'Bot 3000	0
K. V. Southwood's Fine Bracket	2	1	1
Larry's upsetting picks	4		1			1			1	1
LA's Machine Mad Pick	1						1
Leon's Super Legendary Bracket	1		1
Marginal Madness	1					1
Mark's LR bracket	4	1			1		1	1
MatrixFactorizer	3				1	1	1
natebrix's Neat Bracket	2	1							1
noodlebot	1	1
Predict the Madness	1	1
Ryan's Rank 1 Approximation	1	1
Scott Turner's Prediction Mach	2	1	1
ScottyJ's Grand Bracket	0
The Rosenthal Fit	2			1		1
TheSentinel	2	1		1
Tim J's Nets for Nets	4	1	1	1	1
Ave Correct:	1.9	50%	41%	18%	18%	18%	14%	14%	9%	5%	0%

My conclusion here is that UCLA-Minnesota and Notre Dame-Iowa State were probably mis-seeded.

UCLA-Minnesota is an interesting case in human psychology. Minnesota lost 11 of its last 16 games, finished 8th in its conference and lost in the first game of the conference tournament, while UCLA won 11 of its last 16, won the Pac-12 regular season conference title, and lost in the title game of the conference tournament. It's no wonder UCLA got a 6 seed and Minnesota an 11. But in fact, Minnesota was playing against much better competition through the conference games, and most of its losses came to ranked opponents and/or on the road. Machines understand the concept of a "good loss" much better than people.

The Notre Dame-Iowa State mis-seeding wasn't so egregious. This probably should have been an 8-9 matchup instead of a 7-10, in which case a win by Iowa State would have hardly been surprising.

All of the rest of the games were probably true upsets.

Incorrect Upset Predictions

Most of the predictors also made a number of incorrect upset predictions. Most predictors had one or two missed upsets, although six of the predictors made no missed upset predictions (primarily because they made mostly chalk predictions). Here's the full table:

Entry	SUM	Colo	St.Mary	Cincy	Bucknell	Okla	Missou	Belmont	SDak	Davidson	Valpo	Iona	Villa	Akron	Montana
*Danny's Dangerous Picks	2		1				1
Andy's Astounding Bracket	0
Ask me about my T-Rex	2		1	1
Curtis Lehmann's Crazy Bracket	1	1
Dan Tran's Dazzling Bracket	2	1	1
Guess O'Bot 3000	3	1			1	1
K. V. Southwood's Fine Bracket	4	1			1	1		1
Larry's upsetting picks	7	1						1	1	1	1	1			1
LA's Machine Mad Pick	0
Leon's Super Legendary Bracket	8	1	1	1	1			1		1			1	1
Marginal Madness	0
Mark's LR bracket	4	1	1			1	1
MatrixFactorizer	2	1		1
natebrix's Neat Bracket	2	1			1
noodlebot	0
Predict the Madness	1		1
Ryan's Rank 1 Approximation	1			1
Scott Turner's Prediction Mach	2		1			1
ScottyJ's Grand Bracket	0
The Rosenthal Fit	0
TheSentinel	2		1				1
Tim J's Nets for Nets	4	1		1	1				1
Average Missed:	2.1	45%	36%	23%	23%	18%	14%	14%	9%	9%	5%	5%	5%	5%	5%

As a general rule, the predictors that made the most correct upset picks also made the most incorrect upset picks. Notably, "Larry's Upsetting Picks" made the incredible call of the FGCU upset (and also called the second round upset) but also made seven incorrect upset picks.

There was almost a consensus (45%) on Colorado over Illinois. That's an interesting contrast with the Minnesota pick -- Illinois should have benefited in most of the predictors from a tough B1G conference schedule, but many of the predictors thought Illinois was still vulnerable. Illinois had a 16 point halftime lead in this game, but let it slip away and need some late game heroics to win, so this was certainly a reasonable prediction.

St. Mary's over Memphis was another popular pick. Memphis won by 2 when a last-second shot by St. Mary's missed, so this also seemed like a reasonable upset pick.

Upset Profits

An important question is whether any of the predictors profited from their upset predictions -- that is, whether the points they gained from correct upset predictions were more than the points they lost from missed upsets. In general, this is complex to calculate because we have to look at how the predictions affect the later rounds of the tournament. But it's easy enough to look at just the first round scoring. Here's the table:

Entry	Upsets	Missed	Overall
Andy's Astounding Bracket	2	0	2
The Rosenthal Fit	2	0	2
Ask me about my T-Rex	3	2	1
Dan Tran's Dazzling Bracket	3	2	1
LA's Machine Mad Pick	1	0	1
Marginal Madness	1	0	1
MatrixFactorizer	3	2	1
noodlebot	1	0	1
*Danny's Dangerous Picks	2	2	0
Mark's LR bracket	4	4	0
natebrix's Neat Bracket	2	2	0
Predict the Madness	1	1	0
Ryan's Rank 1 Approximation	1	1	0
Scott Turner's Prediction Mach	2	2	0
ScottyJ's Grand Bracket	0	0	0
TheSentinel	2	2	0
Tim J's Nets for Nets	4	4	0
Curtis Lehmann's Crazy Bracket	0	1	-1
K. V. Southwood's Fine Bracket	2	4	-2
Guess O'Bot 3000	0	3	-3
Larry's upsetting picks	4	7	-3
Leon's Super Legendary Bracket	1	8	-7

We see that a couple of the predictors ("The Rosenthal Fit" and "Andy's Astounding Bracket") came out two points positive, fifteen of the predictors gained one or zero points, and five of the predictors lost points. Interestingly, both "The Rosenthal Fit" and "Andy's Astounding Bracket" made only two upset predictions and got both of them right -- and there was no overlap in their predictions. Furthermore, neither of them predicted the "easiest" upset of Minnesota over UCLA.

Conclusions

None of the predictors performed very well at picking upsets, and there wasn't wide agreement on the upset picks. The consensus would have selected only the Minnesota-UCLA upset and been +1 in scoring, but no individual predictor did that. Most of the predictors did not hurt themselves with their upset picks (at least looking at only the first round), but none really saw significant benefit. Given the potentially large downside of missing upset predictions, in future contests it wouldn't be an unreasonable strategy to force your predictor to make all chalk selections in the first round.

Some Final Four Analysis...

2013-03-24T08:23:00.000-07:00

... over at Scott Turner's blog: http://netprophetblog.blogspot.co.uk/2013/03/machine-madness-some-final-four-analysis.html

Predicting March Madness by Jasper Snoek

2013-03-21T13:58:00.000-07:00

Now that March Madness is officially underway, and the deadline to submit new bracket predictions has passed, I'm ready to divulge the details of my super secret, possibly excessively advanced, march madness prediction model. For a few years now, there has been a special "elite" pool to predict march madness. The twist is that all the predictions have to be made by a computer algorithm - no humans allowed. This means we can't use seed information, predictions from experts or the POTUS's executive insight. Instead, we predict based only on data (my model uses only scores). This is the second year that I am entering an algorithm. My entry from last year, which won the pool and beat the vast majority of humans in the Yahoo challenge, is being used as a baseline. This means I have to submit something more sophisticated this year to stay on top.

The model:

The Simple Version:
A few years ago, the world of machine learning (a subfield of artificial intelligence that combines statistics, math and computer science to get computers to learn and infer from data) was rocked by the Netflix challenge. Netflix offered a prize of a million dollars to anyone who could beat their movie recommendation system by 10%. One of the most powerful and surprisingly simple algorithms to come out of that challenge was Probabilistic Matrix Factorization (PMF). The idea was that a movie rating was a simple product of a set of hidden or 'latent' factors pertaining to the movie and the user. Although the factors are not pre-defined, you could imagine that the model may learn one factor for a movie that corresponds to the amount of action and then a user would have a factor encoding how much they like action (and similarly for e.g. romance). We learn the model by adjusting these factors to maximize the probability that the user would give the ratings that we can see. To predict someone's rating for a given movie they haven't seen yet, you just multiply their factors by the movie factors.

Similarly to movie ratings we can create factors for basketball teams to predict game scores. Here the factors (again learned by the model) could correspond to offensive skill and defensive capabilities. This was the basis of my model for last year. There was a small twist in that I altered the way that the model was learned - to focus only on scores for which it predicted the wrong winner.

This year my model is significantly more complex but builds on the same principles. It has two levels of latent or hidden factors. The first encodes factors for each team - such as offensive skill, defensive skill, etc. The second layer combines team factors just like in standard PMF, but instead of mapping directly to the scores they map to a hidden representation that encodes the game. My reasoning is that the resulting score of a game is much more complex than a product of simple factors pertaining to each team. The idea is that the game representation now encodes things like: will the game be close or will it be a blowout - will it be high scoring or a defensive brawl? From the game representation I have a mapping to the difference between the home team score and the away team score. Now this is where things get a little complicated. Since there are relatively only a small number of games in this season (just over 5000) and this model is already fairly complex, rather than directly try to learn a function mapping from the game factors to the scores, I model a distribution over all possible mappings. The idea: given all (infinite) reasonable mappings from factors representing the game to scores, what is the most probable outcome? To do this I use a statistical model called a Gaussian process.

The factors encoding teams.

Now to learn the model:

I take all of the game scores from the past season. For each game, I tell the model which team is the home team, which is the away team, and then adjust the team factors and game factors in order to maximize the probability of the real score. In order to choose the number of factors at each step, I use a new automatic parameter tuning algorithm I personally helped develop called Bayesian optimization.

What do the factors look like?
Just to the right I have an example of the factors that are learned if I train the model using just two factors for the teams (for those of you in machine learning, these are the weights of the neural network) and I have plotted where each of the teams are in this factor space (along with their seeds). You can see that the model is putting the better teams in the lower left and the worse teams near the top right. It doesn't seem to fancy the odds of South Dakota... I'll explain later why I call the model "Turducken".

Below this I have a picture zoomed in on just the bottom left. You can see that the powerhouses are all encoded in this region. You can click on these images to zoom in. Now you can see that two factors already encode quite a bit about which teams are better. My model uses two hundred factors - so it is encoding something that is quite significantly more complex.

Zoomed in on the bottom left.

Below there is a picture of the factors learned to encode games. There is a dot for each game which is colored by relative score. So a 1 means that the home team wins by a lot and a two means that the home team loses by a lot ("a lot" here actually means about 50 points). So the model takes the team factors on the right and multiplies them to get to the game factors below. Then from the game factors it predicts by how much the home team will win or lose.

What is this Bayesian optimization?
One really exciting area of machine learning that has advanced a lot over the past year is related to how to build systems that work more automatically. To really eke out the best performance, you usually need an expert to sit and tweak a bunch of knobs, see what happens, and repeat many times. It's really time-consuming and nearly
impossible for a non-expert (and even difficult for experts). But there is work on automating this process, building a system to automatically tune the knobs and decipher the results. I am using Bayesian optimization that I left running overnight to automatically determine how many factors to use for teams and for games based on how well the model can predict the scores of 500 games that I pulled out of the set of data that the model learns from. The procedure decided to use 200 factors per team and just two per game.

In Machine Leaning Speak:
The devil is of course in the details. The model I am using is a buzz-word powerhouse. I call it a deep semi-parametric Bayesian probabilistic matrix factorization that is optimized using Bayesian optimization. My fellow machine learning PhD friend, George Dahl, calls it a "statistical Turducken". It uses a neural network trained with 'dropout' to perform a nonlinear probabilistic matrix factorization into a latent space that encodes games. A Gaussian process mapping is then used to map from games to the score difference. The input to the neural network is a binary encoding of which team is the home team (so the number of dimensions equals the

An example of the factors learned by the model to encode 'games'.

number of teams) and then similarly a binary encoding of which team is the away team. So the input to the model is a numTeams x 2 dimensional binary encoding with two bits on. This may seem wasteful, but note that now the weights to be learned by the neural network correspond exactly to latent factors pertaining to each team. The teams get different factors depending if they are home or away (as I personally have no college basketball expertise, I have no idea if this is a wise design choice). The neural network maps these factors into a hidden unit representation and then to a latent space. From the latent space I map using a Gaussian process with a squared exponential kernel to score difference.

The model is trained using backpropagation - from the marginal likelihood of the Gaussian process I backpropagate error through the kernel of the GP to the weights of the neural network. I use stochastic gradient descent on randomly chosen minibatches of 250 games at a time and a 50% dropout rate on the hidden units of the neural network. I used Bayesian optimization on a validation set of 500 games to determine the number of hidden units in the neural network (i.e. the number of factors in the PMF), the latent dimension of the input to the GP and the number of epochs to train the model for.

What did it predict?
You can check out the bracket that it predicted here: http://tournament.fantasysports.yahoo.com/t1/1597161
As of this writing, the model is 4/4 including a minor upset of Wichita over Pittsburgh.

You can take a look at our pool here: http://tournament.fantasysports.yahoo.com/t1/group/162045
Interestingly, even though it doesn't know anything about the seeds, it predicted the four number one seeds in the final four. According to the turducken, Indiana is going all the way. This is pretty remarkable - the algorithm is in close agreement with some of the top human basketball experts. That is already a validation that it is doing something reasonable. There are not too many
controversial predictions here, though it is predicting some upsets (e.g. Notre Dame over Ohio St.). It will be really exciting to see how it does as the next days play out!

The 2013 Machine March Madness Field

2013-03-21T13:48:00.000-07:00

Thanks everybody who entered this year's Machine March Madness competition. Based on the descriptions of the approaches, it's clear that a lot of hard work and ingenuity has gone into the contest. I'm excited to see how all the different approaches do.

Below, you can see the competitors's descriptions of their approaches. We'll also have some longer posts diving into more details coming up in the near future. If there are any in particular that you're itching to hear more about, leave a note in the comments.

If you have entered but not sent me a description of your approach yet, please do. I'll update this post as more descriptions come in.

Without further ado, here is your 2013 Machine March Madness field!

-----------------------------------------------------------------------------

Marginal Madness
Kevin Swersky
http://tournament.fantasysports.yahoo.com/t1/2909174

I'm using variational Bayesian matrix factorization with normal priors on the latent factors, and Gaussian-inverse Wishart hyperpriors on the hyperparameters of the priors. Inference is performed using mean-field (no direct optimization of any model parameters is done). The entries of the matrix are R(i,j) = P(team i beats team j) using the empirical counts over the 2012-2013 season. I found that the brackets produced using this were much more stable with respect to the number of factors than any other representation. I used 20 factors, the number of which was chosen based on squared error on 25% randomly held-out entries of R. For my predictions, I just took the mean vectors and ignored any uncertainty learned by the model. Ideally, I should have selected the number of factors, or assessed the stability of the model by using the variational lower bound, but I was lazy. To predict the final score, I used gradient-boosted regression trees from scikit-learn on the feature vectors produced by the factorization.

-----------------------------------------------------------------------------

Larry's Upsetting Picks
Laurent
http://tournament.fantasysports.yahoo.com/t1/1398519

I'm using a PMF-based model and I'm also modelling several other aspects such as teams' strength over time (both over a season and across seasons) as well as conferences' strength. These different aspects are combined linearly together to form a prediction.

I also tried using a team's winning percentage (both over the season and over the last few games) but that didn't lead to an improvement.

On a technical note, I also noticed that in PMF instead of using the raw score, using the difference in scores gives slightly increased (winner determination) accuracy.

-----------------------------------------------------------------------------

K. V. Southwood's Fine Bracket
K.V. Southwood
http://tournament.fantasysports.yahoo.com/t1/3003299

I created an ensemble model based on 3 individual models:

1) multiple linear regression model based on predicting the points margin

2) multiple linear regression model based on predicting offensive points scored

3) logistic regression model based on predicting win vs. loss

-----------------------------------------------------------------------------
Ryan's Rank 1 Approximation
Ryan B.
http://tournament.fantasysports.yahoo.com/t1/1636526

Brief description of approach (same as last year): For each season (e.g. 2006-2007) I have enumerated the teams and compiled the scores of the games into a matrix S. For example, if team 1 beat team 2 with a score of 82-72 then S12=82 and S21=72. Ideally, each team would play every other team at least once, but this is obviously not the case so the matrix S is sparse. Using the method proposed by George Dahl, I define vectors o and d which correspond to each teams offensive and defensive ability. The approximation to the matrix S is then just the outer product od' (for example (od')_12=o1d2=S12est). This is a simple rank one approximation for the matrix. If each team played each other at least once then the matrix S would be dense and the vectors o and d could be found by finding the SVD of S (see http://www.stanford.edu/~boyd/ee263/notes/low_rank_approx.pdf). Because this is not the case, we instead define a matrix P that represents which teams played that season. For example, P12=P21=1 if teams 1 and 2 played a game. Now the problem stated by George can be expressed compactedly as, "minimize ||P.*(o*d')-S||_F". Here, '.*' represents the Hadamard product and ||.||_F is the Frobenius norm. In this from, it is easy to see that, for constant vector o and variable vector d, this is a convex problem. Also, for constant vector d and variable vector o this is a convex problem. Therefore, by solving a series of convex problems, alternating the vector variable between o and d, the problem converges rapidly in about 5 to 10 steps (see "Nonnegative Matrix Factorizations" code here http://cvxr.com/cvx/examples/). From this point the problem is easily expanded to handle higher rank approximations.

-----------------------------------------------------------------------------

Scott Turner's Prediction Machine
Scott Turner
http://tournament.fantasysports.yahoo.com/t1/1760363

Linear regression on a number of statistics, including strength ratings to predict MOV (Margin of Victory). The basic model is used to predict game outcomes throughout the year, but there are some modifications for the Tournament. Additions this year include a new metric for analyzing possible upsets, an algorithm for forcing upset selections based upon the (predicted) score required to win the pool, and some modifications for neutral-court and tournament games. More details at http://netprophetblog.blogspot.com/.

-----------------------------------------------------------------------------
noodlebot
Joe
http://tournament.fantasysports.yahoo.com/t1/2298853

See my blog post and project page.
http://joenoodles.com/2013/02/ncaa-d1-basketball-db/
https://github.com/jnu/ncaa

-----------------------------------------------------------------------------
Danny's Dad (Human Baseline)
Danny's Dad.
http://tournament.fantasysports.yahoo.com/t1/2664431

Literally, Danny's Dad's picks.

-----------------------------------------------------------------------------
Obama's Bracket (Human Baseline)
Barack Obama
http://tournament.fantasysports.yahoo.com/t1/1673628

The President's picks.

-----------------------------------------------------------------------------
MatrixFactorizer
Jasper Snoek
http://tournament.fantasysports.yahoo.com/t1/1597161

Probabilistic matrix factorization augmented with Gaussian Processes and Bayesian optimization. More details will be forthcoming in a longer blog post (Update: here).

-----------------------------------------------------------------------------

LA's Machine Mad Pick
LeAnthony M.
http://tournament.fantasysports.yahoo.com/t1/1647581

I used 2011 final four stats data rather than last years. Including RPI, Off eff, turnovers, & def eff. A fitness function of the final standings NCAA tournament standings feed into an evolving genetic program giving me a final equation. I feed in this equations, this years team of 64 to compute the final standing of the 2013 tournament.

-----------------------------------------------------------------------------

Predict the Madness
Monte McNair
http://tournament.fantasysports.yahoo.com/t1/2002207

???

-----------------------------------------------------------------------------

TheSentinel
Chuck
http://tournament.fantasysports.yahoo.com/t1/2997354

Similar strategy as last year. Used Ken Pomeroy's Pythag ratings with the log5 calculation to determine probability of winning the game.

Used a Monte Carlo simulation at 65 iterations which provided a few interesting upsets, Oregon over Oklahoma St. (I believe they were miss seeded myself!).

-----------------------------------------------------------------------------

Danny's Dangerous Picks
Danny
http://tournament.fantasysports.yahoo.com/t1/1421921

Developed a variant on probabilistic matrix factorization, where the scores of a game are modeled as the output of a neural network that takes as input a learned latent vector for each team as well as the elementwise product of the latent vectors for the two teams. Latent vectors for each team are learned for each team for each season jointly with the neural net parameters, which are shared across all seasons from 2006-2007 through the present. I used 5D latent vectors and a one hidden layer neural net with 50 hidden units.

-----------------------------------------------------------------------------

Human Bracket
Lee
http://tournament.fantasysports.yahoo.com/t1/3297751

The Commissioner's human bracket.

-----------------------------------------------------------------------------

The Rosenthal Fit
Jeffrey Rosenthal
http://tournament.fantasysports.yahoo.com/t1/1666195

Details here: http://www.tsn.ca/story/?id=418503

-----------------------------------------------------------------------------

Last Year's Winner (Baseline)
Jasper Snoek
http://tournament.fantasysports.yahoo.com/t1/1644140

(The winning algorithm from last year, run on this year's data but otherwise unmodified. Entered as a baseline.) I modified Danny's starter code in two ways: First, I added an asymmetric component to the loss function, so the model is rewarded for getting the prediction correct even if the absolute predicted scores are wrong. Second, I changed the regularization so that latent vectors are penalized for deviating from the global average over latent vectors, rather than being penalized for being far from 0. This can be interpreted as imposing a basic hierarchical prior.

I then ran a search over model parameters (e.g., latent dimension, regularization strength, parameter that trades off the two parts of the loss function) to find the setting that did best on number of correct predictions made in the past 5 years's tournaments.

-----------------------------------------------------------------------------

Leon's Super Legendary Bracket
Leon
http://tournament.fantasysports.yahoo.com/t1/1712730

Defensive efficiency vs Offensive efficiency; tie-breakers favored defense over offense. Chose final score using season averages in wins/losses.

-----------------------------------------------------------------------------

Tim J's Nets for Nets
Tim J.
http://tournament.fantasysports.yahoo.com/t1/1546944

Based on full season statics for each team run a discriminant analysis for correlation with wins including seasons 2000-present.

Then I trained a neural network only on neutral location games, measuring both performance in mean squared error and actual past year bracket scores from 2007-2012, and predicting the bracket for this year.

-----------------------------------------------------------------------------

natebrix's Neat Bracket
Nate
http://tournament.fantasysports.yahoo.com/t1/1931619

The method is a variation on Boyd Nation's Iterative Strength Rating that incorporates margin of victory and weights late-season games more strongly. This link has more:
https://nathanbrixius.wordpress.com/2013/03/20/ncaa-tournament-prediction-model-2013/

-----------------------------------------------------------------------------

-----------------------------------------------------------------------------

Mark's LR bracket
Mark???
http://tournament.fantasysports.yahoo.com/t1/2504134

Logistic Regression???

-----------------------------------------------------------------------------

Ask me about my T-Rex
Zach Mayer
http://tournament.fantasysports.yahoo.com/t1/1827557

???

-----------------------------------------------------------------------------

ScottyJ's Grand Bracket
???
http://tournament.fantasysports.yahoo.com/t1/1876867

???

-----------------------------------------------------------------------------

Guess O'Bot 3000
???
http://tournament.fantasysports.yahoo.com/t1/1646914

???

-----------------------------------------------------------------------------

Andy's Astounding Bracket

???
http://tournament.fantasysports.yahoo.com/t1/1645698

???

-----------------------------------------------------------------------------

Dan Tran's Dazzling Bracket
???
http://tournament.fantasysports.yahoo.com/t1/1668480

???

And we're off!

2013-03-21T09:46:00.001-07:00

The Machine March Madness competition has officially begun. It looks like we ended up with 22(!) algorithmic submissions, and we have 3 human baselines. You can see all the picks here:
http://tournament.fantasysports.yahoo.com/t1/group/162045

I'll put together a post of all the methods soon. Note: if you are a competitor and have not emailed me a description of your method yet, please do so ASAP.

Final Stretch for Machine March Madness Predictions

2013-03-20T07:53:00.000-07:00

We're in the home stretch for this 2013 Machine March Madness competition, and it looks like this is going to be an exciting year. So far, 17 entrants have joined the Yahoo system, which makes it easily the most popular year yet, and I'm expecting at least a few more before the deadline tomorrow. It also sounds like there are some pretty sophisticated approaches in the works, so I'm excited to see how it all plays out. We should also have some fun baseline predictors to compare against, although if you have ideas about other non-algorithm brackets you'd like to see added, leave a note in the comments.

Once the brackets are locked in, I'll put up a post describing each of the competitors methodologies, and then we can ask some of the competitors to go into more detail.

Finally, if you are planning on competing, please make sure to read the instructions on the Google group to see how to enter, and remember to send me a brief description of your method. Good luck every person, and every machine.

March Madness Team Embeddings

2013-03-19T16:40:00.001-07:00

I went with a new approach to Machine March Madness predictions this year. I won't go into the details right now, but here's a neat visualization that comes out of the algorithm. What you need to know is that I'm sticking with the basic original idea of using latent real-valued descriptors for each team, but I'm abandoning the requirement that there are segregated offensive and defensive descriptors for each team. Instead, the model this year represents each team with a set of numbers that can be used to explain both offensive and defensive performance.

So I'll skip all of the details and jump straight to showing you what the model has learned from this year's regular season. Below is a visualization of what happens when I ask the model to use two numbers to describe each team, then I plot the learned numbers as x and y coordinates on a standard plot.

These results lose the easy interpretability as offensive and defensive strengths, but the model is such that teams in similar locations on the plot will typically be predicted to perform similarly. To help with eyeballing the results, I've color coded 1 through 4 seeds: #1 seeds are blue, #2's are green, #3's are red, and #4's are magenta.

I won't try too hard to explain what's going on, but it does seem to group the stronger teams in the lower and left parts of the plot, and the weaker teams in the upper and right parts. Anybody notice any other interesting patterns?

Machine March Madness 2013. Want to participate?

2013-03-11T15:47:00.000-07:00

Lee and I have been a bit busy this March Madness season and haven't organized the Machine March Madness competition as well as we usually do.

Now it's March, and the tournament is beginning in a little over a week.

This is a post to see if people would be interested in competing this year despite the short notice (update: there appears to be plenty of interest. It's on). Scott Turner has graciously volunteered to help out with data needs (update: data posted at the Google group), and there is still starter code and the Google group available as resources.

If you'd be interested in competing, please drop me an email or leave a note in the comments. Also, if you'd like to compete, but something is holding you back, please leave a note in the comments, and we'll see what we can do.

Neural network successes

2012-11-01T19:50:00.001-07:00

The learning group at University of Toronto has had some great recent successes, both of which were powered by neural networks. Many of you have probably seen the ImageNet results by Alex Krizhevsky and collaborators (team SuperVision):

More recently, George Dahl, a favored guest poster on this blog, led a team that won the Merck Molecular Activity Challenge over at Kaggle:

http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/

It's impressive stuff all around. Great job, guys.

Final 2012 Full-Bracket Results

2012-04-06T08:15:00.005-07:00

Hopefully everyone had a chance to watch the exciting game between Kentucky and Kansas this past Monday. This post only covers the results of the full tournament bracket and not the second chance Sweet Sixteen bracket.

Here are the full standings, including ESPN analysts (E) and my own picks.

TheMatrixFactorizer	127
Jay Bilas (E)	126
Lee's picks	124
The Pain Machine	122
Baseline	120
Danny's Dangerous Picks	117
By The Numbers	104
Dick Vitale (E)	102
Obama	102
Predict the Madness	99
Ryan Boesch	98
TheSentinel	86
AJsMadness	73
machine_learning_first_try	45

Great contest this year and congratulations to this year's winner, TheMatrixFactorizer! It not only won the full-bracket contest, it also squeezed past ESPN analyst Jay Bilas by a point. Once again, machines triumph over humans in our contest. I, for one, welcome our new March Madness predicting robot overlords.

Round 2 Update + Upset Analysis

2012-03-21T18:19:00.003-07:00

Here's another great guest post from Scott Turner, our #1 Machine March Madness guest poster. Great analysis -- thanks Scott! If you want more where this came from, check out his blog.

On my blog here I took a closer look at how the Pain Machine predicts upsets in the tournament and how effective it was this year. I thought it might be interesting to look at how the top competitors in the Machine Madness contest predicted upsets. I put together the following table with the competitors across the top and an X in every cell where they predicted an upset. Boxes are green for correct predictions and red for incorrect predictions. The final row(s) in the table shows the scores & possible scores for each competitors.

Game	Pain Machine	Predict the Madness	Sentinel	Danny's Conservative Picks	AJ's Madness	Matrix Factorizer
Texas over Cincy	X	X	X		X	X
Texas over FSU	X	X
WVU over Gonzaga	X	X			X
Purdue over St. Mary's	X	X	X		X	X
NC State over SDSU	X
South Florida over Temple	X					X
New Mexico over Louisville	X					X
Virginia over Florida			X
Colorado State over Murray State					X
Vandy over Wisconsin						X
Wichita State over Indiana						X
Murray State over Marquette				X		X
Upset Prediction Rate	43%	25%	33%	0%	25%	29%
Current Score	42	43	42	41	41	39
Possible Points	166	155	166	161	137	163

(I'm not counting #9 over #8 as an upset. That's why Danny has only 41 points; he predicted a #9 over #8 upsets that did not happen.)

So what do you think?

One thing that jumps out immediately is that the competitors predicted many more upsets this year than in past years. Historically we'd expect around 7-8 upsets in the first two rounds. Last year the average number of upsets was about 2 (discounting the Pain Machine and LMRC). The Pain Machine is forced to predict this many, but this year the Matrix Factorizer also predicts 7, and Predict the Madness and AJ's Madness predict 4. From what I can glean from the model descriptions, none of these models (other than the Pain Machine) force a certain level of upsets.

Monte's model ("Predict the Madness") seems to use only statistical inputs, and not any strength measures, or strength of competition measures. This sort of model will value statistics over strength of schedule, and so you might see it making upset picks that would not agree with the team strengths (as proxied by seeds).

The Sentinel uses a Monte Carlo type method to predict games, so rather than always produce the most likely result, it only most likely to produce the most likely result. (If that makes sense :-) The model can be tweaked by choosing how long to run the Monte Carlo simulation. With a setting of 50 it seems to produce about half the expected number of upsets.

Danny's Dangerous Picks are anything but; it is by far the most conservative of the competitors. The pick of Murray State over Marquette suggests that Danny's asymmetric loss function component might have led to his model undervaluing strength of schedule.

AJ's Madness model seems to employ a number of hand-tuned weights for different components of the prediction formula. That may account for the prediction upsets, including the somewhat surprising CSU over Murray State prediction.

The Matrix Factorizer has two features that might lead to a high upset rate. First, there's an asymmetric reward for getting a correct pick, which might skew towards upsets. Secondly, Jasper optimized his model parameters based upon the results of previous tournaments, so that presumably built in a bias towards making some upset picks.

What's interesting about the actual upsets?

First, Texas over Cincy and Purdue over St. Mary's were consensus picks (excepting Danny's Conservative Picks). This suggests that these teams really were mis-seeded. Purdue vs. St. Mary's is the classic trap seeding problem for humans -- St. Mary's has a much better record, but faced much weaker competition. Texas came very close to beating Cincinnati -- they shot 16% in the first half and still tied the game up late -- which would have made the predictors 2-0 on consensus picks.

Second, the predictors agreed on few of the other picks. Three predictors liked WVU over Gonzaga, and the Pain Machine and the Matrix Factorizer agreed on two other games. Murray State over Marquette is an interesting pick -- another classic trap pick for a predictor that undervalues strength of schedule -- and both Danny's predictor and the Matrix Factorizer "fell" for this pick.

So how did the predictors do?

The Pain Machine was by far the best, getting 43% of its upset predictions correct. Sentinel was next at 33%. Perhaps not coincidentally, these two predictors have the most possible points remaining.

In terms of scoring, the Baseline is ahead of all the predictors, so none came out ahead (so far) due to their predictions. The PM and Sentinel do have a slight edge in possible points remaining over the Baseline.

So who will win?

The contest winner will probably come down to predicting the final game correctly. There's a more interesting spread of champion predictions than I expected -- particularly given the statistical dominance of Kentucky.

If Kentucky wins, the likely winner will be the Baseline or Danny. If Kansas wins, the Pain Machine will likely win unless Wisconsin makes it to the Final Four, in which case AJ should win. If Michigan State wins, then the Sentinel will likely win. And finally, if Ohio State wins, then Predict the Madness should win.

Second Chance Competition Announcement

2012-03-19T19:03:00.006-07:00

For all of you who didn't get your algorithms finished in time, and for all of the original competitors who'd like a fresh start, we're pleased to announce this year's "second chance" Sweet 16 contest.

This one will be run a little bit differently. For machines, the rules are all still the same. The difference is that there will now be a pool of human competitors in the mix -- Facebook friends and fans of our sponsor, a knee doctor who likes robots.

The prize pool for the second chance tournament will be $50 and $25 gift certificates for first and second place, respectively, and they will go to the top two entrants, whether they be human or computer.

If you want to participate as a human, you need to add Doctor Tarlow on Facebook and look for his announcement there. For those who wish to enter an algorithm, here are the instructions:

Send me email at dannytarlow+MarchMadness@gmail.com with your team name, along with a short description of your approach. (If you entered the main competition and haven't significantly changed your algorithm, just send me a mail saying you're in for the Sweet 16.) Also, consider joining the Google group.
Follow this link, and enter your picks before 7PM EST on Thursday. When entering your bracket name, add "[C]" before your name, to indicate you are a computer entrant:
http://tournament.fantasysports.yahoo.com/t2/register/joinprivategroup_assign_team?GID=9372&P=robotsvshumans

That's it! Good luck to all the algorithmic competitors out there. I hope we can pull out a victory over those pesky humans.

"Predict the Madness" by Monte McNair

2012-03-19T10:57:00.006-07:00

This is a guest post by Monte McNair, the man behind team "Predict the Madness," which is the leader of the machine competitors after the second round.

Developing a system to fill out the best NCAA Tournament bracket is composed of two parts: matchup prediction and bracket optimization.

MATCHUP PREDICTION
The first thing to do is come up with a method to predict the likelihood of one team beating another. Since we only care about advancement, I want a system that produces a perentage as opposed to a point spread or something else. Therefore, I use a logistic regression with the outcome of games being the dependent variable. For the variables, I use the location of the game, metrics for the team's offense and defense, and metrics of the team's opponents' averages for both offense and defense. The NCAA Tournament is played at all neutral sites, but since I'm training on all games, I want to know how important playing at home is so that I can strip this out for neutral site games. The reason to use components of a team's offense and defense as opposed to simply points is that the different components that contribute to points have varying levels of reliability. As KenPom figured out this year, for example, defensive 3P% is extremely unreliable. My model takes this into account and weights it less than it would be if we used its influence on points against. By breaking it down, we let the model determine which factors are most reliable in predicting future performance.

The main thing we care about is that the model does a good job of predicing future games. Instead of waiting for future games, however, we can just use out of sample games. I took about 1/3 of our games and made them training games and left the other 2/3 as testing games. One thing I did that may be different from most is that I used all of a team's games for the season except for the game in question to create their profile. For example, say North Carolina played Duke on January 7th in one of my training games. For North Carolina's profile, I used stats from all of their games before AND after January 7th. I'm not sure what other systems do but I think they might use all games (without excluding the game in question) or perhaps just games PRIOR to the game in question. In any case, after training the model, I can test it against the out of sample games I set aside for testing. I divided up all the test games into 100 buckets ordered by their predicted win percentage and compared it to the actual win percentage in those games. As we can see, the buckets are closely aligned meaning the predictions are fairly accurate.

BRACKET OPTIMIZATION
The next thing to do is to take our matchup predictions and maximize our expected points based on the scoring system we are presented with. While this is most beneficial when scoring systems provide bonuses for picking upsets or some other unique scoring, it can still be helpful in basic scoring systems and is better than simply advancing winners round by round.

As an example, take Louisville and New Mexico, the 4 and 5 seeds in the West region. My model predicts New Mexico as the favorite in a game against Louisville, projected to win 51.2% of the time. Both are favored in their 1st round matchups as well, so if we were to simply advance them both, we'd then choose New Mexico to advance over Louisville in the 2nd round. However, New Mexico has a tougher 1st round opponent in Long Beach State than Louisville does against Davidson. In the table below, we see that New Mexico wins just 65% against LBSU while Louisville wins 75% of the time against Davidson. This is enough to make it more likely that Louisville advances to the Sweet 16 than New Mexico, despite UNM being the better team.

	1st	2nd
New Mexico	64.9%	37.2%
Louisville	75.3%	40.7%

New Mexico over Louisville: 51.2%

In a basic scoring system, this rarely comes into play and when it does, it provides little benefit. But it still is best to be accurate if you can.

Machine March Madness: Round 1 Update

2012-03-17T13:20:00.013-07:00

As usual, the first round was full of upsets, with two of the #2 ranked teams falling. None of our competitors predicted either of those upsets, but they are still putting on a respectable performance. Here are details of each competitor's entry, along with the current performance.

The favorites at this point look like "The Matrix Factorizer" and "The Pain Machine". Both did quite well in the first round, and both have 7/8 elite eight teams still surviving, along with all 4/4 final four teams still alive.

The Matrix Factorizer

Jasper

I modified Danny's starter code in two ways: First, I added an asymmetric component to the loss function, so the model is rewarded for getting the prediction correct even if the absolute predicted scores are wrong. Second, I changed the regularization so that latent vectors are penalized for deviating from the global average over latent vectors, rather than being penalized for being far from 0. This can be interpreted as imposing a basic hierarchical prior.

I then ran a search over model parameters (e.g., latent dimension, regularization strength, parameter that trades off the two parts of the loss function) to find the setting that did best on number of correct predictions made in the past 5 years's tournaments.

24 of 33 Correct, 25 Pts, 171 Pts Possible

The Pain Machine

Scott Turner

Methodology: Linear regression on a number of statistics, including strength ratings to predict MOV (Margin of Victory). Some modifications for tournament use, particularly to force a likely number of upsets.

23 of 33 Correct, 24 Pts, 170 Pts Possible

TheSentinel

Chuck Dickens

Methodology: Using Ken Pomeroy's Pythag formula to rate teams, then calculated the actual game probabilities with the log5 formula. Used a random number generator to determine outcome of games. This provided some randomness which created a few interesting upsets. Simulate the tournament 50 times and record each team's probability to reach subsequent rounds. Step through each round of the bracket choosing winners based on the team that had a higher probability to win that round.

I found that running the simulation 50 times gave me the most variability in the final four, running the simulation more than 100 times gave me a bracket that had almost no upsets and most all of the higher seeded teams progressed through the tournament.

23 of 33 Correct, 24 Pts, 172 Pts Possible

Baseline

Always pick the higher seed.

23 of 33 Correct, 24 Pts, 168 Pts Possible

Ryan's Picks

Ryan

For each season (e.g. 2006-2007) I have enumerated the teams and compiled the scores of the games into a matrix S. For example, if team 1 beat team 2 with a score of 82-72 then S12=82 and S21=72. Ideally, each team would play every other team at least once, but this is obviously not the case so the matrix S is sparse. Using the method proposed by George Dahl, I define vectors o and d which correspond to each teams offensive and defensive ability. The approximation to the matrix S is then just the outer product od' (for example (od')_12=o1d2=S12est). This is a simple rank one approximation for the matrix. If each team played each other at least once then the matrix S would be dense and the vectors o and d could be found by finding the SVD of S (see http://www.stanford.edu/~boyd/ee263/notes/low_rank_approx.pdf). Because this is not the case, we instead define a matrix P that represents which teams played that season. For example, P12=P21=1 if teams 1 and 2 played a game. Now the problem stated by George can be expressed compactedly as, "minimize ||P.*(o*d')-S||_F". Here, '.*' represents the Hadamard product and ||.||_F is the Frobenius norm. In this from, it is easy to see that, for constant vector o and variable vector d, this is a convex problem. Also, for constant vector d and variable vector o this is a convex problem. Therefore, by solving a series of convex problems, alternating the vector variable between o and d, the problem converges rapidly in about 5 to 10 steps (see "Nonnegative Matrix Factorizations" code here http://cvxr.com/cvx/examples/).

See this post for more details.

23 of 33 Correct, 24 Pts

Danny's Dangerous Picks

I started with the basic matrix factorization approach from my starter code, then I added small neural networks that applied a transformation to the base latent vectors based on whether the team was playing at home, away, or in the tournament. These transformation vectors were learned based on season and tournament performance of teams from other years. I split the data into 5 cross-validation sets, and looked for hyperparameter settings that did best on tournament prediction in past years. Like Jon, I also added an asymmetric component to the loss function.

Interestingly (disappointingly), after finding the setting of parameters that did best on past data, my method made some pretty conservative predictions for this year, predicting only 3 upsets.

22 of 33 Correct, 23 Pts, 165 Pts Possible

Predict the Madness

Monte McNair

Methodology: To determine the probability of any matchup (Team 1 beating Team 2), I use a logistic regression using statistics for offense/defense of team and team's opponents plus location, dependent variable is outcome of the game. To select bracket, I use a program to calculate the best possible bracket by maximizing number of points based on scoring system, this correctly accounts for situations where simply advancing favored teams round by round would fail.

22 of 33 Correct, 23 Pts, 157 Pts Possible

AJ's Madness

AJ Diliberto

The methodology is that I selected various stats and gave weight to those that I feel are important, such as points for and against, offensive rebounds, and turnover margin. I also factored in whether they were from one of the big conferences, the level of experience and success the coach has had, and then overlaid the formula with a strength of schedule formula that would reduce certain teams scores based on how good or bad the competition was that they played to get those stats.

22 of 33 Correct, 23 Pts, 139 Pts Possible

Machine Learning First Try

Joe Gilbert

My methodology is as follows:
1. Develop a matrix that contains only 2011 scores (done using your data)
2. Develop a matrix that contains all of your teams and generate columns for averages over all players in 2011: minutes played, FT attempted/made, 3P attempted/made (done), rebounds, turnovers, fouls (again using your data)
3. Use machine learning, specifically a traditional Forest algorithm to predict each team's score for each game based on the 2011 data only
4. Select the winner for each round and repeat step 3 for the next round to determine the next winners
Currently, the algorithm predicted the first round modeling each team's score as an "Away" team since they are all technically on the road. I think I may change it so that the scores are based on a mean value of the model for an Away team and Home team because currently it is predicting LIU Brooklyn over MSU in the 1st round...if it comes true then so be it.

20 of 33 Correct, 21 Pts, 91 Pts Possible

By The Numbers

Tim Jacobs

Methodology:
I took the data so generously provided, trained a couple of neural networks on the past performance, then used average away performance for each team to predict performance in the tourney. The networks are training as I type.

17 of 33 Correct, 18 Pts, 166 Pts Possible

Data Usage Clarification

2012-03-14T07:06:00.003-07:00

I just realized that the data rules and usage discussion happened on the Google Group and not everyone may have read it. Similarly, a clarification on hand-tweaking.

Basically, no human judgment data should enter your model except for your decisions on how to build the model and hyper-parameters for that model. Also, if you do use data that we did not provide, please let us know and please make it available to all the other competitors so that they might have the opportunity to use it as well.

Fast Company Article

2012-03-13T14:27:00.003-07:00

David Holmes over at Fast Company wrote a nice article on about our Machine March Madness contest:
http://www.fastcompany.com/1824382/march-madness-ncaa-tournament-predictions-algorithms

Thanks David!

To everybody else: I hope you're hard at work on your algorithm.

Prizes and deadline reminder

2012-03-13T07:56:00.009-07:00

Now is the time to make a final push for getting your Machine March Madness algorithms tuned and running smoothly. Remember, submissions are due before tip-off of the first game on Thursday, but you probably want to get them in a little early, just to be safe.

I'm also pleased to announce the prizes: for the main competition, the winning algorithm's owner will get a $50 Amazon or Apple gift certificate, while second place will get a $25 one.

Also, for the "second chance" Sweet 16 contest, we will be hosting a humans versus computers contest, with our field of computers competing against Facebook friends and fans of our sponsor, a knee doctor who is into robotic-assisted surgery. The prize pool for the second chance tournament will also be $50/$25 gift certificates, but the prizes could go either to a human or computer.

If you want to participate as a human, you need to add Doctor Tarlow on Facebook, but if you're reading this blog, hopefully you'll enter an algorithm and participate on our team instead.

The human team has chosen the name, "Dr. T's Robot Powers". We'll need to come up with something better for our computer team. Ideas are welcome in the comments.

How to pick upsets?

2012-03-12T14:45:00.008-07:00

Scott Turner writes...

Doing well in a tournament picking contest probably comes down to picking the right upsets. Anyone can pick the higher seeds to win.

Define an upset as a lower seed beating a higher seed, and ignore upsets where there's only 1 step differential (i.e., a #9 beating a #8). If my math from last year is correct, the upset rate in the tournament is around 22%. Half those upsets happen in the first round, about 7.

Some recent thoughts about upsets:

http://harvardsportsanalysis.wordpress.com/2012/03/12/predicting-ncaa-tournament-upsets-the-importance-of-turnovers-and-rebounding/
http://courtsideanalyst.wordpress.com/2012/03/12/two-potential-ncaa-upset-picks-with-supporting-math/
http://www.teamrankings.com/blog/ncaa-basketball/why-you-should-ignore-the-seeds-when-filling-out-your-2012-ncaa-brackets

I leave it to Danny / Lee to turn this into a blog posting :-)

My response...

From a machine learning perspective, I think Scott raises an interesting issue here. Let me rephrase the problem a little more abstractly, to more clearly get at the crux of the issue. Suppose that some oracle were to come down and tell us that exactly 15 of the games in this year's March Madness tournament will be upsets. How should this affect our prediction strategy?

There are probably two natural answers:

Don't change anything. I have my prediction for each game, and I think it's going to lead to the most number of correct predictions.
Make my base predictions, but go back and find the games that I'm most uncertain about, and flip predictions until I am predicting exactly 15 upsets.

Actually, these both are reasonable strategies, but they say something different about the objective function that we are optimizing with our picks. If the goal is to just get as many games right as possible, and we believe our model captures all of the information we have about the outcome of the games (and we believe the game outcomes are statistically independent), then the first strategy will still maximize the expected number of games that we will get correct. However, by making this choice, assuming our model isn't predicting 15 upsets already, then we've eliminated ourselves from contention for the $5 million prize that Yahoo offers to anybody who picks the perfect bracket.

So if the goal is to win the $5 million prize and you believe the oracle, then the right strategy is to pick the 15 upsets that the model thinks are most likely.

However, while both of these strategies make some sense, they both seem too extreme. Perhaps the more natural objective should be to ensure that we win this year's Machine March Madness prediction contest. If that's our goal, what's the best strategy? What if we had the predictions from all of the competitors for past years, and I told you that this year's field was going to be drawn from a similar set of competitors?

See Scott's picks for most likely upsets over at his blog.

2012 Contest Registration

2012-03-11T18:26:00.003-07:00

In order to facilitate in the contest, we will be using Yahoo! again for you to enter your bracket entries. Please do the following to register your team and participate in the contest:

Send an e-mail to "leezen+marchmadness" at gmail to provide your: team name, team member names, and a brief description of your methodology.
Enter your picks in the Yahoo! tournament group with the entry name being your team name.
Watch the tournament with your friends and have fun!

Data for 2012

2012-03-11T18:15:00.002-07:00

Selection Sunday! What a day! First we have a great post by Scott Turner on using RapidMiner. Then, the Selection Committee has set the seeding. Now, it's YOUR turn to predict who will win the 2012 NCAA Tournament.

There are two files you can download:

The includes everything from the beginning of the 2006 season up to and including the March 11, 2012 games. Please let us know if you find any issues with the data. One known issue is that some scores in the first file do not match the scores if you were to add up all the player scores from the player-level data. This is due to the fact that data we crawled is occasionally inconsistent in this regard and might be off by a few points.

The data format is as before for both files, except that the aggregate game data is now tab-separated. Please see aggregate game data schema and player-level data schema for details. Good luck!

Using RapidMiner to Predict March Madness

2012-03-11T12:15:00.004-07:00

This is a guest post by Dr. Scott Turner, who won the Machine March Madness prediction contest last year, and who was the co-winner of the Sweet 16 contest from two years ago. If you like this post, check out his great blog all about algorithmic prediction of NCAA basketball: http://netprophetblog.blogspot.com/.

Dr. Turner has a Ph.D. in Artificial Intelligence from UCLA. His dissertation subject was a program called MINSTREL that told stories about King Arthur and his knights, as a way to explore issues in creativity and storytelling. Since obtaining his Ph.D. in 1993, Dr. Turner has worked for the Aerospace Corporation, where he advises the nation's space programs on software and systems engineering issues.

Danny & Lee asked me to contribute a guest post as part of the Machine Madness contest. I started writing a posting about using RapidMiner as part of a prediction workflow, but unfortunately I became overwhelmed with other tasks and wasn't able to finish it. I had given up on finishing it when I realized that anyone entering the Machine Madness contest at this late date might well appreciate a tool that could make creating the routine parts of building a predictive model very fast. So I quickly finished it up and hope it will prove helpful to someone. Readers who are expert data miners won't find much here, but I hope that it might be useful to the interested amateur who knows more about basketball (football, baseball, etc.) than about statistics and data mining and wants to put in a quick entry.

I will assume that you have some program or method for generating the statistics or ratings you want to use to predict games and that you've saved those results as an Excel file. (These might just be season averages of the statistics Danny & Lee are providing.) As a tool RapidMiner is not well-suited for this part of the problem; it's strengths are in pulling the predictive value out of those statistics rather than generating them. (Or perhaps I should say that it's not well-suited as I understand it. I wouldn't be surprised to learn that it has useful features in this area that I don't know about.) The Excel file should have one line for each game, with columns for the team names, statistics, ratings, and scores.

The next step is to download and install RapidMiner. You can do that here. The "community edition" of RapidMiner is completely free. (I like free.) There's a user forum here where questions usually get a fairly quick response.

Once you've installed, start up RapidMiner. You'll see this:

RapidMiner has three default perspectives: Design, Results, and Welcome. It starts up in Welcome. Switch to Design by clicking on the icon that looks like a pencil writing in a notebook, from the View menu, or by hitting F8. The Design view looks like this:

The blank central area is the canvas where you'll graphically build your RapidMiner process. The left-side has a menu of Operators as well as Repositories (where processes are stored). The right-side has details about the current operator (Just a blank "Process" in this case because we haven't added anything yet.)

To start, let's read in our Excel file of game data. In the list of Operators on the left-side of the RapidMiner window, you'll see a folder labeled "Import". Clicking on that reveals sub-folders labeled "Data," "Models", and so on. Click on the Data folder and you'll see a list of operators. "Read Excel" should be near the top. Click and drag that operator onto the blank area in the middle of the screen and release. You'll see this:

There are a couple of things to note. First, RapidMiner has automatically drawn a connection from the output of this process (the little semi-circle node on the right of the box) to the right edge of the workspace. Anything going out to that edge will show up in the Results view when the process is executed. Second, the message window at the bottom of the workspace shows an error. It is complaining "The mandatory parameter "excel file" is undefined."

To fix this, look to the right-side. You'll see that is now showing the details for the highlighted "Read Excel" operator. Just below there you'll see a button for an "Import Configuration Wizard" and then some input boxes for the various parameters for this operator, including the "excel file" parameter being complained about. There's also a description/help box for the operator below the parameters section.

Use the "Import Configuration Wizard" to find your Excel file and prepare it to be read in. The wizard does some basic data checking, so you may discover a problem in your file at this point. Here's what the final step of the wizard looks like for my sample data:

There are 8 columns to my data: name, score, TrueSkill mean, and home winning percentage. (The TrueSkill mean is a rating system. You can read more about it here.) These will be the inputs to my prediction model.

To run a process in RapidMiner, you click the right-facing blue triangle button near the top of the window. Right now our process isn't very interesting -- it just reads in the Excel file and sends it to the Results -- but let's run it and see what happens. You may be asked to save your model and whether you want to switch to the Results view. For both questions you can save a default answer, which is handy. When you switch to the Results view you'll see something like this:

The data you read in creates an "Example Set" and this window is showing you the Meta Data View for the data set. In my case, the data set has 3699 examples (games), and for each attribute in the examples, the window shows the Role, Name, Type, Statistics, Range and Missings. There's some interesting stuff here -- for example, home teams scored between 28 and 124 points in this season. A home team scored only 28 points?! That's pretty intriguing.

Let's follow up. Click on the "Data View" checkbutton and then on the Hscore column to look at the actual data sorted by home team's score:

Apparently that 28 point performance was put in by SMU against UAB. That had to be fun to watch! You can do some interesting data analysis with the Plot View and Advanced Chart options here, but let's continue on with building a process.

Switch back to the Design view and let's work on conditioning the data. In many cases, there are problems in the input data -- such as missing values -- that will corrupt your prediction models. RapidMiner provides a number of operators for fixing these sorts of problems. Let's work on fixing missing values. In the Design View on the Operators tab on the right part of the screen you'll see a search box. This is handy for finding operators by name. Type "missing" into the Search box and you should see this:

Click on the "Missing Values"operator, drag it onto the canvas in the middle of the screen and drop it. You'll now have this:

You'll see that RapidMiner is complaining of an error in our process: we don't have an input to the Replace Missing Values operator. We want to connect the output of our Excel file to the input of this operator. To do this, we left click on the output of the Read Excel operator, and drag the resulting orange line to the input of the Replace Missing Values operator and release. This causes a pop-up box asking if we really want to disconnect the current output connection or not. Allow RapidMiner to disconnect the port and you should have this:

And that's all you need do: Add operators and hook them together into a process. By default, the Replace Missing Values operators replaces all missing values with the average value for that attribute. That's fine for now, so we'll leave it as is.

One very important step we need to take is to create a "label". The label is the attribute that we're trying to predict. In our case, we'll be trying to predict the winner of the game: "Home" or "Away". We don't actually have that in our input data, so we'll need to create a new attribute and set it to be our label.

To do this, find the "Generate Attributes" operator and the "Set Role" operator and modify your process to look like this:

Now click on the "Generate Attributes" operator. On the right you'll see a button labeled "function descriptions" and "Edit List(0)". Click on this to bring up a view that will let us define a new attribute in our data set.

This is fairly simple to use. We type in a name for our new attribute in the left-hand column and then an expression for calculating it in the right hand column. We can use any existing attribute in our expression, and if you click on the calculator icon, it will bring up a tool to help create expressions. In our case, we want to create a new attribute called "winner" that has the value "Home" if the home team scored more than the Away team, and "Away" otherwise. The expression to do this is 'if(Hscore>Ascore,"Home","Away")':

And that's it for creating the new attribute. Now we need to set the Role of this attribute to "label" so that our models will know what we're trying to predict. To do this, click on the Set Role operator and in the right-side pane, select our new attribute from the drop-down box next to Name, and "label" from the drop-down box next to "target role":

We're almost ready to start modeling, but let's check to make sure we've added the "winner" attribute correctly. Hit the run button to run the process and let's look at the output in the Results view:

At the top of the results (colored light yellow because of its role as "label") we see the new attribute "winner". In this data set, the Home team won almost twice as often as the Away team. If you click on the Data View button, you can check a few games to make sure the calculation is correct:

Looks good, so let's go back to Design View and train a model. Switch back to the Design View and find the k-NN model, drag it into the process and connect it up to look like this:

Along the right-side you can see the parameters for the k-NN operator. Change "k" to 3. We're almost ready to create a model, but we need to add one last step. Right now the input data to our model includes the scores of both teams. It isn't very hard to predict who will win the game if we know who scored the most points :-) so we'll need to remove that information from our examples. To do this, we need an operator called "Select Attributes". Drop this into our process between "Set Role" and "k-NN".

Highlight the new operator, and on the right-side, set the "attribute filter type" to subset and then click on "Select Attributes". That will bring up this dialog:

Now we simply select attributes we want to include from the left side and use the green arrow to move them to the right side. We want to leave out the Hscore, Ascore and Date attributes.

Save this and we're now ready to run the process to create a model. Hit the Run button and you should see results that look like this:

Great, we created a model! But how good is it? We don't have any idea. To figure that out, we need to apply the model and then measure its performance. Let's do that.

Switch back to the Design View, and find the "Apply Model" and the "Performance (Classification)" and add them to your process after the k-NN operator like so:

Note that the model output of the K-NN operator goes into the model input for the Apply Model operator, and the example set output goes into the unlabeled input. The labeled output of Apply Model goes into the labeled input of the Performance operator, and the performance output of that operator goes out the right-hand side of our process.

Run this, and you should get a Results View that looks something like this:

Wow, 83% accuracy predicting the winner of the game -- pretty good! Good enough to win the Machine Madness contest? Who can say? :-)

This illustrates the basics of using RapidMiner for prediction. RapidMiner has a wealth of features and options, and there are many improvements you can make to the simple process flow I've illustrated above. But hopefully this has given you enough guidance to get started, and good luck!

Preliminary Aggregate Data

2012-02-28T11:26:00.003-08:00

For those of you who want to play with just aggregate game result data, I've posted an updated version that you can play with. The format is the same as described in a previous post: date, home team, away team, home score, away score, and whether or not the home team won.

This data covers the 2006 season through 2/26/2012 and, as with the player-level data, will be updated on Selection Sunday to reflect the most up to date information.

Preliminary 2011 Season Data

2012-02-27T17:17:00.002-08:00

In addition to data from the 2006-2010 seasons shared publicly via Google Docs

We've published some preliminary data for the 2011 season. This uses the same format as past seasons' data and spans the beginning of the 2011 season through 2/26.

Preliminary 2011 data

After Selection Sunday (March 11th), we will publish an updated set of data for the 2011 season. Please let us know if you find any problems with the preliminary data.