Building Models And Understanding Reality

The wonk in me loves models - over the weekend someone asked me for some help with Excel and I was disappointed that they just needed help with some formatting and basic calculations because I love a model that separates out and compares different data sets and uses a variety of measures to create aggregate performance metrics. It's a lot of fun figuring out which numbers have what influence on the whole and there is an analytical purity that you can achieve which is a beautiful thing.

Years ago I worked for PRTM collecting huge reams of operational data related to product development and supply chain performance. After the data was calculated we would use the metrics to tell a story about the business we were analyzing based on how their data compared to a similar set of companies. I remember that many an executive was a bit surprised at how much we could know about their business just looking at the data. Then as an analyst at IDC, I was introduced to systems dynamics modeling which takes modeling to a whole new level - multiple variables can feed into complex feedback loops that represent behaviors. It's really cool and also really instructive at testing hypotheses.

However, way back at PRTM I learned a really critical lesson about models. Models don't and can't tell you what to do. They should not be used to make decisions - at least not on their own. A very common reaction I got when I presented results to large companies was "So that means we are bad/good/average and we should invest/change direction/etc."  I was always a little confused by the question because while the data definitely pointed to possible issues or opportunities, I had none of the other business context to make a good decision. Models are good at identifying potential issues and opportunites and for showing improvement but they are one data point and are pretty dumb when it comes to strategy and relationships.

This frustrates me to no end when I see banks melt down because they thought sub-prime loans were a good idea. Their models told them that the price of houses, on average over the prior decade, grew in value by x such that even for people that put no money down, the increase in housing prices had the probability of giving them equity in a few years - based on historical trends. But even ignoring the ethical responsibilities they should have felt, they forgot to think about the assumption that home mortgages were a good idea to give to people who had no skin in the game. There is something about spending years to save a downpayment that makes one think long and hard about just how much they can afford - and take that responsibility a little more seriously. It seems like they let their models do the deciding for them. I'm not sure why.

In a different area, Google takes a very model heavy approach to everything it does, from its search algorithm to it HR practices. This bothers me in a vague way that I can't always articulate. But it hit me yesterday when someone emailed and told me that I, Rachel Happe, had as many search results as Tina Fey. That makes no sense. They were using it to show how important social media and personal branding is. The problem? Google 'corrects' my typing so when I type in 'Happe' it pulls up those results but also assumes that I really meant 'happy' and pulls up all of those results as well. I'm sure that 99.9% of the time when people type in 'happe' they mean to type 'happy' and the model holds. But what about me? The mode doesn't work all that well for me. Clearly this is kind of a stupid example but it demonstrates how we use models every day that don't always work.

My plea? Use numbers but don't just blindly follow them. Double check, they can be at best usually right and at worst misleading in dangerous ways.

Comments

[Trevor Rotzien]

Excellent call to reason.

Reminds me of that life skill quote "all models are wrong, but some are useful."
http://en.wikiquote.org/wiki/George_E._P._Box

[working girl]

I once wrote an article about just this called 'Stop Measuring and Start Managing.' But you know, I would work that Tina Fey thing more - if Google says it it must be true. ;-)

[Rachel Happe]

Those are both great sayings!

As I mentioned to Trevor over Twitter, it is just as important to know the limits of a tool/discipline as to know how to use the tool itself.