Measure f-ing everything, and assume f-ing nothing!! - Or how mentoring ruined lives :-(
I recommend listening to the podcast if you have the time, but here's the summary. Stephen Dubner describes the Cambridge Sommerville Youth Study. The impact of social intervention programs in general is hard to measure and so they seldom are. This was the first attempt at measuring the impact over a long period of time.
It's a great story and there are a few good take-aways, but here's the main one: troubled or at-risk youth that received mentoring (good mentoring!) had worse life outcomes across every dimension than the kids that were left alone. Despite the recipients saying that the mentoring was incredibly valuable and helped them, they were actually worse off across a bunch of measured dimension.
Everyone's assumption: mentoring troubled youth will help them in life was wrong - at least the kind of mentoring that they were doing then. They were able to actually measure and determine this only because computers were just starting to become available and the person leading the research was determined enough to figure out how to use them.
This isn't unusual. A lot of things can seem obviously good especially to people working on them, but most don't really do the work to understand (or measure) the long-term impact of the changes they're making. Sometimes it's because they don't feel the need to - a lesson learned in this Steven Seagal classic. Other times it's because it's really, really hard to measure these things.
This is a great reminder that the impact can be very, very different from what you expect or what seems "obvious". It can even ruin people's lives.
So, find a way to measure everything. Always.
Comments