Nate Silver’s book “The Signal and the Noise”

 

The book by Nate Silver “The Signal and the Noise …” is an amazing read. Very well written, entertaining as well as deep, it holds lessons and learnings that are applicable in our daily personal and professional lives. Its stated purpose is to look at how predictions are made, their accuracy, in several fields : weather, stock market, earthquakes, terrorism, global warming … But beyond that simple premise, it is a real eye opener when it comes to describing some of the deeply flawed ways in which we humans analyze the data we have at hand, and take decisions.

Nate Silver has very skeptical towards the promises of Big Data, and believes that the exponential growth in available data in recent years only makes it tougher to separate the grain from the chaff, the signal from the noise. One of the way he believes we should strive to make better forecasts, is to constantly recalibrate our forecasts based on new evidence, and actively test our models to improve our predictions and therefore our decisions. The key to doing that is Bayesian statistics … This is a very compelling, if complex, use of the Bayes Theorem, and it’s detailed through a few examples in the book.

As he explains, in the field of economics, the US govt publishes some 45,000 statistics. There are billions of possible hypotheses and theories to investigate, but at the same time “there isn’t any more truth in the world than there was before the internet or the printing press”, so “most of the data is just noise, just as the universe is filled with empty space”.

The Bayes Theorem goes as follows :

P(T|E) = P(E|T)xP(T) / ( P(E|T)xP(T) + P(E|~T)xP(~T) )

Where T is the theory being tested, E the evidence available. P(E|T) means “probability of E being true if we assume that T is true”, and notation ~T stands for “NOT T”, so P(E|~T) means “probability of E being true if we assume that T is NOT true”.

A classical application of the theorem is the following problem : for a woman in her forties, what is the chance of her having a breast cancer if she had a mammogram indicating a tumor ? The basic statistics are the following, with their mathematical representation if T is the theory “has a cancer” and E the evidence “has had a mammogram that indicates a tumor” :

– if a woman in her forties has a cancer, the mammogram will detect it in 75% of cases – P(E|T) = 75%

– if a woman in her forties does NOT have a cancer, the mammogram will still erroneously detect a cancer in 10% of cases – P(E|~T) = 10%

– the probability for a woman in her forties to have a cancer is 1.4% – P(T) = 1.4%

With that data, if a woman in her forties has a mammogram that detects a cancer, the chance of her actually having a cancer is of …. less than 10% !!! That seems totally unrealistic – isn’t there an error rate of only 25% or 10% depending how you read the above data ? The twist is that there are many more women without a cancer (98,6%) than women having a cancer at that age (1.4%), so the number of erroneous cancer detections, even if they represent only 10% of the cases where women are healthy, will be very high.

That’s what the Bayes theorem computes – the probability of a women having a cancer if her mammogram has detected a tumor is :

P(T|E) = 75%x1.4% / ( 75%x1.4% + 10%x98.4% ) = 9.6 %

Nate Silver uses that same theorem in another field – we have many more scientific theories being published and tested every day around the world than ever before. How many of these as actually statistically valid ?

Let’s use the Bayes theorem : if E is the experimental demonstration of a theory, and T the fact that the theory is actually valid, and with the following statistics :

– a correct theory is demonstrated in 80% of cases – P(E|T) = 80%

– an incorrect theory will be disproved in 80% of cases – P(E|~T) = 20%

– proportion of correct to incorrect theories – P(T) = 10%

In that case, the probability of a positive experiment meaning a theory is correct is only of 30% – again a result that goes against our intuition, as it seems from the above statistics that the “accuracy” of proving or disproving theories is 80% !!! The Bayes Theorem does the calculation right, and takes into account the low probability of a new theory being valid in the first place :

P(T|E) = 80%x10% / ( 80%x10% + 20%x90% ) = 30 %

There again, events with rare occurrences (valid theories) tend to generate lots of false positives. And this results in real life in a counter-intuitive fact : at the same time as there is a huge proliferation of published scientific research, it has been found that two-thirds of “demonstrated” results cannot be reproduced !!!

So … this book should be IMO taught in school … It gives very powerful and non-intuitive mental tools to make us better citizens, professionals and individuals. I don’t have much hope of this making its way into the school curriculum any time soon, so don’t hesitate, read this book, and recommend it to your friend and family 🙂

 

Really good fantasy books

I just finished reading the second book of The Kingkiller Chronicle by Patrick Rothfuss. After a fantastic first book, I was wondering if the author would disappoint or run out of steam. But he continues to weave a fascinating and deep tale.

The story on the surface is a very classic fantastic tale of a young man getting into a magic academy, in a typical medieval-fantastic setting. What differentiates it is the depth and details of that world, which are slowly unveiled, with great consistency. It is like seeing a rich tapestry getting woven in front of your eyes. The characters themselves are very deep, and multi-faceted.

Patrick Rothfuss uses classic themes of the genre like magic, fairies, or dragons, but subtly gives them his own twist, and keeps the readers always on their toes, never knowing when what seems familiar will become strange and mysterious.

I am really impressed. The only comparison that comes to mind is the Lord Of The Ring ! It is not (yet) a tale as epic as LOTR – probably halfway in that regards between The Hobbit and LOTR – but what it lacks in drama and scale, it more than makes up with more rich and complex characters, and a very modern read on their motivations and lives. One thing Rothfuss and Tolkien have in common is the place that languages, music and songs occupy in their stories.

I can’t wait to read the third book, which unfortunately will probably only come out in May ’13 !

Amazon US link

 

The evolution of human civilisations – and a glimpse into our future

I am almost done reading “Why the West rules – for now” by Ian Morris. It is simply one of the most impressive books I have read in my life.

The author starts by trying to answer the question “why did the west get to the industrial revolution first, and hence got to dominate the twentieth century ?”, but the book is not really about answering that question. Along the way Ian Morris analyses how civilisations grow, evolve and decline. It is a lot of learnings from the past, but also gives insights into our future – some of them quite scary 🙂

It is extremely well written, fun, entertaining, a real page turner. It’s a book you really enjoy reading, but which messages echo well after you have put it down.

Here are some of the learnings and questions I took away :

  • All human have gone through the same stages of evolution, through social, economic and technological development.
  • They also have all faced a ceiling, a limitation to their development inherent to their social and technological levels.
  • And when they reach that ceiling, most often they don’t just plateau there, they also collapse and lose from their peak level. And some of these collapes are just not pretty to live through.
  • Geography and climate have played a significant role in determining the pace of development and the fate of civilisations.

Now for the great and also the scary part : the “Social development index” that Ian Morris has measured from 14,000 BCE to today shows a huge steep increase in the past 100 years. One component of that index is the energy capture, ie how much energy we use to lead our lives, including the food we eat, animal muscle we leverage, or fossil fuels we burn. In prehistoric times this was at 4 kilocalories/person/day, and 14,000 years later during the Roman empire had gone up to 31. It then collapsed and only reached that level again in the west in 1700 CE (China had reached that level earlier in 1600). By 1800 it was still only at 38. But with the industrial revolution everything changed, and it had soared to 92 by 1900 (The eastern world by then had slipped behind and only reached 49), and further accelerated in the twentieth century to reach a whopping 230 (the east is on the same course and even though it is at “only” 104, its growth curve is rapidly catching up).

So we should feel priviledged to live in that “world of plenty” that is our era. It is not only unprecedented in terms of the civilization levels we have achieved, it has also seen a dramatic acceleration of wealth and possibilities in the last century, and does not give any signs of deceleration. It might also be the first time we have have a shot at building one global civilization, instead of the conflicting empires or nations of the past.

But we should also be scared that our civilization might hit a development ceiling like we always have before, and that we might have to live through a collapse in development level, with its accompanying unrest and destructions.

So how do we build new social models that will take us through that ceiling, and let continued technological developments keep carrying us forward ? Is climate change going to undo our progress at planetary level as it undid might empires in the past ? Will we really build that global civilization we have started to create, or will states, nations and empires prevail and be the main barriers to progress ?

History is the best way to put our humanity in perspective, and it gives us insights into who we are and the challenges lying ahead of us. This books provides a unique lens to think about our future, and to consider how technology and social models will shape our destiny – a must read !

The Dunbar’s number

Happening to read about mammal history, learnt today that the neocortex is unique to that class of creatures. And the size of the neocortex has in turn been correlated to social interactions … Robin Dunbar, a British anthropologist, suggests that for humans the number of people one can entertain stable social relationships is around 150. This has probably some implications on how we can best organise our enterprises … More at wikipedia’s entry.

Of brainstorming, innovation and workspaces

Both easy to read and thoughtful, this article by Jonah Lehrer in the New Yorker begins by debunking the idea that the “no criticism” rule in brainstorms unleashes creativity. Lehrer goes to explain why healthy criticism actually sharpens the mind and results in more and better ideas from a team. Expanding from brainstorms to other modes of interactions, he then looks into how work spaces can foster innovation. He has great examples, from an old MIT building whose layout would have had a big impact on a rich history of innovation, to the Apple Headquarters which Steve Jobs designed himself, creating a central atrium and moving mailboxes and bathrooms there to force people to have these chance hallway encounters that create new sparks of ideas !

Now the question is how will we recreate these workspaces in our era of constant travels and virtual collaborations 🙂

Read the article here.

 

 

The surprising truth about what motivates us

We so often get caught into “management school thinking” that we forget about common sense. We know that people don’t work just for money, we also have seen first hand recently the kinds of disasters created by grossly overpaid people, so this video should not come as a surprise. Research has identified that monetary incentives only drive performance for repetitive, mechanical tasks. For tasks requiring intellect (which is most of the work nowadays), money is only a factor if you don’t have enough of it, but additional incentive don’t drive performance.
What matters most is Autonomy (ie do your work self directed), Mastery (ie have the opportunity to grow your skills), and Purpose (ie knowing the positive impact that your work will have).
When I reflect on my work life, I find that I know this but have still too often succumbed to the concept of cash incentives for performance. Also I have seen others drive these for real reasons, mostly greed frankly. For them, implementing high cash bonuses for their teams had a real advantage : they would mechanically drive their own compensation up, regardless of the real impact of the bonuses.
So, a great video to reflect upon, and to share with your teams.

Democracy for the digital age

At dinner in NYC with Koen and Ann, we had a very interesting discussion about democratic processes. Belgium, France and the USA were obviously great material, respectively with their deliquescent state, banana republic, and polarised politics. Add to these a declining participation to the voting process, and it really looks like there is an opportunity to do better than these old democracies.
It should probably not surprise us, as our political representation systems date back from times when the fastest one could travel was on horseback, when most people lived tilling the soil, and few could read and write. The state then had much less complexities to manage, and the pace of change and decision making was measured in months, years or decades.
Isn’t it time then to move to more direct systems where citizens with access to immense amounts of data and analysis can directly participate in dialog, express their opinions, and take decisions without having to vote for “representatives” ?
This of course would be a huge shift, one that will be resisted by the current professional political class. It is also a complex one, which will require to reinvent all the safeguarding mechanisms, the checks and balances that took centuries to evolve.
But isn’t it already happening ? The political class in France recently showed how out of touch it was with the public opinion during the DSK sex scandal. They initially made statements supporting DSK, hinting that aggressive sexual advances was nothing to be fussy about, really an expression of French Don-Juanism. Within days or hours of their first reactions, they adjusted their stance, no doubt as their political advisors decrypted the online forums… The real-time data mining and analytics around social networks is already shaping the actions of politicians. We should probably take control of that process before they learn how to subvert it.

The most heartening piece of news this year so far !

As a 13 years old boy from New York was looking at trees, he observed that there was a pattern to how branches were positioned. Enquiring further, he found they follow a fibonacci sequence … Nothing extraordinary so far, this is a fact already known, except that few thirteen years kids will figure that out 🙂 … The extraordinary thing is that he then thought of comparing how solar cells placed using that pattern compare to a regular flat array of photovoltaic cells, and found that they generated 20-50% more power !!! Through natural selection, trees have apparently evolved over millions of yrars to adopt that configuration and increase the amount of energy they capture, but no-one had thought of using this for solar panels before.

This is such a refreshing story … One of curiosity, intelligence, and one that proves again that there is no limit to understanding more and more of the world around us, and use the learnings to improve our lives !

To read the paper written by that boy, Aidan Dwyer, himself, follow this link.