Thursday, September 23, 2021

Big data and the curse of dimensionality

I just finished a fabulous book, Everybody Lies, written by Seth Stephens-Davidowitz.  From the Amazon description of the book:
Everybody Lies offers fascinating, surprising, and sometimes laugh-out-loud insights into everything from economics to ethics to sports to race to sex, gender and more, all drawn from the world of big data. What percentage of white voters didn’t vote for Barack Obama because he’s black? Does where you go to school effect how successful you are in life? Do parents secretly favor boy children over girls? Do violent films affect the crime rate? Can you beat the stock market? How regularly do we lie about our sex lives and who’s more self-conscious about sex, men or women?

I particularly liked the metaphors that Stephens-Davidowitz uses to describe his results.  For example,  in describing why it is easy to come up with variables that correlate with the stock market, but hard to find ones that can make accurate predictions, he uses the metaphor of coin flipping:

Suppose your strategy for predicting the stock market is to find a lucky coin -- but one that will be found through careful testing. Here's your methodology: You label one thousand coins - 1 to 1,000. Every morning, for two years, you flip each coin, record whether it came up heads or tails, and then note whether the Standard & Poor's Index went up or down that day. You pore through all your data. And voila! You've found something. It turns out that 70.3 percent of the time when Coin 391 came up heads the S&P Index rose. The relationship is statistically significant! Highly so! You have found your lucky coin! 
Just flip Coin 391 every morning and buy stocks whenever it comes up heads. Your days of Target T-shirts and ramen noodle dinners are over. Coin 391 is your ticket to the good life!

Every statistics user should know that when running 1000 hypothesis tests, on average 50 of them will show statistically significant results, even when there is no relationship.  This is the size of Type I error (5%) in classical hypothesis testing.

Instead, split your sample in two and use half the data to "find" (estimate) one lucky coin; and the other half to test it.

BOTTOM LINE:  the more tests you run, the more likely it is that at least one of them will show a statistically significant relationship, even if there is none.  This is likely behind what has become known as the replication crisis, that has hit the field of psychology particularly hard as only one third of the results from the most cited articles could be replicated.  It is likely that academics are testing lots of hypotheses and publishing the few that turn out to be statistically significant.  This is analogous to finding a lucky coin, as it only appears to be lucky.  Once you test it outside the sample, the luck disappears.

TRUTH IN BLOGGING:  the field of economics has its own replication crisis, only two thirds of top results could be replicated.


  1. I know of a group of private investigators who can help you with they are also hackers but prefer to be called private investigators They can help with your bitcoin issues and your clients will be happy doing business with you,they can also help yo with your bad credit score,hacking into phones,binary recovery,wiping criminal records,increase school score, stolen files in your office or school,blank atm etc. Just name it and you will live a better life
    whatsapp +1 (984) 733-3673
    telegram +1 (984) 733-3673



    I have been in financial mess for the past months, I’m a single mum with kids to look after. My name is REBECCA MICHAELSON, and am from Ridley Park, Pennsylvania. A couple of weeks ago My friend visited me and along our discussion she told me about DR BENJAMIN OWEN FINANCE of (; that he can help me out of my financial situation, I never believed cause I have spend so much money on different loan lenders who did nothing other than running away with my money. She advised, I gave it a try because she and some of her colleagues were rescued too by this Godsent lender with loans to revive their dying businesses and paying off bills. so I mailed him and explain all about my financial situation and therefore took me through the loan process which was very brief and easy. After that my loan application worth $278,000.00USD was granted, all i did was to follow the processing and be cooperative and today I am a proud business owner sharing the testimony of God-sent Lender. You can as well reach him through the Company WhatsApp +19292227023 Email



    Hello everyone, My name is Mrs. Carolin Glowski, I'm from Europe, am here to testify of how i got my loan from BENJAMIN LOAN FINANCE after i applied Two times from various loan lenders who claimed to be lenders right here this forum, i thought their lending where real and i applied but they never gave me loan until a friend of mine introduce me to {Dr. Benjamin Scarlet Owen} the C.E.O of BENJAMIN LOAN FINANCE who promised to help me with a loan of my desire and he really did as he promised without any form of delay, I never thought there are still reliable loan lenders until i met {Dr. Benjamin Scarlet Owen} who really helped me with my loan and changed my life for the better. I don't know if you are in need of an urgent loan also, So feel free to contact Dr. Benjamin Scarlet Owen on his email address