Slashdot Log In
Fooled by Randomness
from the likely-story dept.
| Fooled by Randomness | |
| author | Nassim Nicholas Taleb |
| pages | 220 |
| publisher | Texere |
| rating | 8 |
| reviewer | Max Tardiveau |
| ISBN | 1587990717 |
| summary | Debunking fallacies of observation, Taleb reminds us of the pervasive ineffables that complicate life at every turn. |
The main topic of the book is the fact that all humans are simply terrible at judging probabilities. Taleb is a securities trader, so a lot of the book revolves around financial probabilities, but his argument is mainstream and requires absolutely no knowledge of the markets. The book details examples of people wildly misjudging risks and probabilities in many contexts. Often that misestimation is understandable in advance of certain events, but harder to excuse after they've occurred; Taleb hits pretty hard on what he calls "data snoopers," his term for people who back-fit theories to existing data.
One of the most notorious examples is the Bible code (which has been thoroughly debunked), but Taleb argues that analysts who spend their time trying to find patterns in historical market data are no different: if you try long enough and hard enough, you will unavoidably find apparent regularities, which can be extremely compelling when seen in isolation. In context, though, they dissolve into nothing but meaningless statistical anomalies. Taleb rightfully compares these searches for meaning to the famous monkeys and typewriters parable.
Taleb's best example of poor probability intuition is probably the infamous survivor bias, which is our tendency to be disproportionately impressed by success. We almost always ignore the fact that, for one success story, there are many failures. But we seldom hear about the failures (just like we never hear about the many theories that didn't fit the data). So it's all a game of numbers: out of 10,000 traders, a few are statistically bound to be successful, even if they are nothing more than lucky idiots. The fact that they succeeded does not mean anything. It doesn't mean that they are bad traders, but it doesn't mean that they are good traders either, because on average somebody had to succeed.
One of Taleb's hot buttons is that people tend to be too confident in what they know. He argues convincingly that we should take everything, including science, with a grain of salt. Writing about Karl Popper, he points out that there are only two kinds of scientific theories: those that are demonstrably false, and those that are not yet demonstrably false. An irksome (but sadly true) observation, yet most people behave as if what they know is eternal truth. One could of course argue that Popper's observation is but another kind of truth, but I tend to put a lot more trust in people who question what they know than in people who don't.
Another of Taleb's peeves is the human tendency to over-attribute every random event (the old post hoc, ergo propter hoc). For instance, a commentator saying that "the Dow went down ten points today on concerns about Iraq" is talking nonsense: there is no way anyone can tie such a small market move to any particular reason. I found this specific point (which in retrospect is blindingly obvious) especially enlightening, as I am embarrassed to admit that, until now, I just accepted those market comments at face value.
Taleb also has some fun at the expense of economists and analysts, especially those whose predictions turned out wrong, but who claim that the theories were in fact right, and that the facts simply weren't supposed to be that way. This is what he calls denial of history, and is common among investors and gamblers (the two being of course close cousins).
The style of the book is informal and funny, and often meandering. We hop from one topic to the next, which occasionally may detract from the book's continuity, but overall the author's points come through loud and clear. Ironically for a man who advocates self-doubt, Taleb is starkly self-confident, though not in an irritating way.
Taleb is an intriguing, multi-cultural, iconoclastic character that has been around Wall Street for a while, and now runs his own small firm. Malcolm Gladwell (author of The Tipping Point, an absolute must-read for anyone who owns a brain) has written an excellent article that shows how Taleb's reasoning runs counter to just about every bit of conventional Wall Street wisdom. If you're interested in the markets, especially derivatives, and how Taleb trounces most of Wall Street's voodoo doctors, this moderately technical interview from 1996 is worth reading too.
Overall, a warmly recommended book.
You can purchase Fooled by Randomness from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Dumb Luck (Score:3, Offtopic)
There are times where I feel like I should have died but I haven't. Falling off a ravine into a river, losing control of my car in a turn with traffic on the other side. Just dumb luck.
Fooled by Randomness (Score:5, Funny)
Good point! (Score:4, Funny)
Good point. I'd say I'm that good and
(ughhhh)
NO CARRIER
Conservative/Liberal take on it (Score:5, Interesting)
(http://www.sirsonic.com/)
Thats called the Just World mentality. Its not a Just World
I've found this is the single biggest commenality in partiasian schools of thought:
Conservatives tend to think you are where you are because you deserved it. Dumb and lazy people are poor, smart and hard working people are rich. Dumb people get hit by trains, smart people comment on how dumb they are. Your situation is a result of your disposition.
Liberals go the other way. Luck, circumstance, and opportunity play huge roles in where you are. Poor people are poor because of luck and circumstance. People get hit by trains because they might have just plain been unlucky. Your situation is a result of your environment, including dumb luck.
Personally, this is the single biggest reason I can't stand conservatives. It bothers me to no end how capable they are of assuming that anybody in a bad situation is there because they deserve it.
Re:Conservative/Liberal take on it (Score:5, Funny)
(http://hammeroftruth.com/)
Re:Conservative/Liberal take on it (Score:5, Insightful)
Re:Conservative/Liberal take on it (Score:5, Insightful)
(http://eagerfeet.blogspot.com/)
The truth is that while some people may become successful or suffer terribly largely through luck, there are situations where you can drastically improve or destroy your life through your own choices.
You can't blame everything on circumstance, but you can't take credit for every good thing that happens to you, either.
That's why I can't stand cardboard conservatives or cardboard liberals. The world is so much more complicated than either of them give it credit for.
(Yeah, I know, off-topic
Re:Conservative/Liberal take on it (Score:4, Insightful)
(http://homestarrunner.com/)
Liberals go the other way. Luck, circumstance, and opportunity play huge roles in where you are. Poor people are poor because of luck and circumstance. People get hit by trains because they might have just plain been unlucky. Your situation is a result of your environment, including dumb luck.
Personally, this is the single biggest reason I can't stand conservatives. It bothers me to no end how capable they are of assuming that anybody in a bad situation is there because they deserve it.
This is why both hard line views annoy me. I imagine the truth is right smack in the middle of these two outlooks. It is just as much folly to assume everything is based on luck, and that hard work, skill, intelligence, whatever, have nothing to do with it.
Finkployd
Mischaracterization of conservativism (Score:5, Informative)
This viewpoint is absolutely not essential to conservativism. It is unfair to characterize conservativism as people who punish the poor because they "deserve it."
There is a legitimate conservative idea that many well-meaning social welfare programs actually trap people in poverty by encouraging failure and discouraging success. Clinton famously "ended welfare as we know it" because he believed this kind of reform was the right thing to do.
Of course there is also a wrong kind of conservativism (practiced by many Libertarians and Republicans), in which welfare is cut, while job training programs are cut as well.
Centrists of both parties (such as Clinton) recognize and co-opt the good parts of conservativism and of liberalism. When you say that you can't stand conservatives, I think what you really mean is that you can't stand the self-righteous, self-promoting bigots who make a mockery of the ideology they unfortunately claim to represent.
Re:Conservative/Liberal take on it (Score:5, Insightful)
(http://chrisbbehrens.blogspot.com/)
I've found this is the single biggest commenality in partiasian schools of thought:
Conservatives tend to think you are where you are because you deserved it. Dumb and lazy people are poor, smart and hard working people are rich. Dumb people get hit by trains, smart people comment on how dumb they are. Your situation is a result of your disposition.
As a conservative, I'll take the bait. I think that "deserve" is too complex an issue to really know (abused as a child, ends up as a criminal, etc.). It is a core conservative belief, however, that people's fates are in their own hands. I certainly won't sign on to that smart and hard working people automatically get rich, but I've had pretty extensive personal experience with why people end up poor:
#1 reason: being part of a couple that gets pregnant outside of marriage. If you're the woman, you now have the financial burden of a child without the full support of a father. Additionally, you don't enjoy the economies of scale enjoyed when two or more income earners live together. If you're the father, the latter point still applies, and you have child support to pay.
#2 Reason: not completing high school. In our society, for better or worse, this is a signal to all employers that you haven't got your shit together. There are, however, many cases where high school dropout succeed as entrepreneurs.
#3 Reason: Drug / alcohol abuse. No explanation necessary, presumably.
Here's the defining characterstic of a conservative: the belief that if someone works hard and consistently, they can improve their condition. Not everyone who believes this is a conservative, but if you don't believe it, you can't call yourself a conservative. To address your definition of a conservative, I would say that conservatives believe that in almost every circumstance, people are poor as a consequence of poor choices they made.
So, I'm one of the kind of people you can't stand. I suspect that you don't know me very well. I believe that power over one's circumstances comes from persistent personal action and not from authority. Do people end up with bad luck out of no action of their own? Of course, everyday. But is that the defining circumstance of human existence, or is persistence, faith and audacity?
Re:Conservative/Liberal take on it (Score:5, Insightful)
(https://addons.mozil...&application=firefox)
So hard work does nothing, eh? It's all luck and circumstance. Pfff! Every time I've stopped worrking my situation got worse, and every time I started working again, it got better. What luck I must have had lately, since I've been doing better the whole 8 years I've been working where I am. I've even been lucky enough to have people pay me when I do side work for them on weekends!
Are you on crack ?
This is my favorite quote above:
People get hit by trains because they might have just plain been unlucky
Provably false. People get hit by trains because they're on the railroad tracks when the train goes by. This has nothing to do with luck, or circumstance. It has to do with bad decision making. Nobody's car ever "stalls out" on the tracks like in the movies. People try to "beat the train"... and fail.
Re:Conservative/Liberal take on it (Score:4, Insightful)
(http://www.unixnetworking.net/ | Last Journal: Wednesday December 04 2002, @05:55PM)
8 years ago I was barely making minimum wage had a wife a kid and due to bad decision making had a degree that was basically worthless. Working two jobs and for all intents and purposes living in a cardboard box.
By making a hard choice, go into the Air Force, while in I worked hard and learned everything about anything anybody would teach me. I also looked around the world and came to realize that computers and networks were *very* big and only going to get bigger I started to insert myself into every situation where they needed someone to help out with any computer related. At the same time I started to devour every book on networking that I could get from any library that I had access to. As a result of that I got out 4 years later and worked at a crappy temp job for a few months while shopping out resumes. Since I had very little experince I got a crappy job doing tech support for a medimum sized ISP. While there I continued to work my ass off and learn everything I could and read everything I could get my hands on. As a result of doing that for the past few years I've managed to go from *very* poor to be rather well off. And it was all hard work. I have yet to meet a person, myself included, who does not waste time. Put even a small perctange of that wasted time to work and you can improve your situation. So yes they should work harder and study more.
Re:Conservative/Liberal take on it (Score:5, Insightful)
(http://slashdot.org/)
Where did he say that? He actually said "Luck, circumstance, and opportunity play huge roles in where you are." Huge roles being the keywords. As someone else who replied to you said, how about being hit by a hijacked airplane while sitting in your office? Or being born as a woman or slave in Sudan [aol.com]? Keep working hard, maybe you won't get beaten as badly.
Complete egoism (Score:5, Insightful)
What ruined the book for me was Nassir's incessant and shameless self-promotion. He shows the same hubris that he finds reprehensible in others and he seems to be trying to settle a few old scores by maligning people.
But the most ridiculous part of the book is when he starts drawing links between finance and neuroscience and trying to explain traders behavior based on our very preliminary understanding of brain physiology. Utter crap.
The basic message is sound: Wall Street is full of people getting paid millions who haven't the slightest understanding of math, science or technology. But that message is clear in about 10 pages and the rest is self-serving, self-promoting tripe.
Re:Complete egoism (Score:5, Informative)
The book has one other serious flaw which really merits mention in a review: it is very badly written. 'Meandering' does not describe the book as well as 'completely unstructured'. Also, there is too much repetition for such a short book.
The overall impression is that Taleb has published the notes for a book, rather than the book itself. With proper writing and editing, these notes would, of course, make an extremely brief book; but certainly there is material here for an instructive essay.
Why are YOU alive? (Score:3, Funny)
Its my belief that some people are only alive because its illegal to kill them.
dumb luck and slashdot (Score:3, Funny)
true but, (Score:4, Insightful)
(http://www.joeandmonkey.com/ | Last Journal: Friday March 21 2003, @03:44PM)
In poker, I never expect to get the cards I want, but I do stay in the game when the odds are high mathematically. If I get my flush or full house, then it was my knowledge of probablity that succedeed. If I have to fold, then nothing is lost, because the result was unpredictable, and I made the best choice according to my opinion.
So, while people often overread things (like the possiblity of terrorist attacks), we can still succeed in randomness if we have knowledge of all the variables. This knowledge, is of course, what leads to success in all fields that involve probability.
interesting book (Score:5, Interesting)
One thought - his big point with regard to the market is that people who tend to make a small/medium, consistent profit may be trading in a style which has a low-probability, high-damage outcome. This "black swan" event would/will wipe out those people when it occurs.
His big strategy is to be hedged against the worst which can happen and also make big money on the unlikely events in his favour.
Big question: does anyone know how he did when the markets crashed? (The book was written a couple of years back). If he has been practising what he preaches (and if it works!) then he should have made a killing...
Mistaking cynicism for theory (Score:3, Insightful)
(Last Journal: Wednesday December 13 2006, @06:43PM)
sad (Score:4, Funny)
Obviously trained in the Classics...
Perceptions of coincidence (Score:5, Informative)
(http://www.starwars.com/ | Last Journal: Wednesday February 19 2003, @04:52PM)
http://www.nytimes.com/2002/08/11/magazine/11COINC IDENCE.html [nytimes.com]
denial of history (Score:5, Interesting)
(http://slashdot.org/)
well, i suppose i'm asking for it, but
there are *alot* of cases in poker (and presumably, in investing) where one can do the "right thing" and lose big money. this is because short-term statistical fluctuations dominate in poker (investing?). the only edge one can gain is long-term, and long-term means several years of playing several days a week.
The ability to seperate the correct decision from the outcome, while still being honest about analyzing one's play, is one of the keys to successful poker.
It's like Millikan throwing away the oil drops that didn't have the right e/m
-- p
Limited Scope (Score:5, Insightful)
This is true only in endeavours which are zero-sum for a community as a whole.
Not true otherwise.
For example, take the solving of some math problem. Like Reimann Hypothesis. It's not __necessary__ that someone __has__ to solve it. The one who does, is a good mathematician.
Theories (Score:3, Insightful)
(http://slashdot.org/)
Well, yeah - that's why they are theories and not laws. The same sort of argument is made by fundamentalists against the "Theory" of Evolution. It can't be proven. It's only a theory (so let's go back to something infinitely less provable/disprovable such as the divine). Theories are meant to be correct in so much as there is overwhelming evidence toward their ideas.
The book appears to make some good points, however. (Just from the synopsis provided.) The infinitude of random factors that may cause market fluctuations makes prediction completely (probably?) intractable. In CS lingo, it would appear to be NP-Complete.
Re:Theories (Score:5, Informative)
(http://slashdot.org/)
The infinitude of random factors that may cause market fluctuations makes prediction completely (probably?) intractable. In CS lingo, it would appear to be NP-Complete.
Okay I am a CS theory puritan and a troll. But here is the deal.
Rant #1: Intractable problems are unsolvable problems. Even if you have infinitely powerful machines. Because our logic (the foundation of our math) has "holes" that prevent it from solving them. Halting problem (can you write a program that will verify if a program is correct)is an example. It simply cant be done.
Rant #2: NP hard problems are hard to solve. You just need lots of time. Throw in a trillion years and a very powerful computer and heck - you have a solution. Quantum computing *may* solve these NP hard problems in polynomial time. Still Quantum computing cannot solve intractable problems.
Rand #3: NP problems are not the most difficult to solve problems (among the solvable problems). In fact they are the easiest. There is something called polynomial hierachy of harder and harder problems and NP is at the very bottom.
Rant #4: Only problems that have a one bit solution (yes or no, true or false) can be NP complete. Others are "hard".
Further references to be directed at "computational complexity" by papadimitrou.
Sorry but had to take it out on someone
Another Good Book on a Similar Topic (Score:5, Informative)
Random walks (Score:5, Interesting)
(http://www.doxagora.com/ | Last Journal: Friday February 28 2003, @05:39PM)
Surfing on the Noise Floor (Score:4, Interesting)
(Last Journal: Wednesday December 12, @10:23AM)
The stock market, day traders, etc are all dependant of this sort of thing. even though the possibily that their successes are often not distinguishable from random luck gives them all nightmares.
kinda difficult to pick out signals where they are buried in the noise floor to begin with.
Another good book (Score:3, Informative)
(Last Journal: Tuesday February 18 2003, @12:19PM)
Very good read.
hitchcock stock scam (Score:5, Interesting)
- mailed 1/2 the folks a prediction that stock X would rise, mailed the other 1/2 the opposite.
- threw away the 1/2 who ended up with the incorrect prediction
- repeated until there were a few folks left who had gotten 10 correct predictions (either 4 or 8 people, depending on n)
- asked the remaining folks for investments.
- split with the cash
Now imagine stock analysts all making predictions. Eventually there will be a star.Pretending the world is NOT random is most logical (Score:5, Insightful)
(http://www.joblessjimmy.com/ | Last Journal: Tuesday January 07 2003, @09:27AM)
The solution is to PICK something important that may not be meaningless and random. After all, the world just might not be meaningless and random. This is kind of like Pascal's wager. If you go around believing everything is meaningeless and random and act accordingly like it is, than you gain nothing. You'd sit home and drink beers and watch Jerry Springer all day because theoretically, you aren't any worse or better off than Mother Theresa, Bill Gates, or any other "successful" person. If, however, you pick a goal or ideal to work toward and give it meaning, your life does have meaning and value. At the very least, if others around you deem your efforts worthy, you will gain the love and affection of others (which MOST people are biologically wired to need and crave). Therefore, it is possible to gain something (the pleasure of being valued) out of the void of randomness by giving your life some purpose, even if that purpose is ultimately meaningless in the big scheme of things.
So keep playing the game of life. You've got nothing to lose and only stand to gain.
I read it on Slashdot!
estimation (Score:3, Interesting)
(http://alienbrain.net/mike/ | Last Journal: Thursday February 27 2003, @04:18PM)
Something else I've noticed people being bad at is estimating ratios. Example: back in high school an English teacher asked the class what the population ratio of blacks to whites in Michigan was. Some guessed as high as 50%, but I was the only one with the correct (lowest) answer of around 10%.
I have always wondered what affected this the most. It could be that we were a suburb of Detroit, or that the local television stations' news programs may have featured more black people (scaring white viewers makes for good ratings), or that these kids hadn't travelled very far from the Detroit area, or that they were just really lousy at estimation of distances and population and had no concept of the size of the state.
Certainly, some of the error may have come from cultural bias. I didn't notice any overt racism in the school, but the specter of the "dangerous negro" was/is probably still present.
Is this still on-topic? Anyway, it seems that we need to do better in our educational system of teaching people how to get their heads around the concepts of chance and estimation. Letting any bias influence these will cause errors.
Quantifying the randomness (Score:3, Interesting)
(Last Journal: Tuesday January 21 2003, @12:29PM)
I've long since been a believer in the "largely random" nature of the world and this puts me in mind of a wonderful piece of "research" I came across in New Scientist a few years ago.
It cut through a lot of bullshit to ask the question:
"can you predict the longevity of a system or state if you know almost nothing about it"?
i.e. life's pretty random and bothering to analyse the details will only get you so far
So how can we address, from a probabalistic standpoint, questions like:
"How long will the pyramids stand"?
"How long before my bank goes bust"?
"How long before we're hit by an asteroid"?
The reasoning went like this:
Take any "thing"/"system" etc. with a finite existance and ask the question:
"With a 95% certainty, how much time does it have left"?
For the sake of argument lets say you plant a tree, and you know nothing about trees.
Ten years later you come back and the tree is still alive and so you ask the above question about the tree.
Well, the tree is clearly alive but you don't know where "now" is in it's lifespan.
From a purely probabalistic point of view you've got a 50% chance that "now" is the middle half of it's life, i.e. you're not in the 25% of it's "youth" or the 25% of it's old age. So thats a 50% chance that it's had 75% of it's life already (middle age + youth).
Turning this into an equation, if:
x = current age
y = total age
p = probability
then: (it helps with a diagram)
x = py + (y-py)/2
or, rearranging to solve for y:
y = 2x/(p+1)
Going back to our tree then, if it's 10 years old
it's got:
95% chance of making it to 10.26 years
50% chance of making it to 13.33 years
The magic figure I keep in my head is that if you know NOTHING about something, you've got a 95% chance that it will last 1/39th as long again.
I find this little nugget invaluable when considering how much to spend on insurance or investments - cos I know/care NOTHING about them.
After all, why bother with the details, life is random
Proven with Experiments? (Score:3, Insightful)
I'm just as suspicious of claims that it's all luck as I am that it's all skill.
Another Book to Check Out (Score:3, Insightful)
(http://www.scorchingbeauty.com/)
Money (Score:3, Funny)
(http://www.filejournal.com/)
Taleb makes a living from unforeseen events...
That's his job ? Doesn't that make a cashflow forecast a bit tricky ?
Bank Manager: Mr Taleb, you're mortgage payment bounced this month
Taleb: Oh. I never saw that coming...
Re:Aren't we bred lucky (Score:3, Informative)
(http://slashdot.org/)
Larry Niven played with this idea in the Ringworld books. Basically, overpopulation meant children were allowed by lottery
Hahaha
-- p
Re:Aren't we bred lucky (Score:4, Interesting)
I always found that absurd. Every one of us simply by fact of having been conceived has overcome odds of tens of millions to one simply by winning the sperm marathon. Then we overcame another obstacle of around three to one by surviving to birth; most don't, they fail to implant in the uterus. Luck. If luck was a psi power that could be selected for, then everyone would have it in spades. Or whatever the trump suit happens to be :-)