good afternoon everybody it's really great to be with you the goal of my

session is to talk about two very specific blocks to potentially poor decisions and what the best investors do to overcome them the goal of session two

is to be very practical I hope when you go back to your desks you'll immediately be able to implement some of these ideas and ultimately make better decisions.

Adam Grant mentioned earlier today the good judgment project a project at University of Pennsylvania on forecasting.

turns out they did something very interesting when forecasters uh volunteered to participate in their project they either uh in control gave them no training or they gave them some training.

and the question was did the training make them better at making better forecasts and the answer to that was yes but it's actually somewhat qualified.

there are different flavors of training that have different impacts.

the first is just giving people information about hertica what they were doing before so it doesn't change any Behavior.

the two that we want to focus on is process so very specific tools and techniques we can um work on to uncover biases and then finally giving people feedback certainly a theme that's come through a lot today.

there is a little bit on format so how you can reframe problems I'll give you one example that's a little bit in that in that flavor but we're going to focus mostly on process and feedback.

so let's jump into this I think the first very big barrier to making good decisions is the notion of overconfidence and I think the best investors are are very good at overcoming this.

I think the best investors are are very good at overcoming this.

now it turns out overconfidence which is often considered the root of all problems comes again in different varieties.

the first I'll say to the very bottom is this idea of overplacement people think of themselves is better than they are versus others.

so you've all heard this you know I'm above average driver my sense of humor is better than 90% of other people.

the other is idea of overestimation which people absolutely think they're better at certain tasks you know so I can make this seven foot foot put or whatever it is.

so overplacement but the one that we're most interested in I think is this concept of over Precision.

we actually think we understand the future better than we actually do and I think the really good investors are those that can combat and overcome that.

let me give you an example of how this works this is a sort of a test of calibration in some degree a test of overconfidence.

um by the way the URL is down there if you can't find it or to see it just let me know it's a very fun exercise to do in your own organization or with your loved ones.

it's a it's 50 true false questions some of which you'll know the answer to many of which you will not know the answer to.

so you answer true false and then you give a confidence 100% means I know the answer to this question 50% means I don't know the answer answer and I'm guessing right so this is the basic setup.

now we've had now about 20,000 people do this exercise and this is the scatter plot of all the responses.

by the way I do want to mention two and I'm sure they're guys I'm going to mention two guys that are my two favorite dots.

the one is the upper right hand corner which is 100% correct and 100% confident.

so this this guy Googled everyone and like the second guy that was even better is in the bottom right hand corner who his 35% correct and 95% confident.

you just hit true true true true you do better than that guy did.

now the goal of this exercise not to be right about every question if you can be right that's great but the goal of the exercise as you might imagine is to know what you know and to know what you don't know.

the goal of the exercise is to lie as close to the 45 degree angle line as possible.

now even people in the very back in the room can probably see this when I give you the numbers you won't be shocked to hear this the average confidence is 70% and the average correct is about 60%.

people are 10 percentage points overconfident in their own judgments.

now let's unpack that a little bit to see if there's anything interesting.

well the first is the distribution of overconfidence versus underconfidence by the way I mentioned this 10% overconfidence it turns out the the the standard deviations also tend so it's a beautiful shaped Bell distribution when you have this many people.

and about 82 82 or 83% of people are overconfident that's something we see very consistently.

by the way as a side note and I'm glad you're all sitting to hear the this information men tend to be more overconfident than women and young men tend to be more over confident than older men so there's some patterns there.

now this is where it gets really interesting it turns out that uh how do people answer when they're doing this exercise right?

so again you're getting 50% means I'm guess I don't know the answer to 100%.

so it turns out when people don't know the answer they they know they don't know the answer and they answer they don't answer.

so they're actually pretty they about 40% of the aners is I don't know which is fine.

what's fascinating is the second most common response is I've got this I'm 100% sure I know the answer.

so between I have no idea what's going on and I know the answers about two-thirds of all responses.

and people are very sparing in using those intermediate ranges.

by the way as Adam's talking about super forecasters he alluded to this but super forecasters are much better at updating their views and being more uh granular and actually setting their forecasts.

okay so how do people do given their confidence well this is the most fascinating chart.

turns out when people don't know the answer again they don't they answer they don't answer and they're pretty well calibrated at I don't know.

but as you move from left to right on this chart you see this systematic departure from the 45 degree angle line.

when people are 100% sure they know the answer they're only right 77% of the time when they are 90% sure they're only right 2third of the time.

right there's this yawning Gap especially in this range of I'm very sure what's going on.

Howard Mar remarked about this just this morning that this is the sort of the Mark Twain thing when you're very confident about something you're wrong about it.

this is where trouble can can arise.

this is also very very important for uh portfolio construction concepts.

by the way there's an interesting uh just this there's a brand new paper about this but there's a new survey.

there's a there's a group called the survey of professional uh forecasters that forecast economic things so this is unemployment rates and interest rates and so forth.

and they have a data set going back to 1968 so 55 or 60 years data on this and it's the exact same thing.

they say what's going to happen with this thing with some sort of probability and here again these are professional forecasters and here you see this exact same pattern unfold high confidence uh poor conviction.

now why is this uh you know what's it what's exciting about this on the other hand is uh going back to the concept of feedback is it turns out when people get feedback on their forecast they get better at it over time.

so this is a picture of actually other forecasters the the line the line at the bottom is actually the first year they did this these are large sample sizes.

and then the next year they repeated the exercise with the feedback from the first year and you can see they got marketly better.

so this should be very encouraging to all of us that if we practice and get the feedback we get better at this right.

but keeping track of your decisions is really crucial to this.

okay so what's going on here how can we improve broadly speaker what are the best investors do.

uh Annie mentioned Danny Conan we all mourn his loss he was extraordinary.

but one of the concepts that Danny popularized is this idea of the inside versus the outside view.

see the inside inside view says when I give you a problem it could be any kind of problem the classic way to solve this see if this makes sense to you is you gather lots of information you combine it with your own experience and then you project into the future.

right left door devices that's what we do information my experience what's going to happen in the world.

the outside view by contrast says I'm going to think about my problem is an instance of a larger reference class.

I'm going to ask what happened when other people were in this situation before.

it's a very unnatural way to think about the world right because you have to leave aside your cherish information and you have to find an appeal to the base rate.

what psychologists like conman have demonstrated is some combination of the inside and the outside view um lead to better forecasts.

let me give you one example from the world of business this is pelaton right and this is an analyst report that was written in September 2020.

so please roll your mind back 3 and a half years to the middle of the pandemic.

they were doing they were doing great because everybody wanted to exercise while they were at home.

they had just come finished a blockbuster year 2020 and the analyst wrote in his report that our base case scenario is they're going to grow Revenue 30% for the next decade.

and by the way it's a very I mean I'm tell the guy is a serious model he's got it's all built out you know it's like it's actually quite credible.

so there's that's his inside view.

now the question you are going to ask ask hopefully using the danyc Conan mindset is how many companies with $2 billion of turnover grew 30% a year for 10 years and here are the data on this.

I put the peleton bike the actual the guy's forecast um but you could see the average is about 6% the standard deviation is a little bit more than that.

this is the guy's base case so you might say if he's wildly optimistic he should certainly have a scenario for that but that certainly would not be your base case right.

so this gives you an example of how you might temper that.

there's a great quote from Danny conman by the way which really applies to investors he says people who know uh information about their own individual circumstances feel no need effectively to appeal to the base case the reference class to which it belongs right.

you feel like I've done the work I don't need to know what happened in the rest of the world.

when we know combining these two things is very powerful.

so why do we do this uh a few reasons.

one is Annie just mentioned one of them one is called uh three Illusions.

one is called the illusion of superiority we tend to think of ourselves as better than those around us.

by the way why do I have a picture of Mother Teresa.

25 years ago US News and Ro report did a survey of 1,000 Americans and they said what's the probability of going to heaven.

and they talked about like they asked about Michael you know Michael Jordan and all these people.

anyway Mother Teresa scored 79% which seemed a little low to me by the way she's now a saint so maybe people will bump it up.

but this is the best Mother Teresa 79 they said you uh survey answer what is the probably will you go to heaven 87%.

Mother Teresa I promise you I would not answer higher than Mother Teresa.

um the second is the illusion of optimism it tends to be we tend to be very optimistic about uh our our outcomes that are important to us.

and so as a consequence we tend to over calibrate our success.

and then the illusion of control this is good and when we when we deem ourselves in control situation we think our odds of success are much higher than they are objectively.

when we deem ourselves in control situation we think our odds of success are much higher than they are objectively.

uh turns out when they go to Las Vegas watch people rolling dice when they want High numbers they throw them really hard when they want low numbers they throw them very gently.

doesn't work by the way but just this idea of being in control of the situation.

here's the last thing I'll say in this which you can all relate to which is statist this is statistical thinking.

and it turns out in the real world what carries the day are stories.

your parents told you stories you tell stories to your children.

stories move people stories teach people stories are memorable statistics are not.

so this is an interesting experiment where they went to people and they said you have a horrible disease you have a treatment alternative it's always 50 50 so you have that in your back pocket.

and then they said we're going to give you or they're going to show you one of four Alternatives 30 50 70 or 90% probability of success.

so hopefully I'm making myself clear everyone's got 5050 you're getting 30 neutral K.

so here let me just draw your attention to the extremes if you give people a positive anecdote they select the 30% Effectiveness.

now 30 versus 50 they selected almost eight out of 10 times.

if you give them a negative anecdote they select the 90.

now 90 vers 50 they select it less than four out of 10 times.

the stories swamp the statistics and people making decisions.

by the way extremely important as Leaders very important for you as you think about stocks and narratives that go around them.

let me now turn to the second um major problem this is very much involved with what Annie was just talking about uh I'm going to say it's confirmation bias.

confirmation bias as you all know even if you struggle to make a decision once you've made a decision you seek information that confirms your point of view and you discount dismiss or disavow information that does not support your point of view.

now there's a very interesting point about confirmation bias it turns out almost all the biases heuristics and biases that are out there that con and trki identified are mitigated to some degree by intelligence.

so if you're you're a smarter person on average you learn about them you tend to do better.

the one that that's not true for is confirmation bias this affects everybody no matter who they are and how bright they are.

2005 was the first chance I ever had to meet Danny conman it was an extraordinary day an extraordinary session.

at the very end of him I said Danny if there's one thing a investor can do to improve his or her results what would it be.

without hesitation he said you must keep a journal of your decisions.

write down what you think is going to happen why you think it's going to happen.

he said WR write the date and time he said by the way note how you feel physically and emotionally about this.

but the most important thing is as you're expressing your variant perception as you're expressing your view that's different what's in the market for an investment for specifically he said communicate using probabilities not words.

because if you use words they're going to be vague enough that you can uh massage them after the fact.

but if it's a probability it allows you to keep score.

and here's one thing I would say if you're going to do a probabilistic forecast you want three elements.

one is an outcome that we can agree upon second within a specific period of time and then with some probabil probability.

so you might say for example weather is a good example Tomorrow there's an 80% chance it's going to tomorrow right.

so that would satisfy that if you have those three ingredients you are in business for doing uh probabilistic forecasts.

now to give you some sense of how Wild this can be uh I did a survey with my older son.

we U asked 20 we put out 23 common words or phrases to a population of people and we asked them say what's the associated probability You' sign to this word or phrase.

and what you can see the answers are all over the place especially with a cognate.

now this is for maybe n native English speakers but the cognate possible if you say something is possible people interpret this in extraordinarily wide ranges.

so the reason you want to avoid words like that it's a serious possibility we'll have a recession next year or whatever it is is because a you're not going to be able to keep track of your forecast and you're going to weeze a lot of it after the fact.

a you're not going to be able to keep track of your forecast and you're going to weeze a lot of it after the fact.

but B if you're communicating with someone they may hear something very different than what you mean and that's crucial.

now the last aspect of this is uh this came up a little bit before but I think it's fascinating in in Adam's com.

is this guy I love his name is Keith stanovich he's a developmental psychologist uh most recent Meritus at University of Toronto.

he's written a lot of books but he wrote a book in 2009 called what intelligence tests Miss.

it's not the snappiest book title of all time but it's pretty descriptive of the content.

and what he argues is there's this thing called IQ and you know more of it tends to be better than less of it that's fine.

and there's a thing he calls RQ rationality quotient the ability to make good decisions.

he says that's really what's important and and they're kind of on different sliders.

and we all know this right you've all gone to school people did really well in school and not so well in life and we all know people are not very good in school but very good at life.

this is this RQ IQ.

so what is RQ all about it turns out there are two components to it.

one is called instrumental rationality which is a fancy way of saying you can achieve your objectives given your constraints.

the way we think about this in economics is the axum's utility Theory.

the second one's one I think we care about and that is this idea called epistemic rationality.

it's a fancy word of saying how accurately do your beliefs map to the world.

and by the way you woke up this morning with a certain set of beliefs of how the world works.

you wandered around information confirmed it some information may have disconfirmed it.

what did you do in the face of that evidence?

so here's a quote from Phil techlock which I love and you heard this earlier today in a different form beliefs are hypotheses to be tested not Treasures to be protected.

the best investors take that to heart beliefs are hypotheses to be tested not Treasures to be protected.

let's do this together ready late in the day here's an example of how we might do this right.

this is um an example of how this works in integrating information.

so imagine I hope I can read this imagine a zapper virus causes a serious disease that occurs in one in every thousand people.

say there's a a u a test that can detect perfectly if you have the virus going to you're going to be uh test positive.

additionally let's say that the test actually has a 5% false positive so if you don't have it it's going to say you have it one and every 20 time 5% false positive.

okay so now we choose a person at random we know nothing about his or her medical history and that person tested test positive for the zapper virus.

what is the probability from 0 to 100% that that person has the virus right.

so the test 1000 test is perfect 5% Falls positive we have someone that has the virus what's the probability anybody know this one.

but notice you have like these numbers in your head it's confusing but what you're doing is integrating various pieces of information.

and the answer is just a bit under 2%.

and let me explain how to come up with that very quickly in in a way that's easy to reframe the problem.

let's say there are a thousand people we're going to test all of them.

first person we're going to test the person that has the virus they test positive we put them in the denominator.

we're then going to test the other 999 5% of which will test positive we put them in the denominator.

so now our denominator is 51 we said we picked one person at random they tested positive one over 51 that's your 2% right.

when I say it that way it's easy to follow no.

but you're integrating different pieces of information and that's what great investors were able to do quite effortlessly.

here's the last thing this ties together a bunch of things earlier from Adam and also from what Annie just talked about.

this is something called lynchpin analysis.

this comes from the intelligence Community when you're looking at investment idea what you want to do is identify the key issues that are really important for that for most stocks or even asset classes.

two or three things tend to be important.

pinpoint the working assumptions behind that so-called Lyn pin premises.

um Advanced evidence for or against that and this is where you can come up with things like The Devil's Advocate.

and then any indicators or signposts right.

so the way you can do this is say we have a variant perception here's what we believe is going to happen and you write it all down in advance.

the key to these signposts is that kill criteria that Dan was talking about.

so I know in advance if I have a thesis that's different than the market I should have these I should pass these signposts on the road and if I don't then I must change my mind about what's going on.

and then last by the way I did this with Chad I was very proud of myself I did this with Chad GPT so this is like my my generative AI thing.

but this is just a Winding Road this how I really Envision it.

that you see these signposts as you go along and if your thesis is correct you should anticipate seeing those signposts.

and if you don't see the signpost you expect that is an opportunity or in fact really a challenge to change your view.

so that let me stop what the best investors do is calibrate well to start they do that primarily by appealing to base rates.

and second of course when new information encourages you or forces you to change your mind you should change your mind when appropriate.

thank you very much.