Your understanding is incomplete. Bayesian statistics are not subjective at all. Instead they objectively describe the correlation between prior belief and conclusion. People may draw different conclusions for subjective reasons, but that subjectivity is in the people, not the statistics.
Now it seems you are objecting to the fact that your initial state of mind affects your conclusion. But it is unavoidable that there is no way to make sense of observation except through the lens of prior beliefs. You can be explicit about it and draw conclusions with a correct methodology about it as in Bayesian statistics. You can be implicit about it and draw conclusions with an incorrect methodology as in frequentist statistics. (See my other post in this thread for an explanation of why frequentist methodology is incorrect.)
Let me offer a simple example. Suppose a pregnant woman you know gives birth to a boy. Is that evidence that babies are more likely to be boys than girls? Obviously it is. Is it strong evidence? Obviously not. Should we upon observing that conclude that boys are more likely than girls? Obviously not.
Now suppose that you have no prior knowledge other than we see about similar numbers of boys and girls. I know that of the last 100,000 babies born in the USA, 51,157 are boys. Suppose we are both told that at the local hospital, 92 of the last 200 babies born were boys. I submit that we both will and should wind up with different conclusions about the relative likelyhood of boys and girls. Why? Because different prior knowledge lead to different prior beliefs, and those prior beliefs when modified by identical evidence lead to different conclusions.
You have to admit that your example is pretty contrived though - usually setting priors isn't so clean.
How do you 'properly' set the prior probability that someone really has E.S.P.? That a researcher is secretly colluding with a test subject? That a researcher is flat out fabricating their results? That this is just an instance of the Hans effect? Or a thousand other possible hypotheses ...
I picked my example so that the proper influence of the prior would be clear. You are absolutely right that there are many cases where different people with equivalent information have different beliefs. But being faced with complications like this makes correct reasoning more important, and not less.
When you try to sweep the proper influence of the prior under the carpet, people will manipulate their statistics to draw the conclusions that they want. Worse still, if each is using subjective techniques that they believe are objective, the unexamined assumption will lead to them talking past each other.
By contrast with correct reasoning you can show each why they continue to believe what they believe while showing them how the other side is not going to have their opinions changed. In my experience this opens up a bigger possibility of useful dialog and changed opinions.
Say a proper Bayesian who's a fan of 'da Bears' just saw the bears lose again, badly. How is he supposed to update these two beliefs:
- this season 'da Bears' will tear shit up every Sunday (fwiw assume prior on this has it as likely)
- I am usually over-optimistic about 'da Bears''s performance (fwiw assume prior on this also has it as likely)
...in light of the new evidence?
(Side note: I'm specifically not talking about bayesian analysis or statistics in a formal setting with a prespecified, finite list of allowed hypotheses (like spam or ham).
If I wrote this out in extreme detail I'd get nothing done today, I think you can charitably fill in the gaps and missing pieces of what I'm getting at; if not I'll fill them out in a few.)
There are a ton of things to clarify, but I'll take a stab at it.
A proper Bayesian who is betting and doesn't want to lose badly and still ends up as a fan of the 'da Bears' has probably got a prior that gives some confident edge to them winning. Belief A could be stated using this joint distribution as the product of their predicted chances of winning each game in order. If it's actually 'likely' then that means you've got a pretty incredible edge on them winning (the size proportional to how much season is left).
Belief B is a weird one though. It's a meta-hypothesis about the calibration of your own personal beliefs. The evidence you use to update on this belief is the discrepancy between what you would honestly predict and what actually happened. A proper Bayesian with money on the line would want to recalibrate as best as they could using data already available in order to get the probability of B as low as possible before starting to look at A.
So our proper Bayesian first looks over old predictions he has about the Bear's performance, reworking whatever internal understanding of the factors that go into winning in football he has, until he is well calibrated. At this point, his probability for belief A has almost certainly dropped because it's a pretty unlikely thing for a team to just take a thing apart at every single game for the whole season, but if he still ends up with a strong prior on them winning then a single loss, even if it's pretty bad, won't shift it around a lot.
In short, he'll think about it a lot, cancel out whatever personal biases he can manage, then bet conservatively unless he has some sort of knowledge that provides a really, really strong edge on them winning. IMO, he's got an inside line with some dirty, dirty men.
The specific example of 'da Bears' was a weak attempt at humor. I can't easily clarify what I'm getting at and keep this short, and I've got limited time so I'll do the best I can.
I see a lot of people using "informal" bayesian reasoning (meaning a lot of talk about priors and updating and reference to theorems but never any use of actual distributions beyond super-super-cursory examples applied to trivial situations like the boy/girl thing here or stuff like the monty hall problem).
I don't have any problem at all with bayesian analysis applied in a rigorous setting to a rigorously specified problem (like spam detection and so on).
In an informal setting I'm extremely skeptical of the uses I tend to see b/c there's no careful attempt to clearly delineate which informally-statable hypotheses are valid and which are "invalid" "meta-hypotheses" like the optimism thing.
What you've described here is a way in which someone reasonably smart would eliminate the meta hypothesis, which is fine. In general I wouldn't expect it to be feasible to take a full mental inventory, do a topological sort on your beliefs, and then apply the same procedure; most people most of the time will be running around holding partially-inconsistent beliefs (where "hold" means if you were to ask them to give an estimate of, say, what beliefs they had about what # of their beliefs were likely to wind up revealed to be significantly off in the future, or to give an estimate of what they believe about the frequency with which they'd encounter evidence leading to significant revisions of their beliefs, they'd have an answer on offer which would still have "work to do", the way the unexamined belief that "I'm too optimistic about the bears" really has work to be done).
What I'm curious about is if there's either a clearly-specifiable criteria for which types of beliefs or hypotheses are workable and which are "too meta to work", or there's some kind of theorem guaranteeing that starting out with "inconsistent" beliefs -- in the sense of "meta-hypotheses" like with da bears -- you can apply this algorithm to process evidence and over time you'll converge on beliefs that're at least more consistent than you started with.
It's hard to say much more without getting formal and I'm out of time for now; since I'm mainly concerned with informal use of "bayesian" metaphors it's not hugely critical to formalize this stuff but later I could give it a proper whack.
That's definitely an interesting space. I think the highly principled side of the Bayesian boat would state that meta-hypotheses are tied into your prior on model building information (Pr(I), etc) and that it needs to be updated alongside everything else. So now if your hypothesis' posterior becomes something like Pr(H, theta, I) the whole business needs to be updated and will include all the meta level intellectual rigor. At this point I feel like I might be walking into the space of Structural Causal Modeling and I'm not too well versed there at all.
In the informal setting though you're only ever likely to be trying to "update" one belief at a time, so, yeah, it definitely requires intellectual care to make sure to follow dependencies. Worse though, is that it should be possible to two have codependent estimations and if you aren't aware of that codependency you won't ever be able to get along.
I think that's all interesting, but I'm not sure it applies to informal situations as well as one might hope. Frequently, Bayesian techniques are only used informally in conjunction with strong rationalist heuristics which help to build these reductionist hierarchies of effects and then allow for clear(er) methodology to find an accurate answer.
Few people thinking carefully and rationally would be willing to bet on their beliefs so long as they know that thy have an outstanding miscalibration. That's why scientists, good scientists anyway, will so often preclude things with disclaimers. They want you to be aware of whatever biases they can before you start to judge their opinions.
Now it seems you are objecting to the fact that your initial state of mind affects your conclusion. But it is unavoidable that there is no way to make sense of observation except through the lens of prior beliefs. You can be explicit about it and draw conclusions with a correct methodology about it as in Bayesian statistics. You can be implicit about it and draw conclusions with an incorrect methodology as in frequentist statistics. (See my other post in this thread for an explanation of why frequentist methodology is incorrect.)
Let me offer a simple example. Suppose a pregnant woman you know gives birth to a boy. Is that evidence that babies are more likely to be boys than girls? Obviously it is. Is it strong evidence? Obviously not. Should we upon observing that conclude that boys are more likely than girls? Obviously not.
Now suppose that you have no prior knowledge other than we see about similar numbers of boys and girls. I know that of the last 100,000 babies born in the USA, 51,157 are boys. Suppose we are both told that at the local hospital, 92 of the last 200 babies born were boys. I submit that we both will and should wind up with different conclusions about the relative likelyhood of boys and girls. Why? Because different prior knowledge lead to different prior beliefs, and those prior beliefs when modified by identical evidence lead to different conclusions.