Effective Altruism Forum
EA Forum

GIVING SEASON 2024
Hide table of contents
by David Johnston
1 min read2 answers 0

4

Forecasting
Frontpage

Platforms like Metaculus aggregate crowd forecasts on unstructured collections of questions. Individual forecasters will often consider structured models relating a number of sub-questions in order to answer a single forecasting question. I'm interested in any work trying to help multiple people jointly produce structured models. I'm interested in cases that could be described as "group collaboration" as well as cases that could be described as "crowd aggregation". I'm aware that Ought is working on a similar question and I think Guesstimate was motivated by questions along these lines, and I'd like to know if there's anyone else who is or has been working on something like it.

4

0
0

Reactions

0
0
New Answer
New Comment

2 Answers sorted by
Top

Davidmanheim

5
0
0

There is a large academic field focused on related work, but I will point to our work on modeling transformative AI risk. There is also work done more informally by groups of forecasters on Metaculus, for example, using shared google docs and  collaborative meetings, and there is work on Delphi techniques for forecasting question development.

In addition, there is a large literature on eliciting Bayesian networks from groups of experts, often by using both data and expert input, and to perform similar tasks. I'm unsure what you are working on, but I'm happy to talk more about this and provide references and suggestions for who else to contact.

David Johnston7
0
0

Thanks. I'm not working on anything at the moment, just curious about what has been done in the area. Did you consider other approaches to mapping out key hypotheses and cruxes for MTAIR? Do you have an idea of what advantages  and disadvantages you expect the big Bayesian network to have compared to other approaches? Have you found it to be better or worse in any particular ways?

A particular question I'm curious about: have you found the big Bayesian network approach is helpful in terms of decomposing the problem into sub-problems and efficiently allocating effort to subproblems ?

Reply
3
Davidmanheim
We looked at the options, and choose Analytica largely because of a paucity of other good options, and my familiarity with it. Having spoken with them in the past, and then specifically for this project, I also think that the fact that the company which makes it is happy to be supportive of our work is a big potential advantage. Why Not Large BNs? 1. BNs are expensive to elicit. (You need output∗∏inputi values elicited per node, where inputi and output are the number of discrete levels of each.) They also have relatively little flexibility due to needing those discrete buckets. There are clever ways around this, but they are complicated, and outside the specific expertise of our group.  2. BNs assume that every node is a single value, which may or may not make sense. Most software for BNs don't have great ways to do visual communication of clusters, and AFAIK, none have a way to leave parts undefined until later. You also need to strongly assert conditional independence, and if that assumption is broken, you need to redo large parts of the network. 3. The way we actually did this shares most of the advantages in terms of decomposition and splitting subproblems, though removing duplication /overlap is still tricky.
1
David Johnston
Just to clarify: my understanding was that the MTAIR graph will eventually be extended with conditional probability estimates, so the whole model will define a probability distribution with conditional independences compatible with the underlying graph. This would make it a Bayesian Network in my eyes. However, it seems that we disagree on at least one of thing here! Is the Analytica approach more robust to missing arrows not corresponding to conditional independences than a Bayesian network? If so, I'd be curious to hear a simplified explanation for why this is so.
2
Davidmanheim
Analytica allows you to define algebraic or other relationships between nodes, which can be real-valued, and have more complex relationships - but it can't propagate evidence without explicit directional dependence. That allows more flexibility - the nodes don't need to be conditionally independent, for example, and can be indexed to different viewpoints. This also means that it can't easily be used for lots of things that we use BNs for, since the algorithms used are really limited.

Davidmanheim

3
0
0

QURI is working on related things. (Run mostly by Ozzie Gooen, who made Guesstimate.)

More from David Johnston
View more
Curated and popular this week
Relevant opportunities
View more