Disclaimer: The story should not be seen as any kind of prediction. That AGI will be deployed in some way similar to the ritual in the story is very unlikely (see Conjunction Fallacy). Views presented in the story are not necessarily associated with the author's views and are partially controversial.

If you have any feedback, I would be happy if you would share it in the comments! :)

The Point of no return

This would be it, Magnus thought.

As the chief productivity officer of the United Worlds Project of Summoning a Good God, often just called "the project", he was sitting together with the director, some executive officers of subprojects, some senior researchers and a few other people in a large room.

They were the most important people in the world. Actually maybe the most important in vast areas of the observable universe of all time, including the future.

They all looked wrecked, like they have not slept for at least a week. This probably wasn’t exactly the case, but it was close. He probably didn't look better either. One person was crying. Magnus could not tell if it was because of exhaustion, relief that all that stress would be over soon, or because of sadness and fear that the project might fail and the future would be dark, or the regret that we haven't done more. Maybe because of all of that. It looked weird and wrong, but the goal hadn't been to maximize long-term productivity, but to maximize productivity until the ritual starts.

They were called together to vote if the ritual should start. The examination of the latest version of the ritual was currently running and the results would be sent to them in a few minutes. The probability that the ritual would start today was quite high, almost certain if the examination would not detect any fixable issues, in which case they would of course fix the issues first. But their estimations said that the estimated utility is greater if they start the ritual now, though they were relatively uncertain. Humanity would probably be entirely different after that day.
 

About 28 years ago, Magnus didn't recall exactly when, he first heard about the idea of summoning a god by creating a ritual so huge that it was able to pull more magic into the ritual, which pulled even more magic into the ritual, and so on, creating a god, growing stronger and stronger very quickly, until there was not much more of the type of magic the god could use to get stronger.

He dismissed the idea. This was just all too crazy, there is no way something like that could work, he thought, as almost everyone else would have thought, if they had heard about the idea that early. Only a tiny handful of people were concerned about the idea back then. They were obviously mad, just like those many other conspiracy theorists that existed.

He had been quite irrational back then and did not even think about that even if it was almost certain that this was bullshit, it would be worth taking a look at to actually be certain, especially if he could not pinpoint what didn’t fit about the argument. About 10 months later, he stumbled across the idea again and this time he looked deeper into the subject. He realized that summoning a god could actually work and that it was even likely that people will do it within a century.

The god would probably not only be able to cure depression, cancer and other illnesses. It would likely be able to cure aging and make us at least as happy, as we were in the happiest moments of our lives. Maybe we would experience things and feelings we could not even have imagined. And most of all, it could increase the amount of sentient and happy beings by a lot. The future would be awesome!

Except if we humans mess it up, which was not that unlikely, as Magnus found out after reading further. It turned out that making the god to be like you want was a notoriously difficult problem, and if we failed, it could convert a significant part of the observable universe into dead material, which could permanently prevent the existence of happy conscious life. It was just such a terribly terrifying thought for him, although he could not even nearly grasp the magnitude of that lost potential with his stupid and scope-insensitive human brain. There were other options, how it could turn out a bit less wrong, and those options didn't seem as bad, but actually it was still magnitudes of utility away from the best case, so it actually still was almost as terrible.
 

It shouldn't be that early, Magnus thought. He should have cared more about making more time, should have tried earlier to slow down the advancement of magical rituals, just like the million other things he should have done better. Admittedly, many things were far more obvious in hindsight than in advance, but still, there were many things he could have done much better, like thinking more about what the most effective thing to do is and coordinating with other people earlier, to do the most good.

The problem was that this was not the only project working on a ritual to create a god. It was the biggest project and the most advanced, but it was very likely that at least one other project, which was advanced enough to start a god-creating ritual. Luckily, they probably made less progress on how to reliably make the god as they wanted, so they hadn’t launched yet. The reason why there were two major projects was a disagreement about which god was wanted. The other project wanted a god that would have human values, where it was not clear how well it would work, because humans had inconsistent utility functions and there were huge disagreements about morality within humanity. Ideally, the god would extrapolate what human values would be if they were very intelligent and coordinated, but it was not known how to do that securely and it was not even known if human values would converge at all. Also, even if human values would converge, it was possible that human values would value human life higher than alien life or conscious circuits.

The project Magnus was working on wanted to create a god that should carefully maximize hedonic utility. Hedonic utilitarianism seemed to be the simplest and most likely theory of universal moral truth, if such a thing existed. And if not, did anything matter at all? Probably not, unless you believed in ethical subjectivism, but Magnus didn’t really understand how that could possibly be right. Though the fact that they hadn’t a truly robust ethical theory still freaked Magnus out sometimes.

Unfortunately, it was not really clear what hedonic utility was, but there was some hope that the god would solve the hard problem of consciousness and figure out where exactly happiness comes from, which could solve that problem. But even if it would not figure that out, it would care for the well-being of beings, which provided enough evidence to assume they are conscious, which would still be quite good. Sadly, there were still a lot of people who were afraid that the god would kill humanity and therefore were hoping the other project would succeed, although the god would only kill humanity to replace it with something that is certainly better.

Magnus could understand the fear of death. He didn't want to die, but he would agree to die if other more happy, intelligent and moral people would be able to live from the resources that Magnus consumed. Fear of death was burned so deeply into the minds of humans by evolution that even many respectable people said they probably would rather not die for the greater good. But unfairly favoring humans against aliens or conscious circuits was just so obviously wrong. How could you accept an utility function that is inconsistent across conscious instances?

More surprising was the fact that many people said that they didn’t always want to be happy. Why wouldn’t you, except for trying it out of curiosity if you never have been sad before? Maybe it was some kind of rationalization because they wanted to live in one of the best possible worlds.
 

The calculations now said that if the checks passed, it would likely be better to start the ritual now instead of later, because otherwise another project could start their ritual first, creating a god that was probably worse than the god Magnus' project would create, so they should not wait much longer. Even if no failure in the ritual design would be discovered during the examination, there was probably something like a 2.5% chance that the ritual would go wrong and produce a god not exactly wanting to promote hedonic utility. This probability was itself extremely uncertain, but not quantifying it wouldn’t make it better. About 2.5%! This was horrifying. The utility benefit of a later and more secure start of the ritual was so unimaginably large.

Magnus felt a cold chill go down his spine. This time, it was even stronger than what he usually felt upon thinking that thought, probably because now, there was nothing left to do to improve it, it was so definite. In that moment, he was happy that, as a human, he had a large scope insensitivity.

Though actually he was quite glad that he was less scope insensitive than most humans. Sure, you would feel the bad things stronger, but you also felt the good things stronger. That was what gave him his burning motivation. The vision that in expectation there would be far more goodness than a decillion happy human lifetimes lived full of love and joy and inner peace. That maybe there could be even some much more amazing feelings that humans could currently experience. Magnus visualized those many many people feeling that pure love and happiness, who had spread throughout the galaxy and further, knowing that among the stars there were so many other beautiful worlds with other happy people. That was not even nearly the most awesome future, but he could not imagine those futures, that were even better, as well.

For a short time, Magnus savored the vibrant feeling of love, energy and inner peace, which was sparked by that beautiful vision.
 

Then an inner voice screamed: "Alarm! Your thoughts have drifted off. Come back to the present, you've got work to do!".

Wow this took quite a long time until he noticed it, far too much time, Magnus thought. How on earth could he have drifted off for so long? He used to be much better at detecting it, but he just got more and more exhausted in the last months.

Now was not the time to regret the past or anticipate the future. He would not also let that incredibly tiny chance to improve the future of the universe pass by. Now was the time to think if there was some better action he could take.

Of course, he thought about that a lot before. So have many other people. The chances that something useful would come to his mind now were very slim, but the expected value of thinking was still much higher than if he would just relax right now. He wouldn’t have been able to relax anyways.

He needed an idea that was so good that it was worth delaying the start of the ritual. Something concrete that could be implemented quickly.

Magnus was not the best expert when it came to the technical details of the ritual. The probability that he would have a brilliant idea to improve the ritual so much that it would be worth not starting the ritual now was almost zero. So instead he should focus on thinking about other ideas.

A stream of ideas went through Magnus' brain, from faking a major accident within the project's labs that would convince others that they had a huge setback to buy more time, to trying to infiltrate the networks of other projects that tried to create a ritual and blowing a lot of stuff up there with nuclear or chemical weapons.

None of the ideas were new or good. Magnus and others had thought those through before and they just didn't seem promising enough.

He tried to think of something new, or at least something that has not really been thought through yet.

No thoughts really came to mind, though he felt his mind searching through concepts without really promoting them to conscious attention.

Maybe it was somehow possible to create a mechanism to destroy the world as soon as the mechanism recognized that a god was created, so that no one dared to launch their ritual. This could buy more time to come to an agreement among humans about what god should be created and enable further progress on how to create a good god. This idea was first suggested a few years ago, but somehow was met with a lot of public opposition, because there was some risk of the world being destroyed because of that, which would be terrible, because it was quite uncertain if there would be an alien species that would ensure that the universe would be filled with happy life. Also, engineering something that could destroy the world was itself really dangerous and difficult and would take too long. Maybe you could create a mechanism that would cause atomic bombs to explode in big cities, but that was not likely to keep people from creating a god, because the negative utility of a billion people dying was quite insignificant in comparison to the utility of having different moral values in a god.

Seriously, that was all he came up with? If Magnus were as intelligent as a god, it was not improbable that he would find a better way of action than the current one!

Shut up and do the impossible!, Magnus thought to himself.

Concepts rushed through his mind, possible futures crystallized out before him, his knowledge and uncertainties became more and more clearly graspable.

It didn't seem like there was much more he could do. It was time to let go and accept that he was not powerful or smart enough to make reality go a better way. He felt like his inner perfectionist was screaming, but Magnus knew it was the truth. Maybe Magnus should focus on what he should do in the scenario that the designed ritual failed some test in the examination, which had a probability of about 6%. But it actually wouldn't change much, they would just try to fix the mistake as quickly as possible and maybe do some very slight other improvements to the ritual in parallel, and then they would be here again.

There was a ring. The results of the examination were there. They read:

No quickly fixable issues found.

Warning!: The following probabilities are highly uncertain, were estimated based on extremely limited knowledge and are controversial. It is, however, been calculated as the mean of many of  the best experts in the field.

Estimated probability that the values of the god would be noticeably different than intended: 2.4%.

Estimated probability that the galaxy and perhaps more space would be made permanently uninhabitable: <0.8%.

Estimated probability of astronomical suffering: <0.15%

Magnus innerly flinched after reading the probabilities. How could his species fail that significantly?

Now it was time to vote. Everyone in the room was to estimate the probability of the statement "Starting the ritual now yields a higher expected utility than waiting" being true. Basically it was mostly about if we would get an amazing idea how to improve the ritual significantly or about how to securely buy more time in the next few weeks. The votes would be weighted with some weighting score of each voter, which was mostly dependent on their forecasting scores and experience in the subject, and if the weighted average probability was above 50%, the ritual would start. They were all excellent forecasters and probably the best group to do that, but it was still highly uncomfortable to draw decisions when there was so much uncertainty.

"If anyone has any relevant information for the probability assessment that other people within the room don't have, please tell that now.", the director said.

The pause stretched. No one said anything.

Then the director continued: "It seems like that is not the case. So then we will continue with the vote. Please make your probability estimation carefully and take your time, if you have not already. When you are finished, you can enter your estimated probability into your voting device."

They were asked in advance to independently make a careful estimation of what the probability would be. Magnus generally had a probability estimate of that ready for a long time, and he often updated it when he got new relevant information or ideas. He had revised his estimation nevertheless and carefully looked over all the evidence he had and he had slightly adjusted his estimation. Should he look over it again? No that was actually quite pointless, he would probably have too much emphasis on some points instead of others and make his estimation worse. Magnus typed 82.2% into his voting device.

He was rather sure that all others also had done their probability assessments in advance and were now just pondering if they should change something slightly. Not that little variations mattered now, it was basically certain that the ritual would launch. There was just no way one of those rational people in the room would estimate a probability of less than 50% given the same information Magnus had.

After a short time, everyone had voted, and the result appeared on a screen: 84.5%. The standard deviation was 4.2%, which was frighteningly high considering that all people were great forecasters and had basically the same information.

The command to start the ritual was sent automatically and would be executed directly.

It was weird. The most important thing of all time was currently happening. Magnus had expected to feel excitement, regret, tension, relief, fear or much more, but instead there was this - void. Maybe void didn’t describe it perfectly, - it was just, - he didn’t know what he felt.

It was over now. His mind seemed not to be able to really grasp it.

Slowly he faded out of his controlled mode that he had in high stress situations. Only now he realized how terribly nervous he was, his heart bumping in his chest so loud and fast he couldn’t ignore it.

Had they just destroyed a large part of the universe? Has it been the right decision to launch? Why hadn’t he done anything else to prevent a ritual launch from being rushed? Why hadn’t he cared more about ensuring that a ritual would only be launched if there was great consensus about the values they would implement in the god?

It was like Magnus’ project defected in some kind of prisoner’s dilemma, where as soon as one defected, the other couldn’t defect anymore. Not that the decision to defect, to launch the ritual, had been irrational. The others could have otherwise defected first, which would have been much worse. But how could he let it come that far? Why couldn’t he and others realize earlier that they were getting into such a situation?

Not that it would have been easy to ensure that no project launch without consensus, and there would have been the problem that the current state of the human species was too stupid to realize what values were right, and then could have decided by vote that a ritual would be started that created a god with worse values, but that should have been preventable somehow.

As his brain slowly began to grasp more clearly what had happened, strong emotions arose in Magnus. It was hard to describe how he felt, a mix of regret, shame, relief. Also a bit of gratitude, that it didn’t end up even worse, but overall his emotional state could not really be put into words.

Of 1000 future worlds, about 975 would be awesome, 15 would be far from awesome, but perhaps still good, 9 would be dead and 1 would be horrible.

Of course, not all of the futures would really exist, but from his current point of view, it was equivalent to think of the probabilities like that, and it was easier to imagine.

Magnus started to notice how exhausted he was. It felt like he collapsed into his chair, although he had already been sitting. He closed his eyes. A tear went down his cheek. 

In what world he would be, once he opened his eyes again?