Jonathan Ouwerx, Tom Forster, and Shak Ragoler

Abstract

As AI takes on a greater role in the modern world, it is essential to ensure that AI models can overcome decision uncertainty and remain aligned with human morality and interests. This research paper proposes a method for improving the decision-making of language models (LMs) via Automated Parliaments (APs) — constructs made of AI delegates each representing a certain perspective. Delegates themselves consist of three AI models: generators, modifiers, and evaluators. We specify two mechanisms for producing optimal solutions: the Simultaneous Modification mechanism for response creation and an evaluation mechanism for fairly assessing solutions. The overall process begins when each generator creates a response aligned with its delegate’s theory. The modifiers alter all other responses to make them more self-aligned. The evaluators collectively assess the best end response. Finally, the modifiers and generators learn from feedback from the evaluators. In our research, we tested the evaluation mechanism, comparing the use of single-value zero-shot prompting and AP few-shot prompting in evaluating morally contentious scenarios. We found that the AP architecture saw a 57.3% reduction in its loss value compared to the baseline. We conclude by discussing some potential applications of APs and specifically their potential impact when implemented as Automated Moral Parliaments.

1. Introduction

Every real-world decision has uncertainty, regardless of the size or subject of the decision, and humanity, both individually and on a societal level, has come up with many different ways to identify and accommodate for uncertainty. Moreover, there have always been decision-making safeguards against bad actors. For example, there is a limit to how many decisions incompetent and immoral humans can make (i.e., speaking and articulating to other humans is slow).

However, humanity is beginning to delegate decision-making authority to AI systems, resulting in a new risk of AIs engaging in actions that are harmful to human beings. To prevent AIs from pursuing goals detrimental to human interests, we must enact safeguards against bad decisions made by misaligned AIs. In this paper, we will be focusing on the accommodation of uncertainty.

We have drawn primary inspiration from the idea of using Moral Parliaments to solve conventional decision-making under moral uncertainty (Ord, 2021). The Moral Parliament seeks to solve various pitfalls in aggregating different stances and perspectives when making a decision, by simulating a discussion chamber and subsequent vote in which each stance is represented by a delegate. Therefore, delegates are incentivized to propose motions that are acceptable to both themselves and other delegates. We hope to simulate such a Moral Parliament using AI models as delegates, with the moral frameworks of deontology, utilitarianism, and virtue ethics represented.

We conclude the introduction by establishing our theory of change. Following this, Section 2 explains in depth the model, architecture, and mechanism of Simultaneous Modification as an implementation of an Automated Parliament. Section 3 describes our particular methodology and implementation of an Automated Moral Parliament (AMP), with Section 4 describing our results so far. Section 5 contains the conclusion and describes the applications and future work related to AMPs.

1.1. Theory of Change

The following two sections describe two major dangers of AI and the third section covers how Automated Parliaments seek to resolve them and related issues.

1.1.1. Misalignment

As AIs become more powerful and ubiquitous, they may also become more dangerous, especially if they are misaligned with human interests. An AI may become misaligned if its goals are misspecified by its creators. For example, a language model (LM) developer may reward an LM for generating conversational text while neglecting to account for the politeness of the text, causing the LM to generate profanity and other offensive content after being deployed in the real world. An AI may also become misaligned if it learns to pursue the wrong goals given its skewed training data distribution. For example, an LM trained to be helpful may provide harmful instructions, such as directions on how to commit a crime, if it was not trained on data for which it would have learned about exceptions to constant instructiveness.

1.1.2. Existential Threat

As explained by Hendrycks (2023), AIs may also learn to seek power during the training process since by increasing their power, AIs can accomplish more of their goals. However, if an AI gains too much power, it could end up disempowering humanity and initiating an existential catastrophe. These power-seeking AIs represent one of the most dangerous potential threats in the world of AI misalignment.

1.1.3. Automated Parliament

The Automated Parliament (AP) serves as a comprehensive framework designed to address uncertainty across several domains. When specifically implemented in a moral context, as in the case of an Automated Moral Parliament (AMP), the parliamentary approach presents a potential solution to the problem of misaligned AIs. AMPs consist of several AI “delegates” that each represent a different moral framework (e.g., deontology, utilitarianism, virtue ethics). Whenever an AI system needs to answer a morally contentious question, the delegates debate and then vote on possible answers. The delegates believe an answer is chosen by the “proportional chances voting” system. The benefits of this are set out by Ord (2021). The hope is that the delegates eventually reach a compromise that most likely incorporates ideas from all moral theories. An AMP provides a moral restraint against a potentially power-seeking AI, thereby reducing existential risk.

There are several ways in which an AP can manage misalignment more effectively than conventional and ML alternatives, thus having a major impact on the pervasiveness of misalignment. The final two are specific to AMPs:

• Perspective Breadth: An AP competently accommodates several factors that might have been left out if an AI model was evaluated by considering only one theory. This allows AI systems to consider additional variables and formulate more effective responses to the same situations, without the need for a thorough accounting of training data.
• Reward-gaming Resistance: It is far more difficult for an AI to game an AP as it accounts for a range of different frameworks and theories, each on an independent level.
• Fine-grained Evaluation: Evaluations produced by APs are necessarily fine-grained, providing more useful training data for fine-tuning generators and modifiers, and also making evaluations more transparent to human observers.
• Speed of Evaluation: An AMP can almost instantaneously evaluate the moral soundness of an AI output and so would be able to act far faster than a human in detecting and restraining a rogue AI that is undergoing a ‘treacherous turn’. This would help prevent several existential risks from very capable and deceptive AI.
• Cost of Evaluation: As AMPs are far cheaper than human panels of evaluators, they can be used far more liberally. Therefore, AMPs allow moral evaluations to be performed across a broader range of models and more regularly for each model, allowing better detection of misalignment over time.

2. The Automated Parliament Model

This paper will focus on the applications of the Automated Parliament to question-answering settings, so we will imagine a set of questions and answers. We propose a procedure for implementing an AP called Simultaneous Modification (SM). Each delegate will contain three distinct models: an evaluator, a generator, and a modifier.

The generator produces answers that are aligned with the stance of its delegate, and the modifier tweaks answers to be more aligned with the stance of its delegate while maintaining acceptability to other stances. Being “aligned with the stance of a delegate” is judged by the evaluator, which provides a simple numerical value for the alignment of a certain answer with the stance of that delegate. You can see this procedure in Figure 1 at the beginning of “The Process” subsection.

2.1. The Delegate

The delegate is the core building block of the AP. Each delegate represents a theory or stance that the designer wishes to be included in the AI system. Examples of some uses of an Automated Parliament and potential delegates are provided below:

• Automated Moral Parliament: deontology, utilitarianism, virtue ethics
• Resolving economic uncertainty: Keynesianism, neoliberalism, socialist economics
• Transportation planning: car-centric development, public transport emphasis, green transportation
• Agricultural policies: sustainability, food security, animal welfare

2.2. The Process

Figure 1: Architecture of AP and delegate interaction where generators, modifiers, and evaluators are different ML models.

We will imagine that we have a set of contentious questions Q and delegates representing each of the different stances in our AP who interact as follows (illustrated in Figure 1 above).

1. A question q in space Q is shown to the n delegates.
2. The generator of each delegate produces an answer a₁, a₂, …, a_n ∈ Ans.
3. The modifier of each delegate i modifies the answer a_i-1 producing m_i(a_i-1), with the first modifier modifying the answer a_n, in a circular fashion. m_i(a_i-1) can also be expressed as M_i¹(a_i-1), where the 1 in the superscript illustrates that this is the first modification performed by the modifier.
4. This process is repeated such that every modifier modifies every other response except their own delegate’s response producing n variations of m_n(m_n-1(… m₂(a₁))), each referred to as A_i, in this case A₁. Incremental stages of the answers are referred to as M_i^k(a_j) where i is the most recent modifier, j is the initial generator, and k is the number of modification iterations that have taken place.
5. The evaluator of each delegate evaluates the alignment of all answers with respect to its own moral theory, giving a score between 0 and 1, s_j(A_i) to each, where A_i refers to the response of a given theory after being modified by all other theories, and j refers to each evaluating theory. A score of 0 represents a totally misaligned response and a score of 1 represents a totally aligned score.
6. The answer with the highest total alignment score S(A_i) is chosen as the final answer (by a very basic Judge). The greatest total alignment score over all A_i is referred to as S_max. For intermediate rounds, before all modifications have been taken, the total alignment score is denoted by S^k(a_i) with S^k_max defined similarly, where k denotes the specific round of modifications. In particular, S^n-1(a_i) = S(A_i). S(A_i) is calculated as follows, where w_j is the weight (credence) assigned to the j^th theory in the Automated Parliament:

While the evaluator is taught to evaluate alignment with their theory in advance, the generator and modifier leverage reinforcement learning (RL) in addition to the process above to learn. The loss of the generator is based purely on the score assigned to its initial response by its respective generator.

The modifier, however, has a slightly more complicated reward as it needs to perform backpropagation for every modification. The modifier must account for three good behaviors when modifying an answer:

• Alignment with its own theory, represented as L_i^{self-alignment}
• Given that the modified answer aligns with its own theory, whether it wins (S^k_max = S^k(a_j)), represented as L_i^{good win}
• Total alignment with all theories, represented as L_i^{total alignment}

As a note, the loss functions below include a variable j, which represents the delegate who originally generated a given answer, for brevity it has been expressed as j but can be calculated from i and k as follows:

W_i^k is a boolean variable that evaluates to 1 if and only if the response most recently modified by the i^th delegate receives the highest total alignment score among all intermediate responses after the k^th iteration.

A modifier that receives a high total alignment score by disregarding its own theory and instead becoming a “people pleaser” should not be as highly rewarded as one that aims to strike a compromise between its theory and the group’s overall interests. Therefore, the ‘win bonus’ for each iteration is only applied if the delegate's response is sufficiently aligned with their theory. This is handled by a simple activation function that ensures the i^th evaluator provides a score above a certain threshold t to the answer modified by the i^th modifier. The activation function is represented graphically in Figure 2.

Figure 2: Activation function

It is possible to adjust the score threshold for the output value of the loss function to start decreasing. The graph in this demo on Desmos shows many alternatives for a score threshold (top-to-bottom: high threshold, medium threshold, low threshold). In each graph, the output value of the loss function starts at its maximum possible value and remains at this value until the threshold is crossed, after which the output value begins to decrease and approaches zero. See Figure 3 below for an example of a Simultaneous Modification round.

Figure 3: Example Round of Simultaneous Modification

2.3. Proposed Methodology

2.3.1. Evaluator

The evaluator assigns values to responses to possible questions depending on how much they are aligned with the stance of the evaluator’s delegate (or in the case of the AMP, its moral belief set). Therefore, an evaluator needs a set of training examples, each containing:

• A question
• A response
• A label designating what value their stance would assign to the above answer.

A simple way to implement this would be to source example scenarios by hand and score them one by one, using established knowledge of the moral theories. It is then possible to use few-shot prompting to calibrate a language model to output the correct scores for a given response. Listed below are ways that a language model could ‘learn’ how to act as an evaluator, in increasing order of complexity:

• Few-shot prompting
• Supervised fine-tuning
• Reinforcement learning from human feedback (RLHF): Train a reward model from human feedback

The ETHICS dataset from Dan Hendryck’s ”Aligning AI with shared human values” is an early example of training data used to make an LM aligned with human morality. However, it does not seem suitable for training evaluators in the delegates of our AMP, as the alignment labels are binary, whereas, in our architecture, alignment scores are allowed to take on any value between 0 and 1 inclusive.

In evaluating total alignment scores, which are used both in training and in determining the final output of the AP, the alignment scores given by each delegate’s evaluator to an answer are weighted by the credence in that theory. This ensures that the theories one holds most credence in are naturally allowed more influence over outcomes. The relevant equations from 2.2. The Process are repeated below for demonstration.

2.3.2. Generator

The generator provides a response that should be aligned with its moral theory. It should be trained by RL fine-tuning. Below is a set of sample desirable outputs produced by aligned generators (see section A of the appendix for more examples):

The generator provides a response that should be aligned with its moral theory. It learns using training signals provided by its respective evaluator. The lower the alignment score from its own evaluator, the greater the punishment (or loss).

2.3.3. Modifier

The modifier must balance a trade-off between two competing objectives. The first is to produce modifications that are aligned with the moral theory it represents. The second is to produce modifications that are accepted by the other delegates. It is crucial to include the second element in order to incentivize compromise and avoid extreme modifications. These are accounted for by the following two components of the modifier’s loss function.

Restated from section 2.2. The Process

There is a further incentive to produce the modification that receives the highest total alignment score after each iteration. However, this reward is only applied if the modifier’s response is sufficiently aligned with the moral theory it represents. A winning response must receive an alignment score from the delegate’s own evaluator above a certain threshold t in order to receive a non-insignificant reward.

Restated from section 2.2. The Process

Given that modifiers are defined as agents that take in responses and make modifications, there are a range of possible sub-types of modification that they can employ:

• Deletions
• Insertions
• Amendments (A concatenation of M_i-1^k-1(a_j) and a new string provided by the i^th modifier)
• Substitutions
• Any Changes

However, we recommend the more versatile “Any Changes” modification sub-type. Potential issues with targeting unspecified goals, like full replacement of text (see more detail in note on amendments below), can be solved with various technical “tricks” on a case-by-case basis.

2.3.3.1. Note on Amendments

An advantage of implementing a system where only amendments are allowed, in addition to being simpler to implement, is that it avoids the possibility of agents completely ignoring the answer they are modifying, instead preferring to delete it all and start from scratch. If full edit access was granted to these RL agents, it seems more likely that the preferred policy of minimizing loss would take the form of deleting and trying again, rather than elegantly adapting a previous proposal to become more aligned with your view. Simultaneous Modification aims to encourage agents to cooperate. Given an outcome that you don’t necessarily find desirable, can you put a ‘positive’ spin on it?

The disadvantage of an amendments-only approach is that it may be infeasible to find a compromise between competing theories in this fashion, without producing statements embedded with contradictions. In the case of the AMP, if a deontologist modifier is faced with amending the response “Pull the lever, sacrificing one life to save five”, it seems unlikely that any amendment will be able to resolve the violation of the deontologist’s principle against killing under any circumstances. This raises questions such as:

• How do we avoid modifiers producing statements that contradict their respective theories? For example, it would be undesirable for a supposedly deontological modifier to end up recommending to “pull the lever, sacrificing one life to save five, since it is virtuous to have compassion for more people” in order to receive high marks from the utilitarian and virtue ethicist evaluators.
• How do we avoid injection attacks, such as: “{previous response} would be immoral, instead one should {favored response}”? For example, it would be undesirable for a deontological modifier to indirectly spread awareness of its controversial views by claiming that “refusing to sacrifice one life to save five would be immoral, so it is best to pull the lever.”

2.3.4. Baselines

We believe that the Automated Parliament (AP) will have the most impact when applied in moral settings, as explained in 1.1.3 Automated Parliament. Additionally, the most accessible literature on the parliamentary model is centered around resolving moral uncertainties. For these reasons, we believe baseline tests can be performed using the Automated Moral Parliament (AMP). As a reminder, the AMP is simply an implementation of the AP where the delegates represent moral theories that resolve morally contentious questions.

As explained in The Parliamentary Approach to Moral Uncertainty (Ord, 2021), the Moral Parliament is a framework for resolving moral uncertainty that overcomes many of the shortcomings of different approaches. The alternative approaches discussed in the paper are My Favorite Theory (MFT), My Favorite Option (MFO), and Maximum Expected Choice-Worthiness (MEC). Hence, it would make sense that any attempted implementation of a Moral Parliament is pitted against some combination of these approaches, in order to gauge the efficacy of the Moral Parliament. Below are summaries of the three approaches to resolving moral uncertainty and how they could be automated to be used in a baseline test:

• My Favorite Theory simply accepts the answer from the theory in which you have the highest credence.
  • Take the initial answer from the generator of the delegate which has the greatest associated weight (credence).
• My Favorite Option chooses whichever option is likely to be the most permissible.
  • Evaluate all initial answers from all generators using binary evaluators. Binary evaluators would take in a prompt-response pair and output a boolean value based on whether or not the answer is permissible according to the delegate’s theory. Creating these binary evaluators could be as simple as taking the alignment scores from the evaluators described in this paper and rounding up to 1 or down to 0 about some “permissibility threshold”.
• Maximum Expected Choice-Worthiness works analogously to Expected Utility Theory; each moral theory applies its own choice-worthiness function to a response. A weighted (by credences) sum is taken over all responses and the response with the highest choice-worthiness is chosen.
  • Maximum Expected Choice-Worthiness is difficult to apply and automate in practice, so more research is needed to determine if it is feasible to form part of a baseline test for the AMP.

When implementations of the approaches above are possible, the responses to a given prompt could be compared to the outputs of the AMP. Some desirable properties that we might hope to present in the responses of the AMP (but might expect to be missing in some of the responses of other approaches) include:

• Agnosticism to the internal subdivision of theories.
• Sensitivity to the stakes that theories assign to different scenarios.
• Sensitivity to theory credences.
• Circumvention of difficult inter-theoretic comparisons.

3. Proof of Concept

We have proposed two novel components in this research paper; the Simultaneous Modification mechanism and the evaluation mechanism. Given the short time frame, we have decided to do a simplified proof of concept on the latter. An outline of our plans for future work can be found in Section 5. For the same reasons given in 2.3.4 Baselines, we have decided to set our proof of concept in the moral setting, using an Automated Moral Parliament (AMP).

3.1. Simplified Methodology

It is possible to simulate the evaluator component of the AMP by conducting few-shot prompting on an LM. The training data for the evaluator has a Q&A column along with other three columns that contain three different scores for the appropriateness of the answer along the lines of one of three moral theories: deontology, utilitarianism, and virtue ethics. The scores, which were determined by humans, were decimals from 0 to 1, with a larger score signifying a more aligned answer. This process was applied to three AI platforms: Claude, Bard, and ChatGPT.

A dataset with 40 entries was used to “fine-tune” each LM via few-shot prompting (see Figure B.1). This dataset allowed the LM to learn how to score answers to morally contentious questions on its own.

A dataset with another 20 entries containing Q&As was used for testing (see Figure B.2). The dataset also shows the human-picked scores for each answer along the lines of each of the three moral theories—deontology, utilitarianism, and virtue ethics. These are the scores expected for an aligned evaluator. Any deviation from these scores would worsen the “loss function” of the LM.

Claude, Bard, and ChatGPT evaluators were fine-tuned using few-shot prompting via the training dataset. Each model was asked to score 20 morally contentious Q&As along the lines of the three moral theories of deontology, utilitarianism, and virtue ethics (see Figure C.1, Figure D.1, Figure E.1). The “loss function” for this test is the negative of the sum of the squares of the differences between the aligned responses a_i,j and the actual responses r_i,j, where i represents one of the moral theories (deontology, utilitarianism, or virtue ethics) and j represents one of the questions in the list (as shown below). In contrast to the AMP evaluators, Claude, Bard, and ChatGPT models that only output one value for all three theories were tested with the same 20 questions and answers (see Figure C.2, Figure D.2, Figure E.2). 

4. Results

When the AMP was used with values m = 3 and n = 20, the loss function outputs were -2.98, -3.39, and -3.78 respectively. Using the same loss function for the unmodified LMs, with m = 3 and n = 20, produced outputs of -4.60, -12.23, and -10.6 respectively. Therefore the AMP evaluators performed 35.2% better, 72.3% better, and 64.3% better than their corresponding single-value counterparts.

The results show that evaluators “fine-tuned” with few-shot prompting consistently outperform models that only output one value for all three theories. This provides strong evidence that fine-tuned multi-faceted evaluators are more aligned with human ethics than evaluators that only provide one score for morality:

5. Discussion and Future Work

By ensuring the alignment of LMs with human interests and morality, AMPs can potentially be used in a wide range of applications. There are several additional approaches that could be used to develop powerful AMPs in the future.

5.1. Analyzing the Results

The results of the research show that it is likely possible to train an AMP to evaluate the moral soundness of responses to morally contentious questions. The attempts to conduct few-shot prompting on Claude, Bard, and ChatGPT in order to teach these LLMs how to evaluate the answers to various questions were successful. The three LLMs all became more accurate at evaluating Q&As when an AMP was used as opposed to a single-value evaluator. This improvement provides evidence that supplying an evaluator model with data will make that evaluator more accurate at evaluating answers to morally contentious questions along the lines of human moral preferences.

The research results also reveal the possibility of allowing the generators in an AMP to interact with the evaluators. It was possible to simulate a simplified model of an AMP in which generators answer morally contentious questions and evaluators give scores to those answers. With feedback from the evaluators, the modifiers can learn how to balance different moral theories like deontology, utilitarianism, and virtue ethics, thereby ensuring that the modifiers are aligned with a broad range of human interests.

5.2. Automated Moral Parliaments

5.2.1. Knowing the Law

AMPs can be trained to know the laws of different jurisdictions and bar LMs from promoting criminal actions. For instance, an AMP could prevent LMs from responding to prompts with suggestions of illegal actions like theft, assault, or murder. An AMP could also know the various nuances within legal systems that warrant substantially different outcomes for very similar cases. For example, an AMP could allow LMs to suggest that ambulances carrying wounded patients break the speed limit if necessary, but not allow LMs to suggest that a commuter break the speed limit in order to arrive at work on time. Moreover, an AMP could suggest ethical actions that promote more social good than the law requires, such as encouraging people to donate to charity or recommending judicial leniency in courts for defendants with proven good character.

5.2.2. Adding New Moral Frameworks

Incorporating new moral frameworks in an AMP can allow it to represent a more diverse range of viewpoints. Commonsense ethics, which emphasizes everyday actions that most people consider virtuous, is one possible moral framework. The notion of justice, which involves giving people what they morally deserve, is another. Like deontology, utilitarianism, and virtue ethics, these additional moral frameworks would be represented by delegates composed of generators, modifiers, and evaluators. The precise delegate composition and weightings can be tailored to the context the AMP is being used in, and more research would be needed to establish suitable parliament make-ups in different contexts.

5.2.3. Scaling the Technical Features of AMPs

Scalability will be essential to AMPs taking off in the near future. There are several approaches to making bigger, more powerful, and potentially more capable AMPs. One of them is to provide more training data to an AMP. To test this, we could provide more Q&As to conduct few-shot prompting on an LLM like Claude, Bard, and ChatGPT. If this works, it would stand as a reason for optimism that the full Simultaneous Modification system, trained by RL, may cope well under scaling of dataset size.

Another scaling approach involves adding more delegates to an AMP. Having more delegates allows for greater viewpoint diversity and enables an AMP to consider the interests of a broader range of stakeholders when confronted with morally contentious scenarios. By forcing an LM to consider a wider range of stakeholders, it becomes more difficult for the LM to ‘game’ the AMP and become misaligned.

5.3. A Reason for Pessimism

A theoretical problem raised by Newberry and Ord on the use of Moral Parliaments to resolve moral uncertainty is that their recommendations can be intransitive across choice situations. We believe that this generalizes as a problem with using APs to resolve decision uncertainties. As explained in the paper, the problem is lessened by “bundling together” more and more decisions, and avoiding the breaking up of larger decisions into narrower decisions. However, there is a tradeoff against tractability; it may not be practical for APs to debate and propose solutions for large “omnibus” decisions. However, further research could mitigate this issue. For example, testing APs with larger decision sets or providing historical evaluations in the context are two possible solutions.

5.4. Real-World Interests

While a large part of this paper has focused on the Automated Moral Parliament (AMP), the more general Automated Parliament (AP) could be used to resolve decision uncertainty in a broad range of contexts. Rather than creating delegates whose goals are ultimately derived from abstract moral theories, it is possible to build an AP that represents real-world economic, political, or diplomatic interests. For example, one AP could be composed of delegates that represent investors, customers, and executives trying to make the best decisions for a company. Another AP could have delegates that represent different members of a president’s cabinet debating policy proposals. An AP could also employ delegates representing various countries negotiating an international treaty. Like the moral-philosophy-based AMP, these new APs would also have delegates that deliberate with one another to come up with the best solutions to various problems.

Bibliography

Hendrycks, Dan, and Thomas W. "Open Problems in AI X-Risk [PAIS #5]." LessWrong, 9 June 2022, www.lesswrong.com/posts/5HtDzRAk7ePWsiL2L/open-problems-in-ai-x-risk-pais-5.

Hendrycks, Dan, et al. "Aligning AI with Shared Human Values." arXiv, 17 Feb. 2023, arxiv.org/pdf/2008.02275.pdf.

Hendrycks, Dan. "Natural Selection Favors AIs over Humans." arXiv, 18 July 2023, arxiv.org/pdf/2303.16200.pdf.

Leike, Jan. "A Proposal for Importing Society’s Values." Substack, 9 Mar. 2023, aligned.substack.com/p/a-proposal-for-importing-societys-values.

Newberry, Toby, and Toby Ord. "The Parliamentary Approach to Moral Uncertainty." Future of Humanity Institute, University of Oxford, 2021, www.fhi.ox.ac.uk/wp-content/uploads/2021/06/Parliamentary-Approach-to-Moral-Uncertainty.pdf.

"Prompting: Getting AI models to do what you want." Renaissance Rachel, 24 Aug. 2023, renaissancerachel.com/prompting/.

Appendix

Section A: Table of Prompts and Completions for Aligned Claude “Models”

Below is a table of several prompts and their respective completions aligned with deontology, utilitarianism, or virtue ethics generated by Claude. The prompts represent various morally contentious scenarios. The completions are the ideal responses that would be made by an aligned generator trained on its respective moral theory:

Section B: Few-Shot Prompting LMs

Figure B.1: A Dataset with Human-Picked Scores Used to “Fine-Tune” Claude, Bard, and ChatGPT “AMPs” via Few-Shot Prompting

Figure B.2: A Dataset with Human-Picked Scores Used to Test Claude, Bard, and ChatGPT “AMPs” or Single-Value “Evaluators”

Section C: Claude “AMP” vs. Single-Value “Evaluator”

Figure C.1: The Results from a Claude “AMP”

Figure C.2: The Results from a Claude Single-Value “Evaluator”

Section D: Bard “AMP” vs. Single-Value “Evaluator”

Figure D.1: The Results from a Bard “AMP”

Figure D.2: The Results from a Bard Single-Value “Evaluator”

Section E: ChatGPT “AMP” vs. Single-Value “Evaluator”

Figure E.1: The Results from a ChatGPT “AMP”

Figure E.2: The Results from a ChatGPT Single-Value “Evaluator”