← All field notes

Research Watch · Alignment · Safety

Constitutional AI: aligning a model against a written policy instead of a crowd

Anthropic's Constitutional AI research replaces large volumes of human harm-labeling with a written set of principles the model uses to critique, revise, and eventually judge its own outputs.

AI-assisted / research-based

This field note was drafted with AI assistance and synthesizes publicly available research papers and disclosed industry practice on an emerging AI technique. It is not based on confidential deployment data, is not investment, legal, medical, or security advice, and every primary claim links directly to its source so you can verify it yourself.

Standard reinforcement learning from human feedback needs large volumes of human judgments about which of two responses is better — a slow, expensive process to scale, and one where the actual standard being enforced is implicit in thousands of individual labeling decisions rather than written down anywhere. Anthropic's Constitutional AI research asks whether an explicit, written document can do a meaningful share of that work instead.

When the standard lives in a document instead of a label set, it can be read, audited, and changed — not just inferred from examples.

Two phases, one shared document

Constitutional AI: Harmlessness from AI Feedback ↗, by Bai and colleagues at Anthropic, trains a harmless assistant through self-improvement guided entirely by a list of written principles — the "constitution" — with no human labels identifying harmful outputs at either stage.

Process / AI-supervised alignment

One written document replaces most human labels

FIG 01 - MOTION
Constitutional AI two-phase training loop A supervised phase has the model critique and revise its own responses against a written constitution, then a reinforcement learning phase has the model compare response pairs against the same constitution to train a preference model, which supplies the reward signal for reinforcement learning from AI feedback. SUPERVISED PHASERL PHASE (RLAIF) SAMPLEInitial responsefrom base model CRITIQUESelf-critiqueagainst written constitution REVISE + FINE-TUNESL-CAI modelon self-revised examples COMPAREAI preference labelssame constitution, two samples Preference model → RL reward no human harm labels required at either stage OVERSIGHTthe only human input is the written constitution itself — reviewable and revisable
Both training phases derive their signal from the same constitution — a supervised critique-and-revise pass, then an RL phase where the model's own preference judgments train the reward signal.

In the supervised phase, the model samples an initial response, is prompted to critique that response against the constitution, and then revises it accordingly; the model is then fine-tuned on these self-critiqued, self-revised examples. In the reinforcement learning phase, the model samples response pairs, is prompted to judge which of the two better satisfies the constitution, and this AI-generated preference dataset trains a preference model. That preference model then supplies the reward signal for reinforcement learning — a process the paper names "RL from AI Feedback," or RLAIF, in direct contrast to RLHF's human-labeled preferences.

The written constitution is the actual lever

The mechanism that makes this auditable is specific: both phases route their judgment through the same constitution, so the model's behavior is anchored to a document a person can read end to end, not to a pattern implicit across thousands of independent human comparisons. Changing the constitution changes downstream behavior in a traceable way; changing an RLHF policy typically means re-collecting a large human preference dataset from scratch.

The paper also reports a secondary benefit: both the supervised and RL methods can incorporate chain-of-thought-style reasoning into the critique and preference-judgment steps, which the authors report improves both the human-judged quality of the resulting behavior and the transparency of how the AI reached its judgment — the critique itself becomes inspectable, not just the final revised output.

Comparison / alignment standards

Where does the standard actually live?

FIG 02
A written constitution versus an implicit label-based standard Standard RLHF's behavioral standard exists only implicitly, spread across thousands of individual human preference labels, while Constitutional AI's standard is a single reviewable document that both training phases refer back to. STANDARD RLHFStandard is implicitspread across thousands ofindependent human labels— not written down anywhere CONSTITUTIONAL AIStandard is one documentread end to end by a personboth training phases reference— reviewable and revisable AUDITchanging behavior means re-collecting labels in one approach, editing a document in the other
The practical difference between RLHF and Constitutional AI is where the behavioral standard is recorded — implicitly across many labels, or explicitly in one document a person can audit.

What a constitution cannot do by itself

A written constitution is only as good as its authors' foresight, and self-critique against it inherits a structural limit: a model judging its own output against a document is still using its own capabilities to evaluate itself, which can miss failure modes the constitution's authors did not anticipate and the model itself cannot recognize as violations. This method reduces dependence on large-scale human labeling; it does not remove the need for human oversight of the constitution's content or for evaluation against real user interactions after deployment.

What this changes in production

Policy-anchored assistants in regulated or brand-sensitive contexts

Where behavior needs to trace back to an explicit, reviewable standard rather than an opaque label set — for audit, for regulatory conversations, or simply for internal accountability — a written constitution gives compliance and safety teams a document to review directly, rather than reverse-engineering behavior from training data.

Reducing human-labeling bottlenecks at scale

Organizations operating at a scale where exhaustive human review of every training comparison is impractical can use this method to generate a much larger volume of preference data than human labeling capacity alone would allow, while keeping the actual behavioral standard human-authored and reviewable.

Faster iteration on safety and tone policy

Updating a written constitution and regenerating training signal from it is typically faster than re-collecting a large human preference dataset reflecting a policy change — useful when a team needs to adjust behavior in response to a newly identified edge case without a full RLHF data-collection cycle.

A production checklist

Before relying on constitutional-AI-style training for a production system, the team should be able to answer:

  • Has the constitution itself been reviewed by the people accountable for the model's behavior, not just implemented as a technical artifact?
  • What happens when the model's self-critique fails to recognize a genuine violation the constitution was meant to prevent?
  • Is post-deployment behavior evaluated against real user interactions, or only against the model's own constitutional self-consistency?
  • How and by whom is the constitution updated when a new failure mode is identified in production?
  • Does the RL phase's preference model get validated against any human-labeled comparisons, or only against its own AI-generated preferences?
  • Is there a documented difference between what the constitution says and what the deployed model actually does, and who is responsible for closing that gap?

Research referenced

Continue readingReturn to field notes →