Prompt 优化自动化

自动化 Prompt 优化能大幅提高效率，减少人工迭代成本。

一、优化方法

1.1 基于梯度的优化

# DSPy 示例
import dspy

class QA(dspy.Signature):
    question = dspy.InputField()
    answer = dspy.OutputField()

class QAModule(dspy.Module):
    def __init__(self):
        super().__init__()
        self.qa = dspy.Predict(QA)
    
    def forward(self, question):
        return self.qa(question=question)

# 自动优化
optimizer = dspy.BootstrapFewShot(metric=accuracy_metric)
optimized_module = optimizer.compile(QAModule(), trainset=trainset)

1.2 基于演化的优化

# 遗传算法优化
def evolve_prompts(population, generations):
    for gen in range(generations):
        # 评估
        scores = [evaluate(p) for p in population]
        
        # 选择
        parents = select_best(population, scores)
        
        # 交叉变异
        children = crossover_and_mutate(parents)
        
        population = parents + children
    
    return get_best(population)

1.3 基于强化学习的优化

# RL 优化
def rl_optimize(prompt, reward_fn, episodes=100):
    for episode in range(episodes):
        # 生成变体
        variants = generate_variants(prompt)
        
        # 评估奖励
        rewards = [reward_fn(v) for v in variants]
        
        # 更新策略
        prompt = update_policy(prompt, variants, rewards)
    
    return prompt

二、DSPy 框架

2.1 核心概念

# Signature（签名）
class Summarize(dspy.Signature):
    document = dspy.InputField()
    summary = dspy.OutputField(desc="简洁的摘要")

# Predictor（预测器）
summarizer = dspy.Predict(Summarize)

# Module（模块）
class SummarizerModule(dspy.Module):
    def forward(self, doc):
        return summarizer(document=doc)

2.2 优化器

# BootstrapFewShot
optimizer = dspy.BootstrapFewShot(
    metric=accuracy_metric,
    max_bootstrapped_demos=4
)

# COPRO
optimizer = dspy.COPRO(
    metric=accuracy_metric,
    depth=3
)

# MIPRO
optimizer = dspy.MIPRO(
    metric=accuracy_metric,
    num_candidates=10
)

2.3 实战案例

# 完整流程
import dspy

# 1. 定义任务
class RAG(dspy.Signature):
    context = dspy.InputField(desc="相关文档")
    question = dspy.InputField(desc="问题")
    answer = dspy.OutputField(desc="答案")

# 2. 创建模块
class RAGModule(dspy.Module):
    def __init__(self):
        super().__init__()
        self.rag = dspy.Predict(RAG)
    
    def forward(self, question, context):
        return self.rag(context=context, question=question)

# 3. 准备数据
trainset = [
    dspy.Example(
        context="文档内容",
        question="问题",
        answer="答案"
    )
]

# 4. 优化
optimizer = dspy.BootstrapFewShot(metric=accuracy)
optimized = optimizer.compile(RAGModule(), trainset=trainset)

三、优化工具

3.1 Promptfoo

# promptfooconfig.yaml
prompts:
  - prompt1.txt
  - prompt2.txt

providers:
  - openai:gpt-4
  - anthropic:claude-3

tests:
  - vars:
      input: "测试输入"
    assert:
      - type: equals
        value: "预期输出"

3.2 LangChain Evaluation

from langchain.evaluation import load_evaluator

evaluator = load_evaluator("qa")

result = evaluator.evaluate_strings(
    prediction=predicted_answer,
    reference=expected_answer,
    input=question
)

print(result["score"])

3.3 Arize Phoenix

import phoenix as px

# 启动 Phoenix
px.launch_app()

# 追踪 Prompt
from phoenix.trace import openai_instrumentor
openai_instrumentor.instrument()

# 查看追踪结果
# http://localhost:6006

四、最佳实践

4.1 优化流程

1. 定义评估指标
2. 准备训练数据
3. 选择优化器
4. 执行优化
5. 验证结果
6. 部署上线

4.2 评估指标

metrics = {
    "accuracy": "准确性",
    "relevance": "相关性",
    "fluency": "流畅度",
    "consistency": "一致性",
    "toxicity": "毒性检测"
}

4.3 持续优化

# 监控线上表现
def monitor_performance():
    while True:
        metrics = collect_metrics()
        
        if metrics["accuracy"] < threshold:
            # 触发重新优化
            retrain_prompt()
        
        sleep(3600)  # 每小时检查

五、总结

Prompt 优化核心要点：

自动化 Prompt 优化是大规模 AI 应用的必备能力，能显著提升开发效率。