Skip to content

feat: model_config 支持 summaries_set(PCOC / scalars)训练期 TensorBoard 监控#556

Open
fooSynaptic wants to merge 2 commits into
alibaba:masterfrom
fooSynaptic:feat/summaries-set-pcoc
Open

feat: model_config 支持 summaries_set(PCOC / scalars)训练期 TensorBoard 监控#556
fooSynaptic wants to merge 2 commits into
alibaba:masterfrom
fooSynaptic:feat/summaries-set-pcoc

Conversation

@fooSynaptic

@fooSynaptic fooSynaptic commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

背景

参考Issue:#532

训练过程中 在 TensorBoard 里直接看到 PCOC(预估 CTR / 真实 CTR),并支持按某个特征取值(例如 c1=1005)查看该子人群上的平均预估。原先 eval_config.metrics_set 无法满足「训练期写 summary + 特征切片」的需求。

设计思路

  1. 配置入口放在 model_config.summaries_set,与模型输出(prediction_dict)同层,而不是 eval_config
  2. 两类 summary,职责分离
    • pcoc:全 batch 的 mean(预估) / mean(标签),不做特征切片;
    • scalars:对 pred_name 指定输出做 mean(预估),可通过 feature_name + feature_value 做样本过滤(不算 PCOC)。
  3. 训练图构建EasyRecEstimatormodel_fn 中调用 model.build_summary_graph()RankModel 遍历 summaries_set 写入 tf.summary.scalar
  4. pred_name 解析:从 prediction_dict 取张量;若 key 为 logits / logits_* 则先 sigmoid。
  5. 数值特征切片RawFeaturefeature_dict 中为 float(如 1005.0),切片匹配时按数值比较,避免 as_string 变成 1005.000000 导致与配置 "1005" 不一致。

改动文件

文件 说明
easy_rec/python/protos/summary.proto 新增 PCOC / Scalars / ModelSummaries
easy_rec/python/protos/easy_rec_model.proto EasyRecModel 增加 summaries_set
easy_rec/python/model/rank_model.py 实现 build_summary_graph 及切片逻辑
easy_rec/python/model/easy_rec_estimator.py 训练时调用 summary 构图
easy_rec/python/model/easy_rec_model.py 基类空实现
easy_rec/python/test/eval_summary_test.py 单元测试
samples/model_config/deepfm_combo_on_avazu_ctr.config 配置示例(英文注释)

配置示例

model_config {
  summaries_set {
    pcoc {
      # pred_name: "probs"   # 默认;logits 会先 sigmoid
      # epsilon: 1e-7
    }
  }
  summaries_set {
    scalars {
      name: "c1_1005_pred"      # TensorBoard: summary/c1_1005_pred
      pred_name: "probs"
      feature_name: "c1"
      feature_value: "1005"
    }
  }
}

多塔模型在顶层通过 pred_name: "probs_ctr" 等区分任务,无需在 task_towers 重复配置。

TensorBoard 标签

配置类型 写入标签
pcoc summary/pcocsummary/predicted_ctrsummary/observed_ctr
scalars summary/<name>

测试结论

1. 单元测试

python -m easy_rec.python.test.eval_summary_test

结果:2/2 通过

  • proto 解析 summaries_set(pcoc + scalars)
  • PCOC 图数值:pcoc = mean(pred)/mean(label) 校验通过
  • scalars 切片:float 特征 c1=1005.0feature_value: "1005" 匹配,输出 0.9

2. 样本端到端(deepfm_combo_on_avazu_ctr.config,50 step)

数据:data/test/dwd_avazu_ctr_deepmodel_10w.csvc1=1005 约占 90.7%)

标签 实测值 说明
summary/predicted_ctr 0.660 全 batch 平均预估
summary/observed_ctr 0.176 全 batch 平均标签
summary/pcoc 3.753 predicted/observed 一致
summary/c1_1005_pred 0.657 切片非零,且接近全局预估

校验项:四个 tag 均写入 events;pcoc ≈ predicted_ctr / (observed_ctr + 1e-7)c1_1005_pred > 0通过

3. 回归

test_deepfm_with_combo_feature(同 sample 默认训练流程)通过

@fooSynaptic fooSynaptic force-pushed the feat/summaries-set-pcoc branch from 5953dd6 to 43461dc Compare July 8, 2026 16:41
在 RankModel 实现训练期 TensorBoard summary:PCOC 与 scalars 特征切片。
配置入口为 model_config.summaries_set(非 eval_config)。

修复 alibaba#532
将 build_summary_graph 从 RankModel 移至 EasyRecModel 基类,Match/召回等模型均可配置。
PCOC 新增 label_name 字段以支持多任务标签指定。
@fooSynaptic fooSynaptic force-pushed the feat/summaries-set-pcoc branch from 43461dc to 0cfc75c Compare July 8, 2026 16:46
@fooSynaptic fooSynaptic requested a review from yangxudong July 8, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant