Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,8 @@ api_key = ""

批量运行可用 `for result, err := range client.RunResults(ctx, cfg)` 按完成顺序读取结果,`attempt` 从 1 开始。目标数表示尝试次数;每次失败保留结果并继续后续尝试,不自动重试。除第一次外,每次尝试开始前都会等待配置中的随机提交间隔。原生 Go 配置显式启用 `failStop` 时仍会在连续失败达到阈值后停止。HTTP 任务用 `GET /api/v1/tasks/{id}/results?after=0&limit=200` 读取同样的逐次结果。

解析结果包含规范化的 `url`,支持省略 `https://` 的问卷链接,并将见数短链接转换为 `answer.html#/s/...`。题目保留原始题型、分页信息及显示题号;Python 的 `dataclasses.asdict(survey)` 可直接作为单次提交的 `survey` 快照。问卷星多级下拉会读取完整层级选项,并保留实际提交值。具体字段及兼容说明见 [数据结构](docs/sdk/schemas.md#surveydefinition)。

## 错误响应

请求级错误统一返回稳定错误码、用户消息和调试详情:
Expand Down
9 changes: 6 additions & 3 deletions api/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,7 @@ components:
properties:
provider: { $ref: '#/components/schemas/Provider' }
title: { type: string }
url: { type: string, format: uri, description: Normalized parse URL; optional for manually supplied legacy snapshots }
questions: { type: array, items: { $ref: '#/components/schemas/QuestionMeta' } }
QuestionMeta:
type: object
Expand All @@ -322,6 +323,7 @@ components:
provider_question_id: { type: string }
provider_page_id: { type: string }
provider_type: { type: string }
provider_page_raw: { description: Original provider page value or multilevel dropdown metadata; null when absent }
required: { type: boolean }
is_description: { type: boolean }
is_location: { type: boolean }
Expand All @@ -339,9 +341,9 @@ components:
display_conditions: { type: array, items: { $ref: '#/components/schemas/DisplayCondition' } }
has_dependent_display_logic: { type: boolean }
controls_display_targets: { type: array, items: { $ref: '#/components/schemas/DisplayControl' } }
slider_min: { type: string }
slider_max: { type: string }
slider_step: { type: string }
slider_min: { type: [number, string, 'null'] }
slider_max: { type: [number, string, 'null'] }
slider_step: { type: [number, string, 'null'] }
question_media: { type: array, items: { $ref: '#/components/schemas/QuestionMedia' } }
multi_min_limit: { type: [integer, 'null'] }
multi_max_limit: { type: [integer, 'null'] }
Expand Down Expand Up @@ -371,6 +373,7 @@ components:
condition_mode: { type: string }
condition_option_indices: { type: array, items: { type: integer } }
condition_row_index: { type: [integer, 'null'] }
raw_relation: { type: string }
DisplayControl:
type: object
additionalProperties: false
Expand Down
7 changes: 7 additions & 0 deletions docs/sdk/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,15 @@ Content-Type: application/json
{
"provider": "wjx",
"title": "用户满意度调查",
"url": "https://www.wjx.cn/vm/example.aspx",
"questions": []
}
```

解析会补齐链接的协议并统一协议、主机名大小写,保留查询参数;见数短链接转换为 `answer.html#/s/...`。返回的 `url` 是规范化后的链接。Go 继续校验 HTTP/HTTPS、平台域名及问卷路径。

题目中的 `num` 为原始题号,`display_num` 为页面显示题号,二者可能不同。腾讯问卷保留说明题,并将同页说明内容合并到后续作答题;默认配置跳过说明题。问卷星普通题的 `provider_type` 保留平台代码,多级下拉使用 `multilevel_dropdown`。

## 生成默认配置

```http
Expand Down Expand Up @@ -136,6 +141,8 @@ Content-Type: application/json

接口同步执行一次并返回 `SubmissionResult`,不创建持久化任务。目标数和并发数固定为 1,不等待任务提交间隔,不自动重试提交。省略题目策略时自动生成默认策略。耗时包含本次解析、准备和平台请求。

`survey` 接受 Python `dataclasses.asdict()` 导出的问卷快照,包括 `url`、数字或 null 的滑块范围、原始分页字段和多级下拉提交值。题目绑定使用原始题号及平台 ID;传入的配置与快照不会被修改。

```json
{
"attempt": 0,
Expand Down
4 changes: 4 additions & 0 deletions docs/sdk/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ if result != nil {

该调用固定执行一次。已有问卷快照可通过 `Survey` 传入。原始 `cfg` 中的目标数和并发数保持原值。

`client.Parse(ctx, url)` 返回的 `definition.SurveyURL` 是规范化链接,JSON 字段为 `url`。数字滑块范围和 `ProviderPageRaw` 保留在快照中;不要用 `DisplayNum` 替代 `Num` 绑定答案。

### Go 逐次提交结果

```go
Expand Down Expand Up @@ -194,6 +196,8 @@ result = response.json()
print(result["success"], result["elapsed_seconds"], result["error"])
```

已有 Python 问卷快照时,可传入 `json={"config": serialize_config(config), "survey": dataclasses.asdict(survey)}`。Go 会保留分页、滑块范围和多级下拉提交值。

### 上传二维码

JavaScript:
Expand Down
2 changes: 2 additions & 0 deletions docs/sdk/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ go get github.com/SurveyController/SurveyCore@<commit>

`Submit` 使用 `surveycore.SubmissionRequest`,其中 `Config` 是 `*model.RunRequest`,`Survey` 是可选的 `*model.SurveyDefinition`。它固定提交一次并返回 `SubmissionResult`;失败时同时返回可用 `errors.Is` 或 `ClassifySubmissionError` 判断的错误。参数中的目标数、并发数和提交间隔不会触发额外提交。

`Parse` 返回的快照包含规范化链接 `SurveyURL`(JSON `url`)、原始题型、分页与显示题号,可接收 Python 导出的问卷快照。滑块范围支持数字、字符串或 null;问卷星多级下拉包含完整选项路径及提交值。

`RunResults` 返回 `iter.Seq2[SubmissionResult, error]`,按完成顺序产生结果,尝试编号从 1 开始。单次失败保存在结果的 `error` 中,迭代继续;准备失败或调用方取消通过迭代错误返回。提前 `break` 会取消未完成的请求并等待工作线程退出。`Run` 和 `RunWithEvents` 返回汇总,存在失败时在运行结束后返回首个错误;`RunWithEvents` 的 `event.Result` 包含逐次结果。

批量运行以 `target` 为尝试次数,受 `threads` 并发限制,不自动重试。第 2 次起,每次尝试开始前均在并发槽内等待 `submitInterval` 范围内的随机秒数。显式 `failStop`、停止操作或取消可以使运行提前结束。Python v1 配置导入时 `failStop` 为 false。
Expand Down
14 changes: 14 additions & 0 deletions docs/sdk/schemas.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,10 +119,24 @@ outline: deep
|---|---|---|
| `provider` | `string` | 问卷平台。 |
| `title` | `string` | 问卷标题。 |
| `url` | `string` | 解析后的规范化链接;Go 字段名为 `SurveyURL`,手动构造旧快照时可省略。 |
| `questions` | `QuestionMeta[]` | 题目元数据。 |

`QuestionMeta` 包含题号、题型、选项、分页、必填状态、显示逻辑、跳题逻辑、媒体和平台原始 ID。字段名以 `POST /api/v1/surveys/parse` 返回值为准。

| 字段 | 类型 | 说明 |
|---|---|---|
| `num` / `display_num` | `integer` / `integer` 或 null | 原始题号及显示题号。腾讯说明题占用原始编号,说明题的显示编号为空。绑定答案时使用原始题号或平台 ID。 |
| `provider_type` | `string` | 平台原始题型。问卷星为 `"3"`、`"4"` 等代码,多级下拉为 `"multilevel_dropdown"`。运行层也继续接受旧快照中的 `single`、`multiple` 等名称。默认配置的 `answers.questions[].question_type` 仍使用统一题型。 |
| `provider_page_raw` | JSON 值 | 腾讯原始 `page`,或问卷星多级下拉的 `cid`、`submit_values`、`level_labels`;没有时为 null。Go 字段为 `any`,复制快照时会深拷贝嵌套元数据。 |
| `slider_min` / `slider_max` / `slider_step` | `number`、`string` 或 null | 解析时返回数字或 null,也接受旧 Go 快照中的字符串。Go 字段为 `any`。 |

Python `dataclasses.asdict(SurveyDefinition)` 可用于 HTTP 单次提交的 `survey`,或反序列化为 Go 的 `model.SurveyDefinition`。Go 额外保留每题的 `provider`;可选空集合可能省略。问卷星多级下拉按父子关系展开选项,显示文字与提交值分别保存;空的隐藏填空占位不会出现在题目列表中,提交时仍保留空位置。外部数据源及超过六级的层级数据会返回解析错误。

腾讯问卷未解析出明确逻辑时保留 `logic_parse_status: "unknown"`。腾讯量表、矩阵量表等已有不支持范围保持原有解析错误。

问卷星显示条件保留 `raw_relation`,并将 `题号,选项号` 转为从 0 开始的 `condition_option_indices`,同时为来源题目填充 `controls_display_targets`。空条件或 `-1` 不会生成显示约束。

## SubmissionRequest

单次提交使用 `{ "config": ..., "survey": ... }`。Go SDK 类型为 `surveycore.SubmissionRequest`。
Expand Down
95 changes: 95 additions & 0 deletions pkg/restapi/parse_parity_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
package restapi

import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"

"github.com/SurveyController/SurveyCore/internal/service"
"github.com/SurveyController/SurveyCore/pkg/surveycore"
"github.com/SurveyController/SurveyCore/pkg/surveycore/model"
)

func TestHTTPParseConfigAndPythonSnapshots(t *testing.T) {
data, err := os.ReadFile("../surveycore/testdata/python-parse.json")
if err != nil {
t.Fatal(err)
}
var fixture struct {
Cases []struct {
Name string `json:"name"`
URL string `json:"url"`
Responses map[string]json.RawMessage `json:"responses"`
Expected json.RawMessage `json:"expected"`
} `json:"cases"`
}
if err := json.Unmarshal(data, &fixture); err != nil {
t.Fatal(err)
}
for _, test := range fixture.Cases {
t.Run(test.Name, func(t *testing.T) {
s, err := New(Config{DBPath: t.TempDir() + "/parse.db"})
if err != nil {
t.Fatal(err)
}
defer s.Close()
transport := submissionTestTransport(func(request *http.Request) (*http.Response, error) {
body := test.Responses[request.URL.Path]
if request.Method == http.MethodPost {
body = []byte("10")
} else if body == nil {
return nil, fmt.Errorf("unexpected request: %s", request.URL)
} else {
var html string
if json.Unmarshal(body, &html) == nil {
body = []byte(html)
}
}
return &http.Response{StatusCode: 200, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(string(body))), Request: request}, nil
})
s.manager = service.NewManager(s.store, surveycore.New(surveycore.WithHTTPClient(&http.Client{Transport: transport})))
requestBody, _ := json.Marshal(map[string]string{"url": test.URL})
var expected model.SurveyDefinition
if err := json.Unmarshal(test.Expected, &expected); err != nil {
t.Fatal(err)
}
for _, path := range []string{"/api/v1/surveys/parse", "/api/v1/configs"} {
rec := httptest.NewRecorder()
s.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodPost, path, strings.NewReader(string(requestBody))))
if rec.Code != http.StatusOK {
t.Fatalf("path=%s status=%d body=%s", path, rec.Code, rec.Body.String())
}
var definition model.SurveyDefinition
if strings.HasSuffix(path, "/configs") {
var config model.RunRequest
if err := json.Unmarshal(rec.Body.Bytes(), &config); err != nil {
t.Fatal(err)
}
definition = config.SurveyDefinition
} else if err := json.Unmarshal(rec.Body.Bytes(), &definition); err != nil {
t.Fatal(err)
}
if definition.Provider != expected.Provider || definition.SurveyURL != expected.SurveyURL || len(definition.Questions) != len(expected.Questions) {
t.Fatalf("HTTP lost survey metadata: %#v", definition)
}
}
if expected.Provider == "wjx" {
body, _ := json.Marshal(map[string]any{
"config": map[string]any{"schema_version": 1, "url": test.URL, "reliability_mode": false, "answer_duration": []int{0, 0}},
"survey": test.Expected,
})
rec := httptest.NewRecorder()
s.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodPost, "/api/v1/submissions", strings.NewReader(string(body))))
var result surveycore.SubmissionResult
if err := json.Unmarshal(rec.Body.Bytes(), &result); err != nil || rec.Code != http.StatusOK || !result.Success {
t.Fatalf("Python snapshot status=%d body=%s error=%v", rec.Code, rec.Body.String(), err)
}
}
})
}
}
19 changes: 4 additions & 15 deletions pkg/surveycore/config/reversefill/excel.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"fmt"
"strings"

"github.com/SurveyController/SurveyCore/pkg/surveycore/internal/defaults"
"github.com/SurveyController/SurveyCore/pkg/surveycore/model"
"github.com/xuri/excelize/v2"
)
Expand Down Expand Up @@ -127,7 +128,7 @@ func parseQuestionAnswer(question model.QuestionMeta, columns []Column, row RawR
}

func inferQuestionType(question model.QuestionMeta) string {
if question.ProviderType != "" {
if question.ProviderType != "" && question.ProviderType != question.TypeCode && question.ProviderType != "multilevel_dropdown" {
switch question.ProviderType {
case "radio", "single":
return "single"
Expand All @@ -137,24 +138,12 @@ func inferQuestionType(question model.QuestionMeta) string {
return "matrix"
case "text", "textarea":
return "text"
}
return ""
}
switch question.TypeCode {
case "3":
return "single"
case "5":
return "scale"
case "6":
return "matrix"
case "7":
return "dropdown"
default:
if question.IsTextLike || question.TextInputs > 0 {
case "multi_text":
return "text"
}
return ""
}
return defaults.QuestionType(question)
}

func valuesByColumn(row []string) map[int]string {
Expand Down
12 changes: 11 additions & 1 deletion pkg/surveycore/internal/answerplan/plan.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,11 @@ func BuildActionWithOptions(question model.QuestionMeta, entry model.QuestionStr
}
case "slider":
action.Kind = kind
action.SliderValue = firstNonEmpty(firstProbabilityText(entry.Probabilities), question.SliderMin, defaultSliderValue)
minimum := ""
if question.SliderMin != nil {
minimum = fmt.Sprint(question.SliderMin)
}
action.SliderValue = firstNonEmpty(firstProbabilityText(entry.Probabilities), minimum, defaultSliderValue)
case "text":
action.Kind = "text"
count := maxInt(1, question.TextInputs)
Expand All @@ -135,7 +139,13 @@ func BuildActionWithOptions(question model.QuestionMeta, entry model.QuestionStr
func normalizeKind(question model.QuestionMeta, entry model.QuestionStrategy) string {
kind := strings.TrimSpace(string(entry.QuestionType))
if kind == "" {
if question.IsSliderMatrix {
return "matrix"
}
kind = strings.TrimSpace(question.ProviderType)
if kind == question.TypeCode || kind == "multilevel_dropdown" {
kind = ""
}
}
switch kind {
case "single", "multiple", "dropdown", "scale", "score", "matrix", "order", "slider", "text", "multi_text":
Expand Down
3 changes: 3 additions & 0 deletions pkg/surveycore/internal/defaults/questions.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ package defaults
import "github.com/SurveyController/SurveyCore/pkg/surveycore/model"

func QuestionType(question model.QuestionMeta) string {
if question.IsSliderMatrix {
return "matrix"
}
switch question.ProviderType {
case "single", "multiple", "dropdown", "scale", "score", "matrix", "order", "slider", "text":
return question.ProviderType
Expand Down
4 changes: 2 additions & 2 deletions pkg/surveycore/internal/providers/tencent/logic.go
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ func attachLogicMetadata(rawQuestions []map[string]any, questions []model.Questi
question.LogicStatus = model.LogicParseStatusUnknown
}
} else {
question.LogicStatus = model.LogicParseStatusNone
question.LogicStatus = model.LogicParseStatusUnknown
}
questions[idx] = question
}
Expand All @@ -98,7 +98,7 @@ func attachLogicMetadata(rawQuestions []map[string]any, questions []model.Questi
}
if raw["hidden"] != nil && raw["hidden"] != "" {
questions[idx].HasDisplayCondition = true
if questions[idx].LogicStatus == model.LogicParseStatusNone {
if questions[idx].LogicStatus != model.LogicParseStatusComplete {
questions[idx].LogicStatus = model.LogicParseStatusUnknown
}
}
Expand Down
Loading
Loading