Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions changes/feat-manifest-attio-deals.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
### Added

- **A CRM source from a manifest reads a pipeline answered as a flat list of stages** (`stages.each: "."`, grouped by
`stages.pipeline` or one pipeline), and a hook `pipelines.<kind>.outcome` decides a stage's outcome the data cannot
say. Attio's deal stages and Pipedrive's `/stages` answer this way.
- **`custom.typed` reads the account's own fields from typed entries with their history**, as Attio keeps them: the
current entry's value by its type (`TYPED_VALUE_PATHS`, overridable), a multiple field or several current entries as a list.
`fields.keep` takes `{ path, is }`, so a config file can say which fields are the account's own.
- **`entities.person.search` and `entities.organization.search` give a manifest source a live `list`**, for `people
list` and `organizations list`: words through the search call's `"{query}"`, none through `list`. A condition it
cannot send is refused, not dropped; a live list pages by offset, so a cursor-paged search is refused.
- `hooks.extend` gets the service's `call` as its second argument.

### Fixed

- `crmManifestFake` matches a call whatever page size and offset the source asked for, a live list setting its own, and writes a stage's won or lost flag as a boolean or a word, as its schema takes it.
15 changes: 12 additions & 3 deletions docs/dev/CRM.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,8 +242,17 @@ hooks })` builds the `CrmSource` (design and evidence: cli-private `plans/2026-1
changes mid-round moves nothing under the next page (a service's own page token is an offset into a moving list).
Past `cap` rows at one time the round answers `reset`.
- What a manifest cannot say is a hook: `after` per kind for one record (async, with the metered `call`),
`extend(base)` for whole methods (`self`, `activities`, a live `list`). Bitrix24's `mine` scope is that kind of
policy, and stays code.
`extend(base, { call })` for whole methods (`self`, `activities`), `pipelines.<kind>.outcome(stage, pipeline,
found)` for a stage's outcome the data cannot say (Attio names a lost stage only by its title). Bitrix24's `mine`
scope is that kind of policy, and stays code.
- A pipeline call that answers the stages themselves (Attio's statuses, Pipedrive's `/stages`) sets `stages.each` to
`.`: the stages group by `stages.pipeline`, or are one pipeline, and the pipeline's map reads its first stage.
- `custom.typed` reads values kept as a list of typed entries with their history (Attio's `values.<slug>`): the
entry's type (`by`) picks the path of its value from `TYPED_VALUE_PATHS` or `typed.paths`, an entry with a time at
`current` is history, and several current entries are a list. `fields.keep` takes `{ path, is }` as data.
- `entities.<person|organization>.search` names a call with `"{query}"`: the source then has a live `list` for
`people list` and `organizations list`, words through `search` and none through `list`, paged by the operation's
offset. A condition it cannot send is refused (`query.unsupported-field`), never dropped.
- `crmManifestProblems(manifest, api, definitions)` checks every mapped path against the schema each operation
answers; a CLI runs it in a test, so a wrong path fails before any sync.
- `hooks.enrich` reads more for a whole page in one batch before the map; `optional` makes a refused kind a gap;
Expand All @@ -254,7 +263,7 @@ hooks })` builds the `CrmSource` (design and evidence: cli-private `plans/2026-1
sends the one operation, refuses a token the answer does not call `active`, and keeps the id and name the answer
holds. `hostsFetch(hosts, environment.fetch)` is the fetch for it and the source: only the CLI's hosts, over a test's
fake when there is one. A CLI made from the config holds no check of its own.
- Not yet: deletions named in the answer, a live `list`.
- Not yet: deletions named in the answer.
- `src/crm/manifest/manifest.test.ts` writes HubSpot's contacts, companies and deals as a manifest and runs
`crmSourceContract`, faults included, against a strict fake of HubSpot's API.

Expand Down
67 changes: 61 additions & 6 deletions src/crm/manifest/manifest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ export interface CrmManifestEntity<K extends CrmKind = CrmKind> {
ids?: CrmOperationCall
/** One row by `{id}`, for a live read of a contact or company; a 403 or 404 is no row. */
get?: CrmOperationCall
/**
* Contacts or companies whose name holds `{query}`, for a live `people list` and `organizations list`; a list with no
* words reads `list`. Both page by the operation's own `pagination`. Absent, the source has no live list.
*/
search?: CrmOperationCall
/**
* The service names the fields a read returns (HubSpot `properties=`, Bitrix24 `select`): `{select}` becomes the
* names under `from` that `map` reads, the custom field codes, and `also`.
Expand All @@ -70,19 +75,36 @@ export interface CrmManifestEntity<K extends CrmKind = CrmKind> {
}
map: { readonly [F in keyof CrmRecords[K]]?: CrmRule } & { amount?: CrmRule; currency?: CrmRule }
/** The account's own fields' values, by the codes `fields` lists, under this object of the row. */
custom?: { from: string; split?: string }
custom?: { from: string; split?: string; typed?: CrmTypedValues }
/**
* A kind the plan or the sign-in may not have (HubSpot's leads): the service refusing it is a gap with this text and
* scope, and the other kinds still sync.
*/
optional?: { message: string; scope?: string }
}

/**
* Values kept as a list of typed entries, each naming its type (Attio's `values.<slug>`): the entry's type picks where
* its value is, and a replaced entry, one with a time at `current`, is skipped.
*/
export interface CrmTypedValues {
/** Where an entry names its type: `attribute_type`. */
by: string
/** A time an entry was replaced at; an entry with one is history, not the value. */
current?: string
/** Where the value is for each type, over `TYPED_VALUE_PATHS`. */
paths?: Readonly<Record<string, string>>
}

export interface CrmManifestPipelines {
call: CrmOperationCall
/** Read from the pipeline's row, or from its first stage when the call answers stages (`stages.each` is `.`). */
map: { externalId: CrmRule; name: CrmRule }
stages: {
/** The stages in a pipeline's row; `.` when the call answers the stages themselves, one row each. */
each: string
/** With `each` `.`: the stage's pipeline, by which the stages are grouped; absent, they are all one pipeline. */
pipeline?: string
map: { externalId: CrmRule; name: CrmRule; position?: CrmRule }
/** Which stages are won or lost by one field; any other is open. A rule a field cannot say is a function. */
outcome:
Expand All @@ -105,7 +127,7 @@ export interface CrmManifestFields {
| ((raw: CrmRaw) => { id: string; label: string }[] | undefined)
}
/** Which listed fields are the account's own, not the service's built-in ones the map already reads. */
keep?: (raw: CrmRaw) => boolean
keep?: { path: string; is: unknown } | ((raw: CrmRaw) => boolean)
}

/** A CRM account described as data: what to call for each kind and where each of our fields is in the answer. */
Expand Down Expand Up @@ -143,8 +165,14 @@ export interface CrmManifestHooks {
enrich?: {
readonly [K in CrmKind]?: (rows: CrmRaw[], service: { call: ServiceApiCall }) => Promise<CrmRaw[]>
}
/** The outcome of a stage the data cannot say (Attio names a lost stage only by its title); `found` is the data's. */
pipelines?: {
readonly [K in DealKind]?: {
outcome?: (stage: CrmRaw, pipeline: CrmRaw, found: StageOutcome) => StageOutcome
}
}
/** Adds or replaces source methods (`self`, `activities`, a live `list`); `base` is what the manifest built. */
extend?: (base: CrmSource) => Partial<CrmSource>
extend?: (base: CrmSource, service: { call: ServiceApiCall }) => Partial<CrmSource>
}

const path = v.pipe(v.string(), v.regex(/^[^.\s]+(\.[^.\s]+)*$/, "a dotted path such as properties.name"))
Expand Down Expand Up @@ -189,6 +217,7 @@ const entity = v.strictObject({
),
ids: v.optional(call),
get: v.optional(call),
search: v.optional(call),
select: v.optional(
v.strictObject({
from: path,
Expand All @@ -197,15 +226,24 @@ const entity = v.strictObject({
}),
),
map: v.record(v.string(), rule),
custom: v.optional(v.strictObject({ from: path, split: v.optional(v.string()) })),
custom: v.optional(
v.strictObject({
from: path,
split: v.optional(v.string()),
typed: v.optional(
v.strictObject({ by: path, current: v.optional(path), paths: v.optional(v.record(v.string(), path)) }),
),
}),
),
optional: v.optional(v.strictObject({ message: v.pipe(v.string(), v.minLength(1)), scope: v.optional(v.string()) })),
})

const pipelines = v.strictObject({
call,
map: v.strictObject({ externalId: rule, name: rule }),
stages: v.strictObject({
each: path,
each: v.union([v.literal("."), path]),
pipeline: v.optional(path),
map: v.strictObject({ externalId: rule, name: rule, position: v.optional(rule) }),
outcome: v.union([
v.strictObject({ path, won: v.optional(v.array(v.string())), lost: v.optional(v.array(v.string())) }),
Expand All @@ -225,7 +263,7 @@ const fields = v.strictObject({
filterable: v.optional(rule),
options: v.optional(v.union([v.strictObject({ each: path, id: path, label: path }), v.function()])),
}),
keep: v.optional(v.function()),
keep: v.optional(v.union([v.strictObject({ path, is: v.unknown() }), v.function()])),
})

const kind = v.picklist(CRM_KINDS)
Expand Down Expand Up @@ -308,6 +346,21 @@ export const parseCrmManifest = (input: unknown, api: ServiceApiManifest): CrmMa
if (!hasPlaceholder([one.get.path, one.get.query, one.get.body], "{id}"))
throw invalid(`entities.${kind}.get`, 'no "{id}" in its path, query or body')
}
if (one.search) {
if (kind !== "person" && kind !== "organization")
throw invalid(`entities.${kind}.search`, "only contacts and companies are read live")
checkCall(`entities.${kind}.search`, one.search, true)
for (const [name, live] of [
["search", one.search],
["list", one.list],
] as const) {
const style = operations.get(live.operation)?.pagination?.style
if (style !== undefined && style !== "offset")
throw invalid(`entities.${kind}.${name}`, `a live list pages by offset or not at all, not by ${style}`)
}
if (!hasPlaceholder([one.search.path, one.search.query, one.search.body], "{query}"))
throw invalid(`entities.${kind}.search`, 'no "{query}" in its path, query or body')
}
if (one.map.externalId === undefined) throw invalid(`entities.${kind}.map.externalId`, "every kind maps its id")
const fields = [...Object.keys(FIELDS[kind]), "amount", "currency"]
for (const field of Object.keys(one.map))
Expand All @@ -323,6 +376,8 @@ export const parseCrmManifest = (input: unknown, api: ServiceApiManifest): CrmMa
const one = manifest.pipelines?.[kind]
if (!one) throw invalid(`pipelines.${kind}`, `${kind} is in kinds, and every deal or lead stands in a pipeline`)
checkCall(`pipelines.${kind}.call`, one.call, true)
if (one.stages.pipeline !== undefined && one.stages.each !== ".")
throw invalid(`pipelines.${kind}.stages.pipeline`, 'groups stages only when the call answers them: each "."')
}
for (const [kind, one] of Object.entries(manifest.fields ?? {}))
if (one) checkCall(`fields.${kind}.call`, one.call, true)
Expand Down
50 changes: 46 additions & 4 deletions src/crm/manifest/map.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import type { CrmField, CrmKind, CrmRecords } from "../../domain/crm.js"
import { DEAL_STATUSES } from "../../domain/deals.js"
import { amountOf, type CrmRaw, crmValuesOf, epoch, fullName, id, isRaw, number, text } from "../values.js"
import type { CrmRule } from "./manifest.js"
import type { CrmRule, CrmTypedValues } from "./manifest.js"

/** The service sent nothing at this path: the field is left out, so the stored copy keeps what it held. */
export const MISSING = Symbol("missing")
Expand Down Expand Up @@ -211,13 +211,55 @@ export const mapFields = (
return out
}

/** Where a typed entry keeps its value, by its type, as Attio names them; a manifest's `typed.paths` adds or replaces. */
export const TYPED_VALUE_PATHS: Readonly<Record<string, string>> = {
text: "value",
number: "value",
checkbox: "value",
date: "value",
timestamp: "value",
rating: "value",
currency: "currency_value",
status: "status.title",
select: "option.title",
"record-reference": "target_record_id",
"actor-reference": "referenced_actor_id",
domain: "domain",
"email-address": "email_address",
"phone-number": "phone_number",
"personal-name": "full_name",
interaction: "interacted_at",
}

/** The current entries' values: one alone is the value, several a list, none no value. */
const typedOf = (entries: unknown, typed: CrmTypedValues): unknown => {
const values = listOf(entries).flatMap((entry) => {
if (!isRaw(entry)) return []
if (typed.current !== undefined && at(entry, typed.current) != null) return []
const type = text(at(entry, typed.by))
const path = type === null ? undefined : (typed.paths?.[type] ?? TYPED_VALUE_PATHS[type])
const value = path === undefined ? null : at(entry, path)
return value === null || value === undefined || value === "" ? [] : [value]
})
return values.length > 1 ? values : (values[0] ?? null)
}

/** The account's own fields' values under `from`, labelled by their definitions. */
const customOf = (raw: CrmRaw, custom: { from: string; split?: string }, defined: readonly CrmField[]) => {
const customOf = (
raw: CrmRaw,
custom: { from: string; split?: string; typed?: CrmTypedValues },
defined: readonly CrmField[],
) => {
const under = at(raw, custom.from)
if (!isRaw(under)) return undefined
const values: CrmRaw = {}
for (const field of defined) {
const value = under[field.code]
const typed = custom.typed ? typedOf(under[field.code], custom.typed) : undefined
const value = !custom.typed
? under[field.code]
: field.multiple && typed !== null && !Array.isArray(typed)
? [typed]
: typed
if (value === null || value === undefined || value === "") continue
values[field.code] = field.multiple && custom.split && typeof value === "string" ? value.split(custom.split) : value
}
Expand All @@ -226,7 +268,7 @@ const customOf = (raw: CrmRaw, custom: { from: string; split?: string }, defined

export interface RecordRules {
map: Readonly<Record<string, CrmRule | undefined>>
custom?: { from: string; split?: string }
custom?: { from: string; split?: string; typed?: CrmTypedValues }
}

/** One row as our record of the kind, or null for a row without an id. */
Expand Down
12 changes: 8 additions & 4 deletions src/crm/manifest/paths.ts
Original file line number Diff line number Diff line change
Expand Up @@ -143,20 +143,24 @@ export const crmManifestProblems = (
...(entity.map as Record<string, CrmRule | undefined>),
...(entity.custom ? { "(custom)": entity.custom.from } : {}),
}
for (const call of [entity.list, entity.changes, entity.get]) if (call) check(`entities.${kind}.map`, call, rules)
for (const call of [entity.list, entity.changes, entity.get, entity.search])
if (call) check(`entities.${kind}.map`, call, rules)
}
for (const [kind, pipelines] of Object.entries(manifest.pipelines ?? {})) {
if (!pipelines) continue
check(`pipelines.${kind}.map`, pipelines.call, { ...pipelines.map, "(stages)": pipelines.stages.each })
const { outcome } = pipelines.stages
const { outcome, each, pipeline } = pipelines.stages
// A call that answers stages has one row per stage: the pipeline's map and the stage's read the same row.
const flat = each === "."
check(`pipelines.${kind}.map`, pipelines.call, flat ? pipelines.map : { ...pipelines.map, "(stages)": each })
check(
`pipelines.${kind}.stages.map`,
pipelines.call,
{
...pipelines.stages.map,
...(typeof outcome === "object" && "path" in outcome ? { outcome: outcome.path } : {}),
...(pipeline ? { "(pipeline)": pipeline } : {}),
},
pipelines.stages.each,
flat ? undefined : each,
)
}
for (const [kind, fields] of Object.entries(manifest.fields ?? {})) {
Expand Down
Loading
Loading