Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ team.yaml
tokens.yaml
registry.yaml
feishu.yaml
notion.yaml
google.yaml
github.token
*.local.*

Expand Down
11 changes: 11 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,17 @@ If your team keeps human-written docs (goals, decisions, notes) in a Lark/Feishu

> China Feishu (`open.feishu.cn`) and international Lark (`open.larksuite.com`) are isolated platforms — create the app on whichever one your team uses. Create a **new** wiki later? Repeat step 3 for it, or the brain won't see it.

### Other doc sources (Notion · Google Docs)

Doc mirroring is **provider-pluggable** — the same sync engine (`server/docsync.mjs`) backs every source, so each one is just a small adapter (`core/<provider>.mjs` + `server/<provider>docs.mjs`). Two more ship today, both gated by their own `*.yaml` (leave it out → that layer stays off) and both following the same "share with the bot, then it mirrors" model:

- **Notion** — create an *internal integration* at <https://www.notion.so/my-integrations> (read-only); **share** the pages/databases with it (page → ••• → *Connections*); copy `notion.example.yaml` → `notion.yaml`, fill `api_token`, restart. Pages land under `notion/<workspace>/…`.
- **Google Docs** — create a *service account* (enable the Drive API), download its JSON key, and **share** the docs/folders with the service account's email (read-only); copy `google.example.yaml` → `google.yaml`, point it at the key, restart. Docs land under `google/<workspace>/…`. (Auth is a self-signed JWT — no extra SDK.)

Unshared pages/docs stay invisible (the share is the real access gate); sub-pages and folder contents inherit. After a poll cycle everything is searchable via `grep`.

> Confluence can follow the same adapter shape — contributions welcome.

---

## Changelog
Expand Down
11 changes: 11 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,17 @@ npm run sync -- --once # 收一次(或 `npm run sync` 让它在后台持续

> 国内飞书(`open.feishu.cn`)与国际版 Lark(`open.larksuite.com`)数据隔离——在你团队实际所在的平台建应用。以后每**新建**一个知识库,都要对它重复第 3 步,否则大脑看不见。

### 其它文档源(Notion · Google Docs)

文档镜像是**可插拔多源**的——所有源共用同一套同步引擎(`server/docsync.mjs`),每个源只是一个小 adapter(`core/<源>.mjs` + `server/<源>docs.mjs`)。现已多支持两个源,各由自己的 `*.yaml` 把关(不建即不启用),都遵循"把内容 share 给机器人、它就镜像"的模型:

- **Notion** —— 在 <https://www.notion.so/my-integrations> 建一个 *internal integration*(只读);把页面/数据库 **share** 给它(页面 → ••• → *Connections*);复制 `notion.example.yaml` → `notion.yaml`,填 `api_token`,重启。页面落到 `notion/<workspace>/…`。
- **Google Docs** —— 建一个 *service account*(启用 Drive API),下载 JSON key,把文档/文件夹 **share** 给该 service account 的邮箱(只读);复制 `google.example.yaml` → `google.yaml`,指向 key,重启。文档落到 `google/<workspace>/…`。(认证用自签 JWT,不引入额外 SDK。)

没 share 的页面/文档看不到(share 才是真正的授权闸);子页面、文件夹内容继承。一轮后即可 `grep` 搜到。

> Confluence 可按同一 adapter 范式接入——欢迎贡献。

---

## 更新日志
Expand Down
19 changes: 3 additions & 16 deletions core/feishu.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
import { readFileSync, existsSync } from "node:fs";
import { parse } from "yaml";
import * as lark from "@larksuiteoapi/node-sdk";
import { sleep, withRetry } from "./retry.mjs";

export { sleep, withRetry };

// feishu.yaml(服务器级、gitignore、启动加载 restart 生效,同 registry/tokens 一族)。
// 缺文件 / 解析失败 / 没配齐 app 凭证 → 返回 null = 文档层不启用(与 GITHUB_TOKEN 缺省同款行为)。
Expand All @@ -20,22 +23,6 @@ export function loadFeishu(path) {
};
}

export const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// 重试退避:429(限流)与 5xx(网关偶发)值得等一等再试;4xx 业务错误(权限/不存在)重试无意义直接抛。
// status 取不到(网络层断连)也按可重试算。
const RETRYABLE = new Set([429, 500, 502, 503, 504]);
export async function withRetry(fn, { delays = [1000, 3000, 10000], sleepFn = sleep } = {}) {
for (let i = 0; ; i++) {
try { return await fn(); }
catch (e) {
const status = e?.response?.status ?? 0;
if (i >= delays.length || (status !== 0 && !RETRYABLE.has(status))) throw e;
await sleepFn(delays[i]);
}
}
}

// 统一请求口:client.request 自动注入 tenant_access_token;外面包一层重试。
export function makeReq({ app_id, app_secret }, retryOpts = {}) {
const client = new lark.Client({
Expand Down
91 changes: 91 additions & 0 deletions core/google.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
// Google Docs 只读镜像(Drive + export,原生 fetch + node:crypto 自签 JWT,无 googleapis SDK)。
// 凭证 = 一个 service account;把文档/文件夹 share 给它的邮箱才可见(同 Notion 的"share 给 integration")。
// 正文走 Drive export 到 text/plain:一次拿全文,免解析 Docs 结构。增量靠 Drive 的 modifiedTime(已是 RFC3339 ISO)。
import { readFileSync, existsSync } from "node:fs";
import { createSign } from "node:crypto";
import { parse } from "yaml";
import { sleep, withRetry } from "./retry.mjs";

export { sleep, withRetry };

const DRIVE = "https://www.googleapis.com/drive/v3";
const TOKEN_URI = "https://oauth2.googleapis.com/token";
const SCOPE = "https://www.googleapis.com/auth/drive.readonly";

// google.yaml(服务器级、gitignore、启动加载 restart 生效)。缺文件/解析失败/没配齐凭证 → null = Google 层不启用。
// 凭证两种给法:① key_file 指向下载的 service-account JSON;② 直接内联 client_email + private_key。
export function loadGoogle(path) {
if (!existsSync(path)) return null;
let cfg;
try { cfg = parse(readFileSync(path, "utf8")) || {}; } catch { return null; }
let client_email = cfg.client_email, private_key = cfg.private_key;
if (cfg.key_file) {
try { const k = JSON.parse(readFileSync(cfg.key_file, "utf8")); client_email = k.client_email; private_key = k.private_key; }
catch { return null; }
}
if (!client_email || !private_key) return null;
return {
client_email: String(client_email),
private_key: String(private_key),
poll_hours: Number(cfg.poll_hours) || 4,
workspace: String(cfg.workspace || "google"), // 真相库 google/ 下的目录名(自己起)
};
}

// service-account JWT → assertion(RS256 签名走 node:crypto,免 SDK)。
function signJwt({ client_email, private_key }) {
const enc = (o) => Buffer.from(JSON.stringify(o)).toString("base64url");
const iat = Math.floor(Date.now() / 1000);
const head = { alg: "RS256", typ: "JWT" };
const claim = { iss: client_email, scope: SCOPE, aud: TOKEN_URI, iat, exp: iat + 3600 };
const body = `${enc(head)}.${enc(claim)}`;
const sig = createSign("RSA-SHA256").update(body).end().sign(private_key).toString("base64url");
return `${body}.${sig}`;
}

// 用 JWT 换 access_token(OAuth2 jwt-bearer 流)。
export async function fetchToken(cfg) {
const r = await fetch(TOKEN_URI, {
method: "POST",
headers: { "content-type": "application/x-www-form-urlencoded" },
body: new URLSearchParams({ grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer", assertion: signJwt(cfg) }),
});
if (!r.ok) throw Object.assign(new Error(`Google token ${r.status}: ${(await r.text()).slice(0, 120)}`), { status: r.status });
return r.json(); // { access_token, expires_in, token_type }
}

// 认证请求口:缓存 access_token(~1h),到期前 60s 刷新;req(method, path, {raw}) → json 或纯文本。
// path 以 / 开头时相对 Drive v3;完整 http 原样用。非 2xx 抛带 .status 的错误(供 withRetry 判重试)。
export function makeReq(cfg, retryOpts = {}) {
let tok = null, exp = 0;
const auth = async () => {
if (tok && Date.now() < exp - 60000) return tok;
const t = await fetchToken(cfg);
tok = t.access_token; exp = Date.now() + (Number(t.expires_in) || 3600) * 1000;
return tok;
};
return (method, path, { raw = false } = {}) => withRetry(async () => {
const token = await auth();
const r = await fetch(path.startsWith("http") ? path : DRIVE + path, { method, headers: { authorization: `Bearer ${token}` } });
if (!r.ok) throw Object.assign(new Error(`Google ${r.status} on ${path}: ${(await r.text()).slice(0, 120)}`), { status: r.status });
return raw ? r.text() : r.json();
}, retryOpts);
}

// 列出 service account 可见的全部 Google 文档(分页)。归一为 [{id,name,modifiedTime,webViewLink}]。
export async function listDocs(req) {
const out = [];
const q = "mimeType='application/vnd.google-apps.document' and trashed=false";
const fields = "nextPageToken,files(id,name,modifiedTime,webViewLink)";
let pageToken = "";
do {
const qs = new URLSearchParams({ q, fields, pageSize: "100", ...(pageToken ? { pageToken } : {}) });
const r = await req("GET", `/files?${qs}`);
out.push(...(r?.files || []));
pageToken = r?.nextPageToken || "";
} while (pageToken);
return out;
}

// 导出某文档为纯文本(Drive export,一次拿全文,免解析 Docs 结构)。
export const exportDocText = (req, id) => req("GET", `/files/${id}/export?mimeType=text/plain`, { raw: true });
99 changes: 99 additions & 0 deletions core/notion.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
// Notion 只读 API(REST, 原生 fetch)——单向镜像团队 Notion 页面进真相库,供 grep/read。
// 与 github.mjs 同款:原生 fetch、无 SDK(不给客户端/服务端添依赖);错误带 .status;429/5xx 退避重试。
// 凭证 = 一个 internal integration token;只有被 share 给该 integration 的页面才可见(见 notion.example.yaml)。
import { readFileSync, existsSync } from "node:fs";
import { parse } from "yaml";
import { sleep, withRetry } from "./retry.mjs";

export { sleep, withRetry };

const API = "https://api.notion.com/v1";
const VERSION = "2022-06-28"; // Notion-Version 请求头,固定一个验证过的版本,别跟着平台默认漂

// notion.yaml(服务器级、gitignore、启动加载 restart 生效,同 feishu/registry/tokens 一族)。
// 缺文件 / 解析失败 / 没配 api_token → 返回 null = Notion 文档层不启用(与 loadFeishu 同款行为)。
export function loadNotion(path) {
if (!existsSync(path)) return null;
let cfg;
try { cfg = parse(readFileSync(path, "utf8")) || {}; } catch { return null; }
if (!cfg.api_token) return null;
return {
api_token: String(cfg.api_token),
poll_hours: Number(cfg.poll_hours) || 4,
workspace: String(cfg.workspace || "notion"), // 真相库 notion/ 下的目录名(Notion API 不给工作区名,自己起)
};
}

// 统一请求口:注入 token + Notion-Version;非 2xx 抛带 .status 的错误(供 withRetry 判重试,同 github.mjs)。
// GET 不带 body(分页 cursor 走 query);POST/search 带 JSON body。
export function makeReq({ api_token }, retryOpts = {}) {
const headers = {
authorization: `Bearer ${api_token}`,
"notion-version": VERSION,
"content-type": "application/json",
"user-agent": "team-brain",
};
return (method, path, body) => withRetry(async () => {
const r = await fetch(API + path, { method, headers, body: body ? JSON.stringify(body) : undefined });
if (!r.ok) throw Object.assign(new Error(`Notion ${r.status} on ${path}: ${(await r.text()).slice(0, 120)}`), { status: r.status });
return r.json();
}, retryOpts);
}

// POST /search 分页(cursor 在 body):列出 integration 可见的全部 page。
async function searchAll(req, body) {
const out = [];
let cursor;
do {
const r = await req("POST", "/search", { ...body, page_size: 100, ...(cursor ? { start_cursor: cursor } : {}) });
out.push(...(r?.results || []));
cursor = r?.has_more ? (r?.next_cursor || null) : null;
} while (cursor);
return out;
}
export const searchPages = (req) => searchAll(req, { filter: { value: "page", property: "object" } });

// GET /blocks/{id}/children 分页(cursor 在 query)。
async function childrenAll(req, blockId) {
const out = [];
let cursor;
do {
const qs = `?page_size=100${cursor ? `&start_cursor=${encodeURIComponent(cursor)}` : ""}`;
const r = await req("GET", `/blocks/${blockId}/children${qs}`);
out.push(...(r?.results || []));
cursor = r?.has_more ? (r?.next_cursor || null) : null;
} while (cursor);
return out;
}

// 从 page 对象取标题:properties 里 type==="title" 的那个,拼 plain_text;取不到兜底 untitled。
export function pageTitle(page) {
const props = page?.properties || {};
for (const k of Object.keys(props)) {
if (props[k]?.type === "title") return (props[k].title || []).map((t) => t.plain_text || "").join("").trim() || "untitled";
}
return "untitled";
}

// 一个块的纯文本:任何块类型下的 rich_text 拼接(标题/段落/列表/引用… 字段名都叫 rich_text)。
const blockText = (b) => {
const rich = b?.[b.type]?.rich_text;
return Array.isArray(rich) ? rich.map((x) => x.plain_text || "").join("") : "";
};

// 递归取 page/block 的纯文本正文:children 翻页 → 各块拼文本 → 有子块下钻(限深防御)。pace 由调用方按限流给。
export async function pageBlocksText(req, blockId, { sleepFn = sleep, pace = 0, depth = 0 } = {}) {
if (depth > 8) return "";
const blocks = await childrenAll(req, blockId);
const parts = [];
for (const b of blocks) {
const line = blockText(b);
if (line) parts.push(line);
if (b.has_children) {
if (pace) await sleepFn(pace);
const sub = await pageBlocksText(req, b.id, { sleepFn, pace, depth: depth + 1 });
if (sub) parts.push(sub);
}
}
return parts.join("\n");
}
17 changes: 17 additions & 0 deletions core/retry.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
// 共享重试退避:429(限流)与 5xx(网关偶发)值得等一等再试;4xx 业务错误(权限/不存在)重试无意义直接抛。
// status 取不到(网络层断连)也按可重试算。错误形状两种都认:原生 fetch 抛的 e.status / SDK 抛的 e.response.status。
// 文档源各 provider(feishu/notion/google)共用——退避策略只此一处,改一处全改。
export const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

const RETRYABLE = new Set([429, 500, 502, 503, 504]);

export async function withRetry(fn, { delays = [1000, 3000, 10000], sleepFn = sleep } = {}) {
for (let i = 0; ; i++) {
try { return await fn(); }
catch (e) {
const status = e?.status ?? e?.response?.status ?? 0;
if (i >= delays.length || (status !== 0 && !RETRYABLE.has(status))) throw e;
await sleepFn(delays[i]);
}
}
}
5 changes: 5 additions & 0 deletions core/safe.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ export function safeSegment(s, label = "segment") {
return v;
}

// 文档源标题/库名要进文件名:去掉路径分隔与控制符、压空白、截短到 80;空兜底 untitled。
// 之后仍过 safeSegment(穿越红线,别绕开)。文档镜像各 provider(feishu/notion/google)共用——名字消毒只此一处。
export const saniName = (s, label = "name") =>
safeSegment((String(s || "").replace(/[/\\\0]/g, "-").replace(/\s+/g, " ").trim() || "untitled").slice(0, 80), label);

// 多段相对路径锁在 root 内:解析成绝对路径后必须仍以 root 为前缀(或恰为 root)。
// 给只读查询(ls/log/grep 的 path 收窄)用——客户端给的 rel 直接落进 root 子树前必过。
export function safeRelPath(root, rel, label = "path") {
Expand Down
28 changes: 28 additions & 0 deletions google.example.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# 复制成 google.yaml(服务器级、已 gitignore——含 service account 私钥)。
# 不建该文件 / 不填凭证 → Google 文档层不启用(与 feishu.yaml / notion.yaml 同款"缺省即关")。
#
# 准备(一次性):
# 1. 在 Google Cloud Console 建一个项目,启用 **Google Drive API**。
# 2. 建一个 **service account**,给它生成一把 JSON key 并下载。
# 3. 把要镜像的 Google 文档(或整个文件夹)**share 给该 service account 的邮箱**
# (xxx@xxx.iam.gserviceaccount.com),只读权限即可。
# —— 这步才是真正的授权闸:没 share 的文档 service account 看不到;文件夹内文档随文件夹继承。
# 4. 填下面的凭证,重启 server。启动 120s 后首跑,一轮后文档落到
# google/<workspace>/<标题>--<doc_id>.md,grep/read 即可搜(正文已脱敏)。
#
# 凭证两种给法(二选一):
# ① key_file:指向下载的 service-account JSON(把那个 .json 也放到 server 上、别进仓)。
# ② 内联 client_email + private_key(private_key 是多行 PEM,用 YAML 块标量 |)。
#
# 正本永远在 Google Docs(人在那读写),这棵子树只是可重建的只读检索镜像——要改去 Google Docs 改。

key_file: /var/lib/team-brain/google-sa.json
# 或者内联:
# client_email: team-brain@your-project.iam.gserviceaccount.com
# private_key: |
# -----BEGIN PRIVATE KEY-----
# ...
# -----END PRIVATE KEY-----

poll_hours: 4 # 轮询间隔(小时),可选,默认 4
workspace: google # 真相库 google/ 下的目录名(自己起一个),可选
Loading
Loading