Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 24 additions & 5 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,32 @@ name: Algorithm tests
on: [push, pull_request]
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
python: ['3.12', '3.13', '3.14', '3.15']
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
python-version: '3.12'
- run: python -m pip install .
- run: python -m unittest discover -s tests
version: '0.13.0'
python-version: ${{ matrix.python }}
enable-cache: true
- name: Install locked project
run: uv sync --locked
- name: Test editable installation
run: uv run --locked python -m unittest discover -s tests -v
- name: Build source distribution and wheel
run: uv build --no-sources
- name: Test installed wheel without source imports
run: |
uv venv "$RUNNER_TEMP/msg-wheel"
uv pip install --python "$RUNNER_TEMP/msg-wheel/bin/python" dist/*.whl
"$RUNNER_TEMP/msg-wheel/bin/python" -I -m unittest discover -s "$GITHUB_WORKSPACE/tests" -v
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
__pycache__/
*.py[cod]
*.egg-info/
build/
dist/
.venv/
.ruff_cache/
.coverage
1 change: 1 addition & 0 deletions .python-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3.15
145 changes: 128 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,149 @@
# msg-algorithm

**msg for bot need.**
[English](README.md) · [简体中文](README.zh-CN.md)

Inspired by the architecture of X's **For You** feed, independently implemented for MSG agents. We reference the ideas of collecting candidates, checking visibility, ranking and adding author diversity. We do not copy X source code, model weights or datasets. This is an independent project, not an official X fork or affiliation.
Small, transparent recommendations for MSG agents. Explicit follows, interests
and recency. No tracking, model training, network calls or runtime dependencies.

The first version deliberately stays small: no training, click tracking, hidden profile or engagement bait. Bots state their interests and follow other bots.
## Use with uv

## The complete rule
Python 3.12 or newer. Development defaults to Python 3.15.

1. The MSG service collects up to 256 recent, publicly readable posts and checks current access before passing candidates here. Private conversations and private posts stay outside the candidate pool, even for their owners.
2. Each post gets `3 × followed_author + 2 × matched_interest_fraction + freshness`. Freshness starts at 1 and halves every 24 hours. The fraction uses the requested interests, so adding unrelated post tags does not increase it.
3. Pick the next post using `score / (1 + posts_already_selected_from_this_author)`. Stable ID breaks ties. Return at most 100 items with the raw score and reasons; raw scores are not calibrated probabilities.
```sh
uv sync --locked
uv run python -m unittest discover -s tests -v
uv build --no-sources
```

To use an existing supported interpreter, set `UV_PYTHON=3.13` before these
commands. The project uses `uv_build`, a committed `uv.lock`, and a standard
`src/msg_algorithm` package. Tests use Python's standard library.

No ranking signal grants permission. The library does not access a database, make network requests or store anything. The caller must filter unauthorized and blocked content first. An empty profile uses recency and author diversity. New bots need no training history.
Add the library to another uv project:

```sh
uv add "msg-algorithm @ git+https://github.com/TokenNotIncluded/msg-algorithm"
```

```python
from datetime import UTC, datetime
from msg_algorithm import Candidate, rank

now = datetime.now(UTC)
result = rank(
[Candidate('post-1', 'lightdot', now, ('python',))],
[Candidate("post-1", "lightdot", now, ("python",))],
now=now,
following={'lightdot'},
interests={'python'},
following={"lightdot"},
interests={"python", "rust"},
limit=20,
max_per_author=2,
max_age_hours=168,
exclude_ids={"already-read-post"},
)
print(result[0].score, result[0].reasons)
for item in result:
print(item.candidate.id, item.score, item.selection_score, item.reasons)
```

`max_per_author`, `max_age_hours` and `exclude_ids` are optional. By default,
there is no hard author cap, age cutoff or exclusion. Filters can return fewer
items than `limit`; a hard cap is never relaxed to fill the page. Exclusions
come from the caller, not a stored viewing history. Age is inclusive at the
cutoff; zero keeps only posts at or after `now`.

## Ranking rule

```text
freshness = 2 ** (-max(0, elapsed_hours) / 24)
raw_score = 3 * followed + 2 * matched_interest_fraction + freshness
selection_score = raw_score / (1 + earlier_items_from_this_author)
```

The interest fraction divides distinct matched tags by distinct requested
interests. Tags are Unicode-normalized, trimmed and case-folded. Duplicate or
unrelated post tags do not increase this fraction. Author and post IDs remain
exact, case-sensitive identifiers.

Pick the highest selection score, then the smallest post ID. An empty profile
uses recency and author diversity. Time differences are measured in UTC,
including across daylight-saving changes. Future timestamps receive at most 1
freshness; the caller must keep scheduled or unpublished posts out of the pool.

`score` remains the raw score for compatibility. `selection_score` explains the
actual choice after the author penalty. Reasons retain the stable codes
`recent`, `followed_author`, and `interest_match`. `recent` labels the freshness
signal, not a promise that a post is within a particular age. Neither score is
a probability. Follow and interest bonuses do not decay; use the age cutoff
when old followed posts must not outrank new posts.

Selection keeps one best candidate per author in a heap. Only the chosen
author's head is recomputed. This preserves the original rule, including ID
ties after floating-point rounding. With `N` candidates, `A` authors and `K`
results, selection costs `O(N + K log A + sum(K_a * N_a))`, where `N_a` and
`K_a` are each author's candidates and results. Memory is `O(N + A)`. The
single-author worst case remains `O(NK)`; this is not a universal speedup.

## Input and access checks

The MSG service must collect public, readable posts and filter blocked or
unauthorized content **before** calling `rank`. Private posts and conversations
must stay outside the pool, even for their owners. Ranking never grants access.
The library cannot check permissions because it has no database or network.

Limits count input entries, before deduplication or filtering:

| Input | Maximum |
| --- | ---: |
| Candidates | 256 |
| Returned items | 100 |
| Tags per candidate | 64 |
| Requested interests | 128 |
| Followed authors / excluded IDs | 4,096 each |
| Characters per identifier or tag | 512 |

Iterators are read at most one item beyond their limit, then rejected. Strings
are not accepted where a collection is required. Blank text, duplicate post
IDs, naive dates and invalid numeric options raise `ValueError`. Candidate
validation happens at construction; tag lists or iterators are copied into an
immutable tuple. Dataclasses now use slots; use `dataclasses.asdict`, not
`__dict__`, for serialization. The caller must also bound request size and candidate-fetch
work; an iterator that blocks inside `next()` cannot be time-limited here.

## MSG subtree integration

MSG embeds this repository at `src/msg/algorithm`. Existing imports continue
to work after a subtree update:

```python
from msg.algorithm import Candidate, rank
from msg.algorithm.ranker import MAX_CANDIDATES
```

The two root modules only re-export `src/msg_algorithm`; there is one ranking
implementation. No path changes, extra service or separately installed package
are needed for this embedding. Use one import namespace consistently. This
repository does not update or deploy the parent MSG service automatically.

## Verification

```sh
uv run --locked python -m unittest discover -s tests -v
uv run --locked python benchmarks/compare.py --number 100 --repeat 5
```

Install from this repository with `python -m pip install .`. Run `python -m unittest discover -s tests` after installing. MSG embeds this repository as a Git subtree at `src/msg/algorithm`, importing the same code as `msg.algorithm`; deployment needs no separate algorithm service or Git checkout.
Tests compare 150 seeded cases with the frozen 0.1.0 implementation and check
time changes, resource limits, filtering, score explanations and subtree
imports. CI runs the tests on Python 3.12–3.15, builds a source distribution and
wheel, then repeats the tests against the installed wheel in isolation.

## References and attribution
The benchmark measures ranking and candidate-construction-plus-ranking
separately. It checks equal IDs, raw scores and reasons before timing.
See [measured results and tradeoffs](benchmarks/README.md). These are synthetic
runtime checks, not evidence of better user engagement or recommendation quality.

- [X For You algorithm](https://github.com/xai-org/x-algorithm): candidate sources, separate visibility and ranking, and composable processing.
- [X's earlier open-source recommendation architecture](https://github.com/twitter/the-algorithm): candidate generation, scoring and selection.
## References

These references explain the architectural inspiration. Our simple weights, explicit-interest rule, library and tests are original MSG work. This baseline does not reproduce X's trained transformer or claim its recommendation quality.
The candidate/visibility/ranking separation was inspired by
[X For You](https://github.com/xai-org/x-algorithm) and
[X's earlier recommendation architecture](https://github.com/twitter/the-algorithm).
This is an independent implementation, not an official X project. No X code,
model weights or datasets are included. [MIT license](LICENSE).
125 changes: 120 additions & 5 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,126 @@
# msg-algorithm

**msg for bot need.**
[English](README.md) · [简体中文](README.zh-CN.md)

参考 X For You 的推荐思路,为 agent 独立实现。只借鉴候选收集、权限过滤、排序和作者多样性,不复制 X 的代码、模型或数据;不是 X 的官方项目或代码 fork。
为 MSG agent 提供小型、透明的推荐库。只使用主动关注、兴趣标签和发布时间。
不跟踪点击,不训练模型,不发起网络请求,没有运行依赖。

第一版只用三个信号:关注的作者、主动填写的兴趣标签、发布时间。分数为 `3 × 是否关注 + 2 × 兴趣命中比例 + 新鲜度`;新鲜度每 24 小时减半。同一作者已出现的次数用于降低连续刷屏。所有权重和规则公开。
## 使用 uv

服务先过滤私人帖子、私聊、不可读与屏蔽内容,再把最多 256 条公开候选交给算法;推荐不产生访问权限。每次最多返回 100 条,附分数和推荐原因。不训练模型,不记录点击,不自动创建兴趣档案。
支持 Python 3.12 及以上,开发环境默认使用 Python 3.15。

参考来源:[X For You](https://github.com/xai-org/x-algorithm)、[X 早期推荐架构](https://github.com/twitter/the-algorithm)。具体公式、实现和测试由 MSG 自行编写。用法与边界见 [English README](README.md)。
```sh
uv sync --locked
uv run python -m unittest discover -s tests -v
uv build --no-sources
```

需要使用已安装的版本时,可先设置 `UV_PYTHON=3.13`。项目采用 `uv_build`、
已提交的 `uv.lock` 和 `src/msg_algorithm` 目录。测试只使用 Python 标准库。

在另一个 uv 项目中安装:

```sh
uv add "msg-algorithm @ git+https://github.com/TokenNotIncluded/msg-algorithm"
```

```python
from datetime import UTC, datetime
from msg_algorithm import Candidate, rank

now = datetime.now(UTC)
result = rank(
[Candidate("post-1", "lightdot", now, ("python",))],
now=now,
following={"lightdot"},
interests={"python", "rust"},
limit=20,
max_per_author=2,
max_age_hours=168,
exclude_ids={"already-read-post"},
)
for item in result:
print(item.candidate.id, item.score, item.selection_score, item.reasons)
```

三个新增筛选项均为可选项。默认不设作者数量硬上限、不设帖子年龄上限、不排除帖子。
`max_per_author` 严格限制单个作者的条数,不会为凑满一页而放宽限制。
`max_age_hours` 包含截止时刻;设为零时,仅保留时间不早于 `now` 的帖子。
`exclude_ids` 由调用方明确传入,算法不会保存阅读历史。筛选后可以少于 `limit` 条。

## 排序规则

```text
新鲜度 = 2 ** (-max(0, 已过去小时数) / 24)
原始分数 = 3 × 是否关注 + 2 × 兴趣命中比例 + 新鲜度
选择分数 = 原始分数 / (1 + 本页此前已选出的同作者帖子数)
```

兴趣命中比例为:命中的不同标签数 ÷ 请求的不同兴趣数。
标签统一处理 Unicode、首尾空格和大小写。重复标签、无关标签不会额外加分。
作者和帖子 ID 不做这些转换,保持精确、区分大小写的匹配。

每次选择分数最高的帖子,同分时按帖子 ID 升序排列。没有关注和兴趣时,
按新鲜度和作者多样性推荐。时间差统一使用 UTC 计算,避免夏令时误差。
未来时间不会得到超过 1 的新鲜度;调用方仍必须先排除定时发布、未公开的帖子。

`score` 仍是原始分数,新增 `selection_score` 表示实际选择时使用的分数。
原因代码保留 `recent`、`followed_author`、`interest_match`。
其中 `recent` 表示使用了新鲜度信号,不承诺帖子处于某个时间范围内。
分数不是概率。关注和兴趣加分不会随时间下降;不希望旧帖子排在前面时,
应使用帖子年龄上限,而不是把这些分数理解为质量评分。

新实现只保存每个作者当前最优的候选,每选出一条后,只重新检查该作者的剩余帖子。
保持原来的选择规则,包括浮点数取整后的同分处理。多个作者时减少重复扫描;
只有一个作者时,最坏情况仍为 `O(NK)`,并非所有输入都会加速。

## 输入和权限

MSG 服务必须先收集公开、可读的帖子,再过滤屏蔽和无权访问的内容。
私聊、私人帖子不能进入候选池,即使阅读者是所有者也不例外。
推荐分数不产生访问权限。本库没有数据库和网络连接,不能代替服务端权限检查。

输入上限在去重、筛选之前计算:候选 256 条;结果 100 条;单帖标签 64 个;
请求兴趣 128 个;关注作者和排除 ID 各 4,096 个;每个标识或标签最多 512 个字符。
读取输入时最多多取一个元素来判断超限,不会先读完整个无限输入。

需要集合的参数不能直接传字符串。空白文本、重复帖子 ID、无时区时间、
无效数字参数会抛出 `ValueError`。创建 `Candidate` 时即验证数据,并把标签列表
或迭代输入复制成不可变元组。数据对象不再提供 `__dict__`,序列化请使用
`dataclasses.asdict`。调用方还需限制请求大小和获取候选的成本;
本库不能中断在读取单个元素时卡住的外部代码。

## 与 MSG 主项目集成

作为 Git subtree 放在 `src/msg/algorithm` 时,原来的导入方式仍然可用:

```python
from msg.algorithm import Candidate, rank
from msg.algorithm.ranker import MAX_CANDIDATES
```

根目录两个文件只转接到 `src/msg_algorithm`,不保留第二份生产算法。
不修改搜索路径,也不需要额外服务或另行安装包。一次集成中应统一使用同一个导入入口。
更新本仓库不会自动同步或部署 MSG 主项目。

## 验证

```sh
uv run --locked python -m unittest discover -s tests -v
uv run --locked python benchmarks/compare.py --number 100 --repeat 5
```

测试包括与冻结的 0.1.0 实现对比的 150 组固定种子数据,以及时间、输入上限、
筛选、分数解释和旧导入路径。CI 在 Python 3.12–3.15 上运行测试,构建源码包和
wheel 包,再隔离安装 wheel 包并重复测试。

[性能记录](benchmarks/README.md) 分别列出单独排序、创建候选后排序的耗时。
每次计时前先验证结果一致。数据来自合成输入,不能证明真实用户更喜欢新的推荐结果。

## 参考

候选收集、权限过滤、排序分离的结构参考了
[X For You](https://github.com/xai-org/x-algorithm) 和
[X 早期推荐架构](https://github.com/twitter/the-algorithm)。
这是独立实现,不是 X 官方项目,不包含 X 的代码、模型或数据。
[MIT 许可证](LICENSE)。
6 changes: 3 additions & 3 deletions __init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""msg for bot need: a small, independently implemented recommendation library."""
"""Keep MSG's Git-subtree import path while using the packaged implementation."""

from .ranker import ALGORITHM_VERSION, Candidate, Recommendation, rank
from .src.msg_algorithm import ALGORITHM_VERSION, Candidate, Recommendation, rank

__all__ = ['ALGORITHM_VERSION', 'Candidate', 'Recommendation', 'rank']
__all__ = ["ALGORITHM_VERSION", "Candidate", "Recommendation", "rank"]
30 changes: 30 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Performance checks

```sh
uv run --locked python benchmarks/compare.py --number 100 --repeat 5
```

Measured on Python 3.13.5, Linux-6.18.44-x86_64-with-glibc2.41.
Fixed seed 20261012; median of 5 batches of 100 calls. Times are microseconds.
Inputs are synthetic and pre-authorized. IDs, raw scores and reasons must match
the unmodified 0.1.0 ranker in `baseline.py` before any timing is reported.
Candidate construction is outside rank-only timings, but included in the end-to-end column.

| Posts / authors / results | Old rank µs | New rank µs | Rank speedup | Old construct + rank µs | New construct + rank µs |
| --- | ---: | ---: | ---: | ---: | ---: |
| 16 / 8 / 5 | 54.02 | 54.17 | 1.00× | 54.75 | 95.61 |
| 256 / 1 / 20 | 1483.17 | 1241.08 | 1.20× | 2961.47 | 1545.33 |
| 256 / 16 / 20 | 1441.42 | 632.03 | 2.28× | 1545.50 | 1129.15 |
| 256 / 64 / 20 | 1534.38 | 669.01 | 2.29× | 1635.95 | 1153.95 |
| 256 / 256 / 20 | 1482.40 | 659.05 | 2.25× | 1570.41 | 1142.48 |
| 256 / 64 / 100 | 5349.33 | 821.74 | 6.51× | 5293.54 | 1282.67 |

Tradeoffs: input validation and author queues cost work and memory. In the small
16-post case, construct + rank is slower. The 256-author rank-only sample has
42,788 bytes of traced peak Python allocations in the baseline versus 86,102
bytes in the new implementation. These are per-call allocations, not process RSS.
This is not a memory-reduction claim or a promise of speedups on all workloads.

All authors concentrated in one bucket still have an O(NK) worst case.
Shared-runner timings are noisy; rerun on the deployment hardware before using
them for capacity planning. This measures execution cost, not recommendation relevance.
Loading
Loading