Repository navigation
图片改由 DeepSeek 切块读 - #106
Merged
Merged
Conversation
本地的 RapidOCR 认不全图片里的截止时间和网址,#102 比较后改由 DeepSeek 切块读(#105)。 - 图片和 PDF 里的扫描页切成每块不超过约 1300×1300 像素,尽量切在没有字的行,一次请求按顺序送完;宽过 2048 像素的先缩小 - 同一张图每次读出的文字可能不一样,读过的文件按附件网址存在本机的 SQLite 里,再下载到一字节不差的文件就用上次的文字,不然通知会被当成变了;30 天没用到的删掉 - 读图的用量和这条通知的抽取一起交;DeepSeek 没读成的,这条通知这一轮算没抓成 - 删掉 RapidOCR 和 ONNX Runtime,连带 numpy、OpenCV 等 15 个包;Pillow 改成直接依赖。镜像里给存储留好目录,部署时挂一个卷
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
关联
Closes #105。设计和比较在 #104(总体设计 §5.2、§7.4、§8.6、§16),先合并 #104 再合并这个。
改了什么
#102 比较后定了图片改由 DeepSeek 切块读:9 张样本的关键信息认对 111/113,本地的 RapidOCR 是 94/113。
parse/image.py):图片和 PDF 里的扫描页切成每块不超过约 1300×1300 像素,尽量切在没有字的行。宽过 2048 像素的先缩小:大的扫描件和海报字照样看得清,也不会切出几十块,一次请求 15 张以上时 DeepSeek 只收每边 4096 像素以内的图。file_text改成file_parts,按阅读顺序返回文字和待读的图片extraction/pictures.py):一张图的各块按顺序放在一次请求里,用抽取的同一个 DeepSeek 客户端。提示词和 写下图片改由 DeepSeek 读的比较和结论 #104 实验的一样,只是放进instructions,标点照抽取的提示词改成半角(RUF001),并且总带着「几张图片是同一张长图切开的」这一句,这样前缀每次都一样crawl/readings.py):同一张图每次读出的文字可能不一样,不然通知会被当成变了。读过的文件连同读出的文字按附件网址存在本机的 SQLite 里,再下载到一字节不差的文件,就用上次的文字;直接比较字节,不算哈希;30 天没用到的删掉。只存要 DeepSeek 读的文件StepError,这条通知这一轮算没抓成,下一轮再抓;文件本身坏了的照旧记日志、文字留空。读图的用量记在附件上,和这条通知的抽取一起交(Extraction.usage),没走到抽取的只在每次调用的日志里/home/crawler/.cache/mouseclock,部署时在这里挂一个卷,Dockerfile 开头写了怎么验证的
./check.sh通过:格式、lint、Pyright 严格模式、344 个测试file_parts和ReadPicture读一次:切成 4 块,210 项全对,全角冒号照原样,120 行不重不漏;输入 3326、输出 3300 token,9.3 秒,按高峰价约 0.005 美元crawler用户和mkdir,挂一个命名卷跑两次容器:第一次用Readings写进/home/crawler/.cache/mouseclock/readings.sqlite3,换一个新容器能读回来影响
docker run ... -v <卷名>:/home/crawler/.cache/mouseclock;没挂也能跑,只是重建容器后每张图要再读一次。镜像和内存都小了,不再跑识别模型