TMDB 刮削的本地代理

[Verdure-oss]

感谢大佬，目前已在 Windows 环境下顺利安装并完成基础配置。在尝试使用刮削功能时发现一个关于网络代理的问题：

问题描述：
目前正在使用 v0.1.7 版本。确认本地网络环境可以正常访问 TMDB 官网，且在开启 TUN 模式（全局流量接管）时，服务端刮削功能完全正常。
但在使用常规本地代理模式（不开tun模式下）时，发现服务端发出的 TMDB 刮削请求似乎绕过了系统代理或未读取环境变量，导致刮削失败。

运行环境：
版本： v0.1.7
系统： Windows
网络情况： 浏览器可正常打开 TMDB，本地代理端口正常。

诉求：
请问在当前版本下，是否有配置文件或特定的环境变量（如 HTTP_PROXY / HTTPS_PROXY）可以强制指定服务端请求的代理路径？
如果目前尚未支持该功能，是否考虑在后续更新中增加一个代理配置项，以便在不开启 TUN 模式的情况下也能正常使用刮削功能？

再次感谢！

[HaoweiLi97]

我看下加一组指定dns，因为tmdb并没有无法访问，不用挂代理

[HaoweiLi97]

两个都加上要更好一些，谢谢建议，应该会在下一个beta版更新

[HaoweiLi97]

import { BaseScraper } from '../scrapers/base.scraper';
import { ScrapedActor, ScrapedMetadata } from '../scrapers/types';
import axios from 'axios';
import dns from 'node:dns';
import http from 'node:http';
import https from 'node:https';
import type { LookupFunction } from 'node:net';
import { prisma } from '../db';
import { deriveActorAge } from '../utils/actor';

const tmdbResolver = new dns.Resolver();
tmdbResolver.setServers(['8.8.8.8', '119.29.29.29']);

const tmdbLookup: LookupFunction = (hostname, options, callback) => {
    const family = typeof options === 'object' && options.family === 6 ? 6 : 4;
    const all = typeof options === 'object' && options.all === true;
    const resolve = family === 6 ? tmdbResolver.resolve6.bind(tmdbResolver) : tmdbResolver.resolve4.bind(tmdbResolver);

    resolve(hostname, (err, addresses) => {
        if (err || addresses.length === 0) {
            callback(err, undefined as any, family);
            return;
        }
        if (all) {
            callback(null, addresses.map((address) => ({ address, family })) as any);
            return;
        }
        callback(null, addresses[0], family);
    });
};

const tmdbAxios = axios.create({
    httpAgent: new http.Agent({ lookup: tmdbLookup }),
    httpsAgent: new https.Agent({ lookup: tmdbLookup }),
    proxy: false
});

const tmdbGet = async (url: string, config?: any) => {
    try {
        return await tmdbAxios.get(url, config);
    } catch (error: any) {
        console.warn(`[Script:TMDB] Custom DNS request failed, retrying with system DNS: ${error?.message || error}`);
        return axios.get(url, config);
    }
};

export class TMDBScraper extends BaseScraper {
    public supportedTypes: ('movie' | 'tv')[] = ['movie'];
    private readonly BASE_URL = "https://api.themoviedb.org/3";
    private readonly IMAGE_BASE_URL = "https://image.tmdb.org/t/p/original";
    private readonly IMG_W500 = "https://image.tmdb.org/t/p/w500";

    constructor() {
        super('TMDB');
    }

    private getLanguageOptions(options?: any) {
        const language = String(options?.tmdbLanguage || options?.language || 'zh-CN').trim() || 'zh-CN';
        const imageLanguage = String(options?.tmdbImageLanguage || options?.imageLanguage || language || 'zh-CN,en,null').trim() || 'zh-CN,en,null';
        const countryCode = language.split('-')[1]?.toUpperCase() || 'CN';
        return { language, imageLanguage, countryCode };
    }

    private getImageLanguagePreferences(imageLanguage: string): Array<string | null> {
        const prefs = String(imageLanguage || '')
            .split(',')
            .map((item) => item.trim())
            .filter(Boolean)
            .map((item) => item.toLowerCase());
        const out: Array<string | null> = [];
        for (const pref of prefs.length > 0 ? prefs : ['zh-cn', 'en', 'null']) {
            if (pref === 'null') {
                out.push(null);
                continue;
            }
            out.push(pref);
            const base = pref.split('-')[0];
            if (base && base !== pref) out.push(base);
        }
        return Array.from(new Set(out));
    }

    private pickImagePath(images: any[] | undefined, imageLanguage: string, fallbackPath?: string): string {
        const list = Array.isArray(images) ? images.filter((img: any) => !!img?.file_path) : [];
        const prefs = this.getImageLanguagePreferences(imageLanguage);
        const normalizeLang = (value: any) => value === null || value === undefined ? null : String(value).toLowerCase();

        for (const pref of prefs) {
            const matched = list.find((img: any) => normalizeLang(img?.iso_639_1) === pref);
            if (matched?.file_path) return matched.file_path;
        }

        return fallbackPath || list[0]?.file_path || '';
    }

    private pickPreferredTitle(data: any, language: string): string {
        const out: string[] = [];
        const push = (v: any) => {
            if (typeof v !== 'string') return;
            const s = v.trim();
            if (!s) return;
            if (!out.includes(s)) out.push(s);
        };

        const normalizedLanguage = String(language || 'zh-CN').toLowerCase();
        const languageBase = normalizedLanguage.split('-')[0];
        const translations = Array.isArray(data?.translations?.translations) ? data.translations.translations : [];
        const preferredTranslations = translations
            .filter((t: any) => String(t?.iso_639_1 || '').toLowerCase() === languageBase)
            .sort((a: any, b: any) => {
                if (languageBase !== 'zh') return 0;
                const rank = (iso: string) => {
                    if (iso === 'CN') return 0;
                    if (iso === 'TW') return 1;
                    if (iso === 'HK') return 2;
                    if (iso === 'SG') return 3;
                    return 9;
                };
                return rank(a?.iso_3166_1 || '') - rank(b?.iso_3166_1 || '');
            });

        for (const t of preferredTranslations) {
            push(t?.data?.title);
            push(t?.data?.name);
        }

        push(data?.title);
        push(data?.name);
        push(data?.original_title);
        push(data?.original_name);
        return out[0] || '';
    }

    private async getApiKey(): Promise<string> {
        try {
            // @ts-ignore
            if (typeof prisma !== 'undefined') {
                const configSetting = await (prisma as any).setting.findUnique({ where: { key: "custom_configs" } });
                if (configSetting && configSetting.value) {
                    const configs = JSON.parse(configSetting.value);
                    if (Array.isArray(configs)) {
                        const found = configs.find((c: any) => c.key === "TMDB_API_KEY");
                        if (found && found.value) return found.value;
                    }
                }
            }
            return process.env.TMDB_API_KEY || "";
        } catch (e) { return ""; }
    }

    private async fetchPersonDetail(apiKey: string, personId: number, options?: any): Promise<any | null> {
        const { language, imageLanguage } = this.getLanguageOptions(options);
        try {
            const res = await tmdbGet(`${this.BASE_URL}/person/${personId}`, {
                params: {
                    api_key: apiKey,
                    language,
                    append_to_response: 'images,external_ids',
                    include_image_language: imageLanguage
                }
            });
            return res.data;
        } catch (e) {
            return null;
        }
    }

    private async fetchMovieCredits(apiKey: string, movieId: string): Promise<any[]> {
        try {
            const res = await tmdbGet(`${this.BASE_URL}/movie/${movieId}/credits`, {
                params: { api_key: apiKey }
            });
            return Array.isArray(res.data?.cast) ? res.data.cast : [];
        } catch (e) {
            return [];
        }
    }

    private buildActorProfile(base: any, detail?: any): ScrapedActor {
        const profilePath = detail?.profile_path || base?.profile_path || '';
        const aliases = Array.isArray(detail?.also_known_as) ? detail.also_known_as.filter(Boolean) : [];
        const intro = (detail?.biography || '').trim();
        const images = Array.isArray(detail?.images?.profiles)
            ? detail.images.profiles.slice(0, 8).map((p: any) => `${this.IMG_W500}${p.file_path}`)
            : [];

        return {
            id: String(base?.id || ''),
            name: base?.name || '',
            role: base?.character || '',
            image: profilePath ? `${this.IMAGE_BASE_URL}${profilePath}` : '',
            images,
            altName: aliases[0] || undefined,
            aliases,
            nationality: detail?.place_of_birth || '',
            bornDate: detail?.birthday || '',
            age: deriveActorAge(detail?.birthday || base?.birthday || '') ?? undefined,
            intro,
            url: base?.id ? `https://www.themoviedb.org/person/${base.id}` : undefined
        };
    }

    async search(query: string, options?: any): Promise<ScrapedMetadata[]> {
        const apiKey = await this.getApiKey();
        if (!apiKey) return [];
        const endpoint = "/search/movie";
        const { language } = this.getLanguageOptions(options);
        try {
            const res = await tmdbGet(`${this.BASE_URL}${endpoint}`, { params: { api_key: apiKey, query, language } });
            return (res.data.results || []).map((item: any) => ({
                uniqueId: item.id.toString(),
                title: `${item.title || item.name} (${(item.release_date || item.first_air_date)?.split("-")[0] || "N/A"})`,
                source: "TMDB",
                type: "movie",
                poster: item.poster_path ? `${this.IMG_W500}${item.poster_path}` : ""
            }));
        } catch (e) { return []; }
    }

    async scrape(id: string, options?: any): Promise<ScrapedMetadata> {
        const apiKey = await this.getApiKey();
        if (!apiKey) throw new Error("Missing API Key");
        const { language, imageLanguage } = this.getLanguageOptions(options);

        const realId = id.toString().replace(/^(tv|movie)_/, '').split('_')[0];
        console.log(`[Script:TMDB] Raw ID: ${id}, Real ID: ${realId}`);

        const scrapeOne = async (): Promise<ScrapedMetadata> => {
            console.log(`[Script:TMDB] Scraping movie with internal ID: ${realId}`);
            const endpoint = `/movie/${realId}`;
            const res = await tmdbGet(`${this.BASE_URL}${endpoint}`, {
                params: {
                    api_key: apiKey,
                    language,
                    append_to_response: "credits,external_ids,images,content_ratings,translations",
                    include_image_language: imageLanguage
                }
            });
            const data = res.data;
            const posterPath = this.pickImagePath(data.images?.posters, imageLanguage, data.poster_path);
            const backdropPath = this.pickImagePath(data.images?.backdrops, imageLanguage, data.backdrop_path);
            const poster = posterPath ? `${this.IMAGE_BASE_URL}${posterPath}` : "";
            const backdrop = backdropPath ? `${this.IMAGE_BASE_URL}${backdropPath}` : "";
            let cast = data.credits?.cast || [];
            if (!Array.isArray(cast) || cast.length === 0) {
                cast = await this.fetchMovieCredits(apiKey, realId);
            }
            cast = cast.slice(0, 12);
            const actorDetails = await Promise.all(cast.map((a: any) => this.fetchPersonDetail(apiKey, Number(a.id), options)));
            const actorsList: ScrapedActor[] = cast.map((a: any, idx: number) => this.buildActorProfile(a, actorDetails[idx] || undefined)).filter((a: ScrapedActor) => !!a.name);

            return {
                uniqueId: id.toString(),
                title: this.pickPreferredTitle(data, language),
                originalTitle: data.original_title || data.original_name,
                year: parseInt((data.release_date || data.first_air_date)?.split("-")[0]) || 0,
                summary: data.overview || "",
                rating: data.vote_average || 0,
                directors: (data.credits?.crew?.filter((p: any) => p.job === "Director").map((d: any) => d.name) || []),
                actors: cast.map((a: any) => a.character ? `${a.name} (饰 ${a.character})` : a.name),
                actorsList,
                genres: data.genres?.map((g: any) => g.name) || [],
                posters: poster ? [poster] : [],
                fanarts: backdrop ? [backdrop] : [],
                thumbs: poster ? [poster] : [],
                source: "TMDB",
                type: "movie",
                episodes: [],
                supportsVideo: false
            };
        };

        try {
            return await scrapeOne();
        } catch (e: any) {
            console.error('[Script:TMDB] Scrape failed:', e.message);
            throw e;
        }
    }

    async searchActor(query: string, options?: any): Promise<ScrapedActor[]> {
        const apiKey = await this.getApiKey();
        if (!apiKey || !query) return [];
        const { language } = this.getLanguageOptions(options);
        try {
            const res = await tmdbGet(`${this.BASE_URL}/search/person`, {
                params: { api_key: apiKey, query, language, include_adult: true }
            });
            const list = res.data?.results || [];
            return list.slice(0, 15).map((p: any) => this.buildActorProfile(p));
        } catch {
            return [];
        }
    }

    async scrapeActor(id: string, options?: any): Promise<ScrapedActor | null> {
        const apiKey = await this.getApiKey();
        if (!apiKey || !id) return null;
        const personId = String(id).replace(/^person_/, '').trim();
        if (!/^\d+$/.test(personId)) return null;
        const detail = await this.fetchPersonDetail(apiKey, Number(personId), options);
        if (!detail) return null;
        return this.buildActorProfile({ id: personId, name: detail.name, profile_path: detail.profile_path }, detail);
    }
}

你能试一下这个刮削器吗，我在代码中加入了dns设置，如果能刮削到就没必要做其他设置了

[Verdure-oss]

我在服务端的刮削器里修改了这个刮削器代码，就是在图中的代码替换了这段代码，但好像还是刮削失败？