Skip to content

Fixes, improvements & Fix for LiveChart Website changes - #4

Open
AkitoTheGambler wants to merge 50 commits into
cedya77:mainfrom
AkitoTheGambler:fixes-and-improvements
Open

AkitoTheGambler wants to merge 50 commits into
cedya77:mainfrom
AkitoTheGambler:fixes-and-improvements

Conversation

@AkitoTheGambler

@AkitoTheGambler AkitoTheGambler commented Oct 28, 2026 •

Copy link
Copy Markdown

A collection of fixes for running the crawlers on current systems, rate limiting closer to the providers' rules and clearer logging.

Rate limits and retries

  • Retry-After: DefaultHttpClient waits as long as the server asks via Retry-After (seconds or HTTP date) before retrying. Without the header the previous waiting time (60–90 s × attempt) is used. Applies to all providers.
  • AniDB
    • Minimum wait between API requests raised from 1.5 s to 2 s, which is AniDB's documented limit of one page every two seconds. Still configurable via modb.anidb.crawler.minWaitInMilliseconds / maxWaitInMilliseconds.
    • A banned response is sent with HTTP 200, so the http client never retried it. The downloader now retries it with the same backoff (60–90 s × attempt, up to 5 times). If the ban persists, the new AnidbBannedException stops the AniDB crawler immediately instead of skipping up to 25 more IDs while banned.
    • Entries which the title dump names but the API doesn’t serve yet (pending approval) are kept in a separate pending list instead of the dead entries. They are skipped, and modb.app.crawler.deadEntriesToRecheck rechecks them for AniDB instead of dead entries, which the dump confirms as non-existent.
  • AniList: minimum wait decreased to 0.67 s (90 requests per minute, AniList's documented limit).

AniList: official API

The AniList crawler used the website's internal endpoint https://anilist.co/graphql and had to load the homepage to obtain a session cookie and CSRF token. It now uses the official public API https://graphql.anilist.co, which needs neither. The query and the converter are unchanged, the response is identical. Anime links in the dataset still point to anilist.co.

AnilistHttpClient and the token classes are removed, since they're no longer used.

kommand

  • Missing commands were only recognised by the exact message error=2, No such file or directory. Newer JDKs report Exec failed, error: 2 (No such file or directory), so a plain IOException was thrown instead of CommandNotFoundException. The check now looks for No such file or directory.
  • The test for an incorrect sudo password is skipped if sudo doesn't ask for a password (passwordless or cached sudo), because the password is never checked there.
  • zstd-jni loads a native library, which JDK 24+ warns about. Tests and the app jar now allow native access (--enable-native-access=ALL-UNNAMED / Enable-Native-Access manifest entry).

Startup

  • The sudo password is only requested if the network device will actually be restarted: not when running as root, not when proxies are configured for AniDB and aniSearch, not when ifconfig is unavailable.
  • If it is requested, it is verified immediately with sudo -k -S -v (3 attempts) instead of failing hours later.
  • When running as root, ifconfig is called without sudo.

Logging

  • Progress at start
    • ID-based crawlers (AniDB, AniList, Kitsu, MAL) log the highest ID and how many IDs are skipped (dead, not scheduled, already downloaded) and left to download.
    • Page-based crawlers (aniSearch, anime-planet, ANN, LiveChart, Simkl) log already downloaded entries and the page they continue from.
  • Readability
    • Logger names shortened to the class name (AnimePlanetCrawler instead of i.g.m.m.a.c.a.AnimePlanetCrawler$Companion).
    • The aniSearch relations crawler logs as anisearch.com-relations, matching its name in modb.app.disabledCrawlers and its working directory.
  • Clearer messages
    • Site names added to messages which didn't contain one (DCS scheduler, LiveChart 404, network wait, HTTP 103).
    • The network controller choice is logged once per site instead of once per class.
    • LiveChart logs "Detecting newest year" instead of "Fetching highest id".
    • An AniDB ban that outlasts the retries stops the crawler with a single log line instead of an AnidbBannedException stack trace.
    • Empty .conv files (e.g. after stopping the app during a conversion) are deleted at startup and converted again. Broken ones are skipped with a warning when the DCS files are updated. Previously both ended the run with an uncaught java.io.EOFException.

Configuration

  • New config.example.toml as a starting point for config.toml.
  • The number of retries is now configurable: modb.core.httpclient.maxRetries for all sites and modb.anidb.banRetries for AniDB bans. Both default to 5.
  • If no config.toml is found, the app offers to create one from config.example.toml, with the directories set to subfolders of the current folder, instead of failing in the logback setup.

Testing

  • ./gradlew build passes on JDK 25 (Ubuntu).
  • Tests added for Retry-After parsing and the AniDB ban handling. Existing tests adapted where messages or endpoints changed.
  • The AniList query was verified against https://graphql.anilist.co.
  • Used for a full crawl run.

🤖 Generated with Claude Code

Copilot AI balanced review requested due to automatic review settings October 28, 2026 01:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Network restarts can fail because startup skips required sudo credentials when only the aniSearch relations crawler is enabled, and root execution still builds sudo-prefixed interface commands that are rejected.

2 open issues
What changed in this PR

This PR updates crawler compatibility and provider access, improves retry handling, and adds clearer startup and progress logging.

Changes:

  • Honor Retry-After, retry AniDB bans, and adjust provider request intervals.
  • Move AniList downloads to its public GraphQL API without session tokens.
  • Add conditional sudo validation, modern-JDK command detection, clearer logs, and example configuration.
File Description
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​network/​SuspendableHttpClient.kt Identify the host in network-wait logs.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​network/​NetworkControllers.kt Detect restart requirements and deduplicate controller logs.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​network/​LinuxNetworkController.kt Select sudo usage based on password availability.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​extensions/​MetaDataProviderConfigExtensions.kt Add distinct crawler names for logging.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​downloadcontrolstate/​DefaultDownloadControlStateScheduler.kt Identify crawlers in scheduling logs.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​simkl/​SimklCrawler.kt Log skipped entries and continuation year.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​livechart/​LivechartPaginationIdRangeSelector.kt Clarify missing-page warnings.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​livechart/​LivechartNewestYearDetector.kt Correct year-detection log wording.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​livechart/​LivechartCrawler.kt Log skipped entries and remaining pages.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​IntegerBasedIdRangeSelector.kt Summarize excluded and remaining IDs.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​anisearch/​AnisearchPaginationIdRangeSelector.kt Distinguish relations-crawler logs.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​anisearch/​AnisearchCrawler.kt Add progress logs and distinct crawler names.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​animeplanet/​AnimePlanetCrawler.kt Log skipped entries and continuation page.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​animenewsnetwork/​AnimenewsnetworkCrawler.kt Log skipped entries and remaining pages.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​anilist/​AnilistHighestIdDetector.kt Remove session-client and referer dependencies.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​anilist/​AnilistCrawler.kt Replace the session client and adjust request intervals.
lib/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​crawlers/​anidb/​AnidbCrawler.kt Propagate persistent bans and increase minimum waits.
kommand/​src/​test/​kotlin/​io/​github/​manamiproject/​kommand/​JavaProcessBuilderTest.kt Conditionally skip password-rejection testing.
kommand/​src/​main/​kotlin/​io/​github/​manamiproject/​kommand/​JavaProcessBuilder.kt Recognize missing commands across JDK error formats.
core/​src/​test/​kotlin/​io/​github/​manamiproject/​modb/​core/​httpclient/​RetryAfterKtTest.kt Test retry-delay parsing and invalid values.
core/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​core/​httpclient/​RetryAfter.kt Parse seconds and date-based retry delays.
core/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​core/​httpclient/​DefaultHttpClient.kt Honor server retry delays and improve retry logs.
config.example.toml Provide starter crawler configuration.
app/​src/​main/​resources/​logback.xml Shorten logger names.
app/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​app/​App.kt Conditionally request and validate sudo credentials.
anilist/​src/​test/​kotlin/​io/​github/​manamiproject/​modb/​anilist/​AnilistHeaderCreatorTest.kt Verify public-API headers without session tokens.
anilist/​src/​test/​kotlin/​io/​github/​manamiproject/​modb/​anilist/​AnilistConfigTest.kt Assert the public API endpoint.
anilist/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​anilist/​AnilistHeaderCreator.kt Simplify public-API request headers.
anilist/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​anilist/​AnilistDownloader.kt Use the default HTTP client without referer headers.
anilist/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​anilist/​AnilistConfig.kt Switch downloads to the public GraphQL endpoint.
anidb/​src/​test/​kotlin/​io/​github/​manamiproject/​modb/​anidb/​AnidbApiDownloaderTest.kt Test ban recovery, exhaustion, and other errors.
anidb/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​anidb/​AnidbBannedException.kt Introduce a dedicated ban exception.
anidb/​src/​main/​kotlin/​io/​github/​manamiproject/​modb/​anidb/​AnidbApiDownloader.kt Retry ban responses with bounded backoff.
analyzer/​src/​main/​resources/​logback.xml Shorten analyzer logger names.

Review effort: Balanced

AkitoTheGambler and others added 21 commits October 29, 2026 22:18
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
…rve as pending, recheck them via deadEntriesToRecheck
… though the title dump names them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
The title dump replaced the dead entries under AnidbConfig, which isn't the
supported config in website mode, so the replacement was silently skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
…8+ entries via a list in website mode

The website no longer writes .pending files. A pending entry goes to
anidb-pending-entries.txt and is rechecked through deadEntriesToRecheck, like
in API mode. Existing .pending files of the current week are moved there.

18+ entries are kept in anidb-restricted-entries.txt and skipped on every
website run, until the title dump no longer names them. API mode ignores the
list, because the API serves them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
ClearanceHttpClient dropped a 403 response without closing it whenever it
renewed the clearance or changed the connection, which OkHttp reported as a
leaked connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
A retry waits up to minutes, but was only logged at debug and trace level, so
a crawler waiting for its next attempt looked like a hung one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
…e year list is missing

The stack trace of a stopped crawler moves to debug level. Livechart names the
url and the page title it got instead of "Unable to extract headers."

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
… any other failure

The challenge page (403, or even 200) was taken as content and the crawler
stopped on a missing year list. It now counts as a retry case, and the retry
log names it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
The page no longer has the grouped list with year headings. The season links
(/winter-2027, formerly /winter-2026/tv) are read instead, which work with
either layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
The cards no longer have the class "anime" since the redesign of october 2026.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
Once all season pages had been crawled in a week, anime added to livechart in
the meantime were only found the next week. The pages from the current season
up to the last season of the newest year, and tba, are now requested on every
run. They aren't memorized as last page and only new ids are downloaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
The values no longer follow their label as text, but sit in the next element.
Type, status, duration, season, year, episodes, the next episode, related
anime and tags are read from the new layout first, with the earlier selectors
as fallback. Blank studio names from the navigation are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
Debug is the default level of the log, so the stack trace still showed up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
@AkitoTheGambler

Copy link
Copy Markdown
Author

AniDB: consistent handling of pending, removed and 18+ entries (API and website)

  • Pending entries (addition not approved yet) go to anidb-pending-entries.txt in both modes and are rechecked via deadEntriesToRecheck (longest unchecked first). The website no longer writes .pending files; existing ones of the current week are moved to the list.
    • API: an entry the API refuses but the title dump names is pending, otherwise dead.
    • Website: the page states it directly.
  • Removed entries become dead entries in both modes.
  • 18+ entries (website only, the page shows no data without an account) are no longer marked dead. They are kept in anidb-restricted-entries.txt and skipped on every run until the title dump no longer names them. API mode ignores this list.
  • The title dump now also derives the dead entries in website mode (it was silently skipped there before).
  • Closes refused responses in the clearance client (fixes OkHttp's "connection leaked" warning). AniDB's maximum wait between requests is 3 s.

Livechart: adapted to the redesign of October 2026

  • Newest year: read from the season links of /charts instead of the removed year headings.
  • Season pages: entries are read from any card with data-anime-id.
  • Anime pages: type, status, run time, season, premiere, episodes, the next episode, related anime and tags are read from the new layout, with the previous selectors as fallback.
  • Cloudflare challenge pages (also those served with 200) are recognised and retried like other failures.
  • The current and upcoming seasons and tba are rechecked on every run, so new anime are found during the week and not only on the next weekly pass.

Logging

  • Every retry wait is logged at info level with URL, duration, attempt and reason (it looked like a hang before).
  • A stopped crawler logs a single line; the stack trace is only logged at trace level.

@AkitoTheGambler

AkitoTheGambler commented Oct 30, 2026 •

Copy link
Copy Markdown
Author

TL;DR:

  • Set AniDB minimum to 2s to decrease the chance of getting banned.
  • AniDB now uses the data dump file to create the dead entries list instead, making it able to detect dead IDs without requesting every ID via API.
  • Also uses a new pending list for entries that are set to pending and give an "Anime not found" response from the API.
  • Decreased minimum rate limits for AniList to the ones documented on AniList for faster completion.
  • For APIs returning a Retry-After header, it waits the indicated time instead of the default min/max rate limit to not trigger timeout resets.
  • Changed AniList to the correct public URI for the API, so token and refresh are no longer required.
  • Fixed kommand test failing when running gradlew build.
  • Check sudo and verify password at startup, and skip if already running as root.
  • New config file template, and instead of giving a Java error if none is found, it asks to create one from the template.
  • Updated some commands giving out deprecation warnings, so it will be compatible with future versions of Gradle and JDK.
  • Improved logging to be clearer and newly listing current progress and items left to download.
  • Also, some logs were not showing which crawler was logging them.
  • And some exceptions now get caught and logged nicer.

Example showing progress at start:

03:03:03.508 [Network-Dispatcher-24] INFO IntegerBasedIdRangeSelector - [anilist.co] highest ID is [219063]. Skipping [150461] dead entries, [0] entries not scheduled for this week and [15850] already downloaded. [52752] of [219063] IDs left to download.
  • AniDB crawler can now correctly detect and track dead, removed, pending and 18+ entries independently, so it doesn't need to retry on later runs and prevent bans due to requesting dead ids multiple times.
  • LiveChart website layout changed, and this works now with the new version of the website.
    @cedya77

@AkitoTheGambler AkitoTheGambler changed the title Fixes and improvements Fixes, improvements & Fix for LiveChart Website changes Oct 30, 2026
AkitoTheGambler and others added 3 commits October 30, 2026 22:49
The crawler decides between dead and pending and logs that itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
OkHttp's message only names localhost:8191, which reads as if the site itself
were unreachable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants