Repository navigation
Fixes, improvements & Fix for LiveChart Website changes - #4
Open
AkitoTheGambler wants to merge 50 commits into
Open
AkitoTheGambler wants to merge 50 commits into
AkitoTheGambler wants to merge 50 commits into
Conversation
…er if the ban persists, AniList minimum wait 0.67 s
…rk controller choice once per site
… test without password prompt
There was a problem hiding this comment.
🟡 Changes recommended
Network restarts can fail because startup skips required sudo credentials when only the aniSearch relations crawler is enabled, and root execution still builds sudo-prefixed interface commands that are rejected.
2 open issues
What changed in this PR
This PR updates crawler compatibility and provider access, improves retry handling, and adds clearer startup and progress logging.
Changes:
- Honor
Retry-After, retry AniDB bans, and adjust provider request intervals. - Move AniList downloads to its public GraphQL API without session tokens.
- Add conditional sudo validation, modern-JDK command detection, clearer logs, and example configuration.
| File | Description |
|---|---|
| lib/src/main/kotlin/io/github/manamiproject/modb/app/network/SuspendableHttpClient.kt | Identify the host in network-wait logs. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/network/NetworkControllers.kt | Detect restart requirements and deduplicate controller logs. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/network/LinuxNetworkController.kt | Select sudo usage based on password availability. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/extensions/MetaDataProviderConfigExtensions.kt | Add distinct crawler names for logging. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/downloadcontrolstate/DefaultDownloadControlStateScheduler.kt | Identify crawlers in scheduling logs. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/simkl/SimklCrawler.kt | Log skipped entries and continuation year. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/livechart/LivechartPaginationIdRangeSelector.kt | Clarify missing-page warnings. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/livechart/LivechartNewestYearDetector.kt | Correct year-detection log wording. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/livechart/LivechartCrawler.kt | Log skipped entries and remaining pages. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/IntegerBasedIdRangeSelector.kt | Summarize excluded and remaining IDs. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/anisearch/AnisearchPaginationIdRangeSelector.kt | Distinguish relations-crawler logs. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/anisearch/AnisearchCrawler.kt | Add progress logs and distinct crawler names. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/animeplanet/AnimePlanetCrawler.kt | Log skipped entries and continuation page. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/animenewsnetwork/AnimenewsnetworkCrawler.kt | Log skipped entries and remaining pages. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/anilist/AnilistHighestIdDetector.kt | Remove session-client and referer dependencies. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/anilist/AnilistCrawler.kt | Replace the session client and adjust request intervals. |
| lib/src/main/kotlin/io/github/manamiproject/modb/app/crawlers/anidb/AnidbCrawler.kt | Propagate persistent bans and increase minimum waits. |
| kommand/src/test/kotlin/io/github/manamiproject/kommand/JavaProcessBuilderTest.kt | Conditionally skip password-rejection testing. |
| kommand/src/main/kotlin/io/github/manamiproject/kommand/JavaProcessBuilder.kt | Recognize missing commands across JDK error formats. |
| core/src/test/kotlin/io/github/manamiproject/modb/core/httpclient/RetryAfterKtTest.kt | Test retry-delay parsing and invalid values. |
| core/src/main/kotlin/io/github/manamiproject/modb/core/httpclient/RetryAfter.kt | Parse seconds and date-based retry delays. |
| core/src/main/kotlin/io/github/manamiproject/modb/core/httpclient/DefaultHttpClient.kt | Honor server retry delays and improve retry logs. |
| config.example.toml | Provide starter crawler configuration. |
| app/src/main/resources/logback.xml | Shorten logger names. |
| app/src/main/kotlin/io/github/manamiproject/modb/app/App.kt | Conditionally request and validate sudo credentials. |
| anilist/src/test/kotlin/io/github/manamiproject/modb/anilist/AnilistHeaderCreatorTest.kt | Verify public-API headers without session tokens. |
| anilist/src/test/kotlin/io/github/manamiproject/modb/anilist/AnilistConfigTest.kt | Assert the public API endpoint. |
| anilist/src/main/kotlin/io/github/manamiproject/modb/anilist/AnilistHeaderCreator.kt | Simplify public-API request headers. |
| anilist/src/main/kotlin/io/github/manamiproject/modb/anilist/AnilistDownloader.kt | Use the default HTTP client without referer headers. |
| anilist/src/main/kotlin/io/github/manamiproject/modb/anilist/AnilistConfig.kt | Switch downloads to the public GraphQL endpoint. |
| anidb/src/test/kotlin/io/github/manamiproject/modb/anidb/AnidbApiDownloaderTest.kt | Test ban recovery, exhaustion, and other errors. |
| anidb/src/main/kotlin/io/github/manamiproject/modb/anidb/AnidbBannedException.kt | Introduce a dedicated ban exception. |
| anidb/src/main/kotlin/io/github/manamiproject/modb/anidb/AnidbApiDownloader.kt | Retry ban responses with bounded backoff. |
| analyzer/src/main/resources/logback.xml | Shorten analyzer logger names. |
Review effort: Balanced
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…do password if either aniSearch crawler is enabled
…bans without a stack trace
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
…rve as pending, recheck them via deadEntriesToRecheck
… though the title dump names them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
The title dump replaced the dead entries under AnidbConfig, which isn't the supported config in website mode, so the replacement was silently skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
…8+ entries via a list in website mode The website no longer writes .pending files. A pending entry goes to anidb-pending-entries.txt and is rechecked through deadEntriesToRecheck, like in API mode. Existing .pending files of the current week are moved there. 18+ entries are kept in anidb-restricted-entries.txt and skipped on every website run, until the title dump no longer names them. API mode ignores the list, because the API serves them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
ClearanceHttpClient dropped a 403 response without closing it whenever it renewed the clearance or changed the connection, which OkHttp reported as a leaked connection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
…k trace Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
A retry waits up to minutes, but was only logged at debug and trace level, so a crawler waiting for its next attempt looked like a hung one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
…e year list is missing The stack trace of a stopped crawler moves to debug level. Livechart names the url and the page title it got instead of "Unable to extract headers." Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
… any other failure The challenge page (403, or even 200) was taken as content and the crawler stopped on a missing year list. It now counts as a retry case, and the retry log names it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
The page no longer has the grouped list with year headings. The season links (/winter-2027, formerly /winter-2026/tv) are read instead, which work with either layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
The cards no longer have the class "anime" since the redesign of october 2026. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
Once all season pages had been crawled in a week, anime added to livechart in the meantime were only found the next week. The pages from the current season up to the last season of the newest year, and tba, are now requested on every run. They aren't memorized as last page and only new ids are downloaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
The values no longer follow their label as text, but sit in the next element. Type, status, duration, season, year, episodes, the next episode, related anime and tags are read from the new layout first, with the earlier selectors as fallback. Blank studio names from the navigation are dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
…e list Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
Debug is the default level of the log, so the stack trace still showed up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
Author
AniDB: consistent handling of pending, removed and 18+ entries (API and website)
Livechart: adapted to the redesign of October 2026
Logging
|
Author
|
TL;DR:
Example showing progress at start:
|
The crawler decides between dead and pending and logs that itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
OkHttp's message only names localhost:8191, which reads as if the site itself were unreachable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVzhaDVosMJJcWUJS3p5Sj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

A collection of fixes for running the crawlers on current systems, rate limiting closer to the providers' rules and clearer logging.
Rate limits and retries
DefaultHttpClientwaits as long as the server asks viaRetry-After(seconds or HTTP date) before retrying. Without the header the previous waiting time (60–90 s × attempt) is used. Applies to all providers.modb.anidb.crawler.minWaitInMilliseconds/maxWaitInMilliseconds.bannedresponse is sent with HTTP 200, so the http client never retried it. The downloader now retries it with the same backoff (60–90 s × attempt, up to 5 times). If the ban persists, the newAnidbBannedExceptionstops the AniDB crawler immediately instead of skipping up to 25 more IDs while banned.AniList: official API
The AniList crawler used the website's internal endpoint
https://anilist.co/graphqland had to load the homepage to obtain a session cookie and CSRF token. It now uses the official public APIhttps://graphql.anilist.co, which needs neither. The query and the converter are unchanged, the response is identical. Anime links in the dataset still point toanilist.co.AnilistHttpClientand the token classes are removed, since they're no longer used.kommand
error=2, No such file or directory. Newer JDKs reportExec failed, error: 2 (No such file or directory), so a plainIOExceptionwas thrown instead ofCommandNotFoundException. The check now looks forNo such file or directory.zstd-jniloads a native library, which JDK 24+ warns about. Tests and the app jar now allow native access (--enable-native-access=ALL-UNNAMED/Enable-Native-Accessmanifest entry).Startup
ifconfigis unavailable.sudo -k -S -v(3 attempts) instead of failing hours later.ifconfigis called without sudo.Logging
AnimePlanetCrawlerinstead ofi.g.m.m.a.c.a.AnimePlanetCrawler$Companion).anisearch.com-relations, matching its name inmodb.app.disabledCrawlersand its working directory.AnidbBannedExceptionstack trace..convfiles (e.g. after stopping the app during a conversion) are deleted at startup and converted again. Broken ones are skipped with a warning when the DCS files are updated. Previously both ended the run with an uncaughtjava.io.EOFException.Configuration
config.example.tomlas a starting point forconfig.toml.Testing
./gradlew buildpasses on JDK 25 (Ubuntu).Retry-Afterparsing and the AniDB ban handling. Existing tests adapted where messages or endpoints changed.https://graphql.anilist.co.🤖 Generated with Claude Code