Skip to content

README.md

Latest commit

 

History

History
232 lines (162 loc) · 9.35 KB

README.md

File metadata and controls

232 lines (162 loc) · 9.35 KB
Raw

MapleStory Worlds Resource ID Tools

This repository contains Python tools for scraping and managing resource metadata from the MapleStory Worlds API. These tools help collect GUIDs, tags, and image paths for various game assets.

Overview

The tools consist of two main scripts:

  • gen-ruids.py: Bulk scraper that discovers resources by category
  • pop-ruids.py: Targeted scraper that fetches metadata for specific GUIDs

Both scripts interact with the Nexon MapleStory Worlds API to collect resource information including:

  • Resource GUIDs (Globally Unique Identifiers)
  • Display names (tags)
  • Image file paths
  • Asset metadata

Prerequisites

Python Dependencies

python3 -m pip install httpx

asyncio is part of Python's standard library. Run the scripts from this repository's root; data paths are relative to the working directory.

Environment Setup

Set your API token as an environment variable:

export API_TOKEN="your_api_token_here"

The API token should be obtained from the MapleStory Worlds platform and is required for authenticating with the Nexon API. In the Network tab in Google Chrome (or other browser), you can obtain it from the X-Mverse-ifwt header. You must be logged into the MapleStory Worlds API to obtain this header.

Scripts

gen-ruids.py - Category-based Resource Generator

This script scrapes resources by category, discovering all available assets within specified categories. It also builds populate.json for hidden resource categories that need GUID-specific population later.

Features

  • Category-based scraping: Processes different asset categories (sprites, sounds, character parts, etc.)
  • Concurrent processing: Uses async/await with configurable concurrency for efficient API usage
  • Resume capability: Tracks completed pages and can resume interrupted scraping sessions while always re-fetching the last completed page to catch tail-page additions

Usage

python gen-ruids.py

Configuration

Edit the CATEGORIES dictionary in the script to enable/disable specific categories:

CATEGORIES = {
    'sprite': '0',           # Enable sprite category
    # 'sound': '1,19',      # Uncomment to enable sound category
    # 'hair': '25,27',      # Uncomment to enable hair category
    # Add other categories as needed
}

Output Files

For each category (e.g., 'sprite'):

  • tags/sprite_tags.json: Maps display names to GUIDs
  • guids/sprite_guids.json: Maps GUIDs to image paths
  • done/sprite_done.json: Tracks completed pages for resume functionality

For populate-only categories (for example chatballoon, nametag, damageskin):

  • populate.json: Maps GUIDs to category names used by pop-ruids.py
  • done/populate_<category>_done.json: Tracks completed populate-manifest pages

pop-ruids.py - GUID-specific Resource Populator

This script fetches detailed metadata for specific GUIDs listed in populate.txt, discovered in populate.json, and auto-detected from category tag stores when a GUID is missing from the matching guids/*_guids.json file.

This is useful for categories that need GUID-specific metadata lookup, including:

  • Name tag RUIDs
  • Chat balloon RUIDs
  • Tileset RUIDs

Usage

  1. Run gen-ruids.py when you want to refresh populate.json for hidden categories.

  2. Optionally create or edit populate.txt with additional GUIDs to process:

# This is a comment - lines starting with # are ignored
guid-1234-5678-9abc-def0
guid-abcd-efgh-ijkl-mnop
# Another comment
guid-qrst-uvwx-yz12-3456
  1. Run the script:
python pop-ruids.py

To limit processing to specific manifest categories:

python pop-ruids.py --category back
python pop-ruids.py --category back,object
python pop-ruids.py -c back -c object

Output Files

  • tags/<category>_tags.json: Manifest-backed GUIDs are written to their category file, such as tags/back_tags.json
  • guids/<category>_guids.json: Manifest-backed GUIDs are written to their category file, such as guids/back_guids.json
  • tags/populate_tags.json and guids/populate_guids.json: Fallback outputs for GUIDs that come only from populate.txt and have no category in populate.json

When a GUID has category metadata in populate.json, pop-ruids.py preserves the API tag name when it already matches the category, such as portal-1. pop-ruids.py processes the union of GUIDs from populate.txt, populate.json, and any tags/*_tags.json entries whose GUID is still missing from the matching guids/*_guids.json. The effective priority order is populate.txt first, then missing GUIDs discovered from category tag stores, then the remaining populate.json GUIDs. During preprocessing, pop-ruids.py logs category-store scanning and candidate filtering progress so long startup phases do not appear stalled. When --category is provided, only GUIDs whose discovered category matches the requested values are processed, whether that category came from populate.json or an existing tag store.

Configuration

Performance Tuning

Both scripts include configurable parameters:

COUNT = 100          # Items per API request page
CONCURRENCY = 1      # Number of concurrent requests
TIMEOUT_SEC = 15.0   # Request timeout in seconds
MAX_QPS = 2.0        # Shared API request cap within one process

COUNT is defined in gen-ruids.py; the other settings are in maplestory_api.py. All outbound Nexon API GET requests, including page-count probes, use the shared 2 QPS limiter within each process. Separate scraper processes do not share a rate limit.

Tracked Registry and Local Outputs

guids/*_guids.json, tags/*_tags.json, and populate.json are intentionally tracked exceptions to ignoring generated files:

  • Sibling MSW-Tools/tools/guid.go reads ../MSW-RUID/guids/ when rebuilding data/guid/all_guids.json. Its GUID matching and tools/spine.go also read skeleton_guids.json directly for skeleton and Spine metadata.
  • tags/ preserves API display names used by pop-ruids.py for name repair, missing-GUID discovery, and avoiding repeated requests. Those names cannot be reconstructed from GUID-to-path mappings alone.
  • populate.json preserves discovered GUID/category pairs, including GUIDs absent from the tag stores and categories currently disabled in POPULATE_CATEGORIES.

These are accumulated snapshots from an authenticated, mutable external API. The scripts merge into existing stores; neither a fresh scrape nor the current default category list guarantees faithful reconstruction. Keep these files tracked until an equivalent versioned snapshot and an offline restore path are available to consumers. Refresh with gen-ruids.py followed by pop-ruids.py using an API token, then review the registry diff; this is a live refresh, not a deterministic rebuild.

done/ contains ignored, local page checkpoints. If restoring older registry files or starting a fresh scrape, move the corresponding checkpoints aside so completed pages are fetched again. Python caches (including .pycache_tmp/) are also ignored. populate.txt is optional authored input, not generated output.

Offline Tests

python3 -B -m unittest discover -s tests -v

The tests use in-memory data, require httpx for module imports, and make no API requests.

Directory Structure

ruid/
├── maplestory_api.py         # Shared API utilities module
├── gen-ruids.py              # Category-based scraper
├── pop-ruids.py              # GUID-specific scraper
├── populate.json             # GUID->category manifest for populate flow
├── populate.txt              # Input file for pop-ruids.py
├── tags/                     # Tag-to-GUID mappings
│   ├── sprite_tags.json
│   ├── back_tags.json
│   └── populate_tags.json
├── guids/                    # GUID-to-path mappings
│   ├── sprite_guids.json
│   ├── back_guids.json
│   └── populate_guids.json
└── done/                     # Progress tracking
    └── sprite_done.json

API Reference

MapleStory Worlds API

The scripts interact with the Nexon MapleStory Worlds API:

  • Base URL: https://mverse-api.nexon.com/resource/v1/search
  • Authentication: API token passed directly in the X-Mverse-ifwt header
  • Rate Limiting: 2 requests per second per process by default

Data Format

Tags JSON Format

{
  "sprite-example-001": "guid-1234-5678-9abc-def0",
  "sprite-example-002": "guid-abcd-efgh-ijkl-mnop"
}

GUIDs JSON Format

{
  "guid-1234-5678-9abc-def0": "character/face/001.img/default",
  "guid-abcd-efgh-ijkl-mnop": "map/background/forest.img"
}

Error Handling

Requests have timeouts; failed or invalid responses are skipped, without automatic retries. Re-run to retry unfinished pages or GUIDs. Both scripts attempt to save accumulated progress after interruption or a processing error.

Contributing

Feel free to raise issues in case any logic is incorrect.

License

The tool code is licensed under the MIT License.

Troubleshooting

HTTP timeout errors / JSON decode errors

  • Check network connectivity, response status, and whether API_TOKEN is valid.
  • For rate limiting, lower MAX_QPS in maplestory_api.py and avoid running multiple scrapers concurrently.