Skip to content

Latest commit

Β 

History

History
1019 lines (783 loc) Β· 37.8 KB

File metadata and controls

1019 lines (783 loc) Β· 37.8 KB

🀝 Contributing to yt-dlp

Thank you for your interest in contributing! This guide will help you understand our codebase conventions and write code that feels like it belongs here. Every rule exists because it's already applied consistently across the entire codebase β€” please follow them to keep things uniform.


πŸ“‹ Table of Contents


πŸš€ Getting Started

Prerequisites

  • Rust (edition 2024) β€” install via rustup
  • Rust nightly (for rustfmt) β€” rustup toolchain install nightly --component rustfmt
  • cargo-hack β€” cargo install cargo-hack
  • cargo-deny β€” cargo install cargo-deny

Running the checks

Every PR must pass these commands:

# Lint all features combined (all backends in a single pass)
cargo clippy --workspace --all-features -- -D warnings

# Check formatting (requires nightly)
cargo +nightly fmt --all -- --check

# Run all doc-tests (workspace-wide)
cargo test --doc --workspace --all-features

# Check dependencies (licenses, advisories, bans)
cargo deny check

# Check for unused dependencies
cargo machete

Branch workflow

  1. Fork the repository and create a branch from develop
  2. Make your changes following the guidelines below
  3. Run the verification checks above
  4. Open a PR against develop

πŸ—οΈ Project Architecture

The codebase is a Cargo workspace with two crates. Understanding the layout is essential before making changes:

yt-dlp/
β”œβ”€β”€ Cargo.toml               ← workspace manifest ([workspace] + [package])
β”œβ”€β”€ src/                     ← yt-dlp crate source
└── crates/
    └── media-seek/          ← standalone container index parsing crate
        β”œβ”€β”€ Cargo.toml
        └── src/
            β”œβ”€β”€ lib.rs       β€” RangeFetcher trait + parse() dispatch
            β”œβ”€β”€ error.rs     β€” Error enum + Result<T> alias
            β”œβ”€β”€ detect.rs    β€” magic-byte format detection
            β”œβ”€β”€ index.rs     β€” ContainerIndex, SegmentEntry, Inner
            β”œβ”€β”€ audio/       β€” mp3, ogg, flac, pcm (wav+aiff), adts
            └── video/       β€” mp4, webm, flv, avi, ts

The yt-dlp crate module hierarchy:

src/
β”œβ”€β”€ lib.rs              # 🏠 Crate root β€” Downloader struct lives here (NOT in a submodule)
β”œβ”€β”€ prelude.rs          # πŸ“€ Convenience re-exports for `use yt_dlp::prelude::*`
β”œβ”€β”€ macros.rs           # 🧩 Macros: youtube!, ytdlp_args!, install_libraries!, ternary!
β”œβ”€β”€ error.rs            # 🚨 Single unified Error enum + type Result<T>
β”‚
β”œβ”€β”€ client/             # πŸ”§ Builder, download builder, proxy, deps, stream orchestration
β”‚   β”œβ”€β”€ builder.rs      #    DownloaderBuilder (fluent builder)
β”‚   β”œβ”€β”€ download_builder.rs  # DownloadBuilder<'a> (fluent download API)
β”‚   β”œβ”€β”€ proxy.rs        #    ProxyConfig, ProxyType
β”‚   β”œβ”€β”€ deps/           #    πŸ“¦ Auto-installation of yt-dlp & ffmpeg from GitHub releases
β”‚   └── streams/        #    🧩 Format selection (VideoSelection trait), orchestration
β”‚
β”œβ”€β”€ download/           # πŸ“₯ DownloadManager, Fetcher, segment-based parallel downloads
β”œβ”€β”€ events/             # πŸ”” EventBus, DownloadEvent, EventFilter, hooks, webhooks
β”œβ”€β”€ executor/           # βš™οΈ Process runner, FfmpegArgs builder, temp-file+rename
β”œβ”€β”€ extractor/          # πŸ“‘ VideoExtractor trait, Youtube & Generic extractors
β”œβ”€β”€ metadata/           # 🏷️ MP3/MP4/FFmpeg/Lofty metadata writing, chapter injection
β”œβ”€β”€ model/              # πŸ“Š Data types: Video, Format, Chapter, Playlist, Caption, etc.
β”‚   β”œβ”€β”€ utils/          #    Serde helpers
β”‚   └── selector.rs     #    VideoQuality, AudioQuality, StoryboardQuality enums
β”œβ”€β”€ cache/              # πŸ” VideoCache, DownloadCache, PlaylistCache (feature-gated)
β”‚   └── backend/        #    Backend trait + implementations (memory/moka, json, redb, redis)
β”œβ”€β”€ live/               # πŸ”΄ Live recording/streaming (features: live-recording, live-streaming)
β”‚   β”œβ”€β”€ hls.rs          #    HLS manifest parsing via m3u8-rs
β”‚   β”œβ”€β”€ recording.rs    #    Reqwest-based HLS segment recorder (primary)
β”‚   └── ffmpeg_recording.rs  # FFmpeg-based recorder (fallback)
β”œβ”€β”€ stats/              # πŸ“Š StatisticsTracker, GlobalSnapshot (feature: statistics)
└── utils/              # πŸ› οΈ fs, http, platform, retry, validation, url_expiry, subtitle

πŸ“ Module conventions

Rule Example
Each directory has a mod.rs that declares submodules and re-exports public types pub use video::VideoCache; in cache/mod.rs
lib.rs re-exports the most-used types to crate root pub use client::{DownloadBuilder, DownloaderBuilder};
prelude.rs re-exports everything for basic usage Feature-gated with #[cfg(feature = "...")]
Module-level //! doc comments on every mod.rs Describes the module's purpose and architecture
Feature-gated modules in lib.rs #[cfg(feature = "statistics")] pub mod stats;

πŸ‘οΈ Visibility rules

Visibility When to use Example
pub Types and methods exposed to library users pub fn fetch_video_infos(...)
pub(crate) All fields of Downloader, internal helpers pub(crate) youtube_extractor: Youtube
Private Implementation details fn audio_codec_for_mux(...)

πŸ’‘ Builder struct fields are always private. TypedBuilder config struct fields are always pub.


✍️ Code Style

🌍 Language

All comments, docs, variable names, error messages, and log messages must be in English. No exceptions.

πŸ“₯ Imports

// βœ… GOOD β€” All imports at the top of the file
use crate::error::Result;
use crate::model::Video;
use std::path::PathBuf;

#[cfg(target_os = "windows")]
use std::os::windows::process::CommandExt;

// ❌ BAD β€” Never import inside function bodies
fn my_function() {
    use std::collections::HashMap; // WRONG
}

🧩 Exception: inside macro_rules! definitions, $crate:: paths may require local imports.

🏷️ Naming conventions

Item Convention Example
Variables & functions snake_case download_video, is_ready
Types & structs PascalCase DownloaderBuilder, VideoQuality
Constants SCREAMING_SNAKE_CASE DEFAULT_RETRY_ATTEMPTS, FORMAT_URL_LIFETIME
Constants prefix Context prefix DEFAULT_, CONSERVATIVE_, BALANCED_, AGGRESSIVE_
Booleans Intent-driven is_ready, has_data, include_full_data

πŸ”€ Conditional logic

No more than two raw conditions directly in an if (or while) guard. When three or more sub-expressions are combined with && or ||, each sub-expression must first be bound to a short, descriptively-named let boolean before the guard. Boolean variable names must be short and intent-revealing: is_year, is_endlist, is_timeout, etc.

// βœ… single condition β€” OK
if probe.len() < 4 { … }

// βœ… two raw conditions combined β€” OK
if e.starts_with("HTTP 4") && !e.starts_with("HTTP 429") { … }

// βœ… named booleans combined β€” OK (required when β‰₯ 3 conditions)
let is_timeout = error.is_timeout();
let is_connect = error.is_connect();
let is_request = error.is_request();
if is_timeout || is_connect || is_request { … }

// ❌ three or more raw expressions inline β€” NOT OK
if error.is_timeout() || error.is_connect() || error.is_request() { … }

🚫 Lint suppressions

#[allow(…)] attributes are forbidden in this codebase, with one explicit exception:

  • #[allow(clippy::large_enum_variant)] on DownloadEvent β€” boxing all variants for one large variant would add unnecessary indirection throughout the event system.

Fix the root cause instead of suppressing the lint:

Lint Preferred fix
dead_code Remove the item, or gate with #[cfg(feature = "…")]
unreachable_code Use unreachable!("…") or gate the fallback with #[cfg(not(…))]
clippy::too_many_arguments Group related parameters into a dedicated struct
unused_* Remove unused imports/variables, or prefix with _ for intentional non-use

πŸͺ† Nesting depth

Maximum two levels of nesting inside any function body. Each loop (for, while, loop), conditional (if, else if, match), or closure that contains control flow counts as one level. Exceeding two levels raises the SonarCloud Cognitive Complexity above the enforced threshold of 15 and will block your PR.

When a third level is needed, extract the inner logic into a private helper function that returns an Option, Result, or a dedicated struct.

// ❌ BAD β€” three levels of nesting (loop β†’ if β†’ if)
fn scan_tags(probe: &[u8]) {
    while let Some(tag) = next_tag(probe) {           // level 1
        if tag.kind == TagKind::Video {               // level 2
            if tag.frame_type == FrameType::Key {     // level 3 ← NOT allowed
                keyframes.push(tag.offset);
            }
        }
    }
}

// βœ… GOOD β€” max two levels; the inner predicate is extracted
fn is_video_keyframe(tag: &Tag) -> bool {
    tag.kind == TagKind::Video && tag.frame_type == FrameType::Key
}

fn scan_tags(probe: &[u8]) {
    while let Some(tag) = next_tag(probe) {           // level 1
        if is_video_keyframe(&tag) {                  // level 2
            keyframes.push(tag.offset);
        }
    }
}

The same rule applies to match arms that contain their own if/loop/match:

// ❌ BAD β€” match arm body itself opens a new level
match block_type {
    BlockType::StreamInfo => {
        if block_len >= MIN_SIZE {    // level 3 when already inside a loop + match
            parse_stream_info(block);
        }
    }
}

// βœ… GOOD β€” delegate to a helper that handles the guard internally
match block_type {
    BlockType::StreamInfo => parse_stream_info(block), // helper does its own guard
}
Rule Detail
Hard limit 2 nesting levels per function
What counts for, while, loop, if/else if/else, match, closures with control flow
Remedy Extract inner body into a private fn, or use early-return / guard-clause patterns
SonarCloud Max Cognitive Complexity per function: 15

🎯 Parameter types

Use the most appropriate type for public API parameters:

// βœ… GOOD β€” Flexible public API
pub fn new(url: impl Into<String>) -> Self { ... }
pub fn with_cookies(mut self, path: impl Into<PathBuf>) -> Self { ... }
pub fn input(mut self, path: impl AsRef<str>) -> Self { ... }

// ❌ BAD β€” Too restrictive
pub fn new(url: String) -> Self { ... }
pub fn new(url: &str) -> Self { ... }

For internal functions, use the most optimized type for the operations applied:

  • &str if you only read the string
  • String if you need ownership
  • &Path if you only read the path
  • PathBuf if you need ownership

πŸ§ͺ Testing

There are no #[cfg(test)] modules in src/. No tests live in tests/common/ (only shared helpers).

Test harnesses β€” three separate binaries under tests/:

Harness Command Scope
Unit cargo test --test unit --all-features Pure logic, no I/O, no network
Integration cargo test --test integration --all-features wiremock servers, tempdir I/O, async flows
E2E cargo test --test e2e --all-features -- --test-threads=1 Full download pipeline with wiremock
Doctests cargo test --doc --workspace Code examples in rustdoc

Directory conventions β€” test directories mirror src/ module hierarchy:

tests/unit/model/       ← matches src/model/
tests/unit/download/    ← matches src/download/
tests/integration/cache/ ← matches src/cache/

Create a subdirectory when a domain has β‰₯ 2 test files.

Adding a new test:

  1. Create the test file in the appropriate subdirectory (e.g. tests/unit/download/new_test.rs)
  2. Register it in the harness entry point (tests/unit.rs) with #[path = "unit/download/new_test.rs"] mod new_test;
  3. Feature-gated tests use #[cfg(feature = "...")] on the module declaration in the entry point

Conventions:

  • Test names follow fn verb_noun_condition() (e.g. fn parse_format_returns_video_type())
  • All test output goes to tempfile::tempdir(), never to project root
  • Use assert_matches! for error variant checks, pretty_assertions for struct comparisons
  • Mock servers use wiremock::MockServer (dev-dependency)
  • Fixtures: JSON in tests/fixtures/json/, media in tests/fixtures/media/
  • πŸ“Š Benchmarks β€” benches/benchmarks.rs with criterion
  • πŸ§ͺ Integration examples β€” examples/ directory

πŸ”’ Magic Numbers & Constants

Never use raw numeric or byte literals in logic. Every literal must be extracted to a named const at the top of the file.

// βœ… GOOD β€” Named constants with clear intent
/// ID3v2 header fixed size in bytes.
const ID3V2_HEADER_SIZE: usize = 10;
/// Maximum bytes to scan for the first sync word.
const SYNC_SEARCH_LIMIT: usize = 8192;

fn skip_id3(data: &[u8]) -> usize {
    if data.len() < ID3V2_HEADER_SIZE { return 0; }
    // ...
}

// ❌ BAD β€” What does 10 mean? What about 8192?
fn skip_id3(data: &[u8]) -> usize {
    if data.len() < 10 { return 0; }
    // ...
}
Rule Detail
Location File top, before any fn or impl
Naming SCREAMING_SNAKE_CASE with context prefix (DEFAULT_, BALANCED_, etc.)
Lookup tables Bitrate tables, sample rate tables β†’ const arrays at file top
Magic bytes const EBML_MAGIC: &[u8] = &[0x1A, 0x45, 0xDF, 0xA5]; β€” never raw in conditionals

πŸ“¦ Return Types (No Tuples)

Never return tuples from functions. Use a named struct instead β€” even for two fields.

// βœ… GOOD β€” Clear field semantics at call site
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ByteRange {
    pub start: u64,
    pub end: u64,
}

fn find_range(&self, time: f64) -> Option<ByteRange> {
    // ...
}

// ❌ BAD β€” Opaque meaning, easy to swap fields
fn find_range(&self, time: f64) -> Option<(u64, u64)> {
    // ...
}
Rule Detail
Scope Module-private structs are fine if only used internally
Derives At minimum Debug, Clone β€” add Copy, PartialEq, Eq when applicable
Fields Descriptive names that convey semantics

πŸ”— Function Call & Type Qualification

Qualify function calls with at most one :: β€” import deeper paths at the top of the file.

// βœ… GOOD β€” Import then use short paths
use reqwest::header::{self, HeaderMap, HeaderValue};

let mut headers = HeaderMap::new();
headers.insert(header::CONTENT_TYPE, HeaderValue::from_static("text/plain"));

// ❌ BAD β€” Double-qualified paths
let mut headers = reqwest::header::HeaderMap::new();
headers.insert(reqwest::header::CONTENT_TYPE, reqwest::header::HeaderValue::from_static("text/plain"));
Rule Example
Self:: for associated fns in impl Self::new(), Self::parse_header(data)
module::function() detect::probe(data)
Type::method() String::from("hello")
Import heavily-used types directly use std::collections::HashMap; then HashMap::new()

🚨 Error Handling

We use a single unified error type in src/error.rs. Never introduce new error enums (except HookError which already exists for hook-specific failures).

Rules

Rule Detail
One Error enum All variants in one enum, grouped by // === Category === comment banners
Type alias pub type Result<T> = std::result::Result<T, Error>; β€” import as use crate::error::Result;
Structured fields Every variant uses named fields (operation, url, reason, path, source) β€” never just a string
#[source] Always on the inner error field for proper chaining
Helper constructors Error::io(...), Error::http(...) β€” each logs tracing::warn!/tracing::error! before constructing
From impls For std::io::Error, reqwest::Error, serde_json::Error, JoinError, ZipError β€” each logs with "(automatic conversion)" suffix
Parameter style impl Into<String> β€” not concrete types
Feature-gated #[cfg(feature = "cache-redb")] Database { ... }, #[cfg(feature = "cache-redis")] Redis { ... }
No anyhow Always use the crate's own Error / Result

Example: Adding a new error variant

// In src/error.rs, add to the appropriate category section:

// ==================== Video & Format Errors ====================

/// My new error description.
#[error("Something failed for {video_id}: {reason}")]
MyNewError {
    video_id: String,
    reason: String,
},

And add a helper constructor:

pub fn my_new_error(video_id: impl Into<String>, reason: impl Into<String>) -> Self {
    let video_id = video_id.into();
    let reason = reason.into();

    tracing::warn!(video_id = video_id, reason = reason, "Something failed");

    Self::MyNewError { video_id, reason }
}

πŸ”§ Builder Patterns

Two builder styles coexist β€” use the right one for the right job:

A) Manual builder (consuming mut self)

Used for: DownloaderBuilder, DownloadBuilder, WebhookConfig, FfmpegArgs

// βœ… Builder methods prefixed with `with_` and consuming `mut self`
pub fn with_timeout(mut self, timeout: Duration) -> Self {
    self.timeout = timeout;
    self
}

// βœ… Terminal method
pub async fn build(self) -> Result<Downloader> { ... }
Rule Detail
Method prefix with_ (e.g. with_args, with_timeout, with_proxy, with_cache)
Self parameter Always mut self (consuming) β€” never &mut self
Terminal method .build() or .execute()
Field visibility Private

B) TypedBuilder derive

Used for: config structs (ManagerConfig, RetryPolicy, ExpiryConfig)

#[derive(Debug, Clone, TypedBuilder)]
pub struct ManagerConfig {
    #[builder(default = SpeedProfile::default().max_concurrent_downloads())]
    pub max_concurrent_downloads: usize,
}
Rule Detail
Field visibility pub
Defaults #[builder(default = ...)]

C) Post-build mutation on Downloader

After .build(), use set_*/add_* methods (not with_*) to mutate the Downloader instance:

downloader.set_user_agent("my-agent");
downloader.set_timeout(Duration::from_secs(30));
downloader.set_args(vec!["--no-playlist".into()]);
downloader.add_arg("--flat-playlist");
downloader.set_cookies("cookies.txt");
downloader.set_cookies_from_browser("chrome");
downloader.set_netrc();
Rule Detail
Self parameter &mut self (borrowing) β€” returns &mut Self for chaining
Prefix for replacing set_ (e.g. set_cookies, set_user_agent, set_timeout)
Prefix for appending add_ (e.g. add_arg)

πŸ’‘ Don't confuse builder with_* methods (consuming mut self, used before .build()) with post-build set_*/add_* methods (borrowing &mut self, used after .build()).


πŸ“¦ Model & Data Types

Standard derive sets

Type Derives
Simple enums Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize + Default with #[default]
Complex structs (with f64) Debug, Clone, PartialEq, Serialize, Deserialize β€” manual Eq/Hash
Simple structs (no floats) Debug, Clone, PartialEq, Eq, Serialize, Deserialize

Serde patterns

Pattern Usage
#[serde(flatten)] Struct composition (e.g. Format flattens CodecInfo, VideoResolution, etc.)
#[serde(rename = "...")] Field name mapping from JSON ("timestamp", "acodec")
#[serde(rename_all = "snake_case")] Enum variant renaming
#[serde(default)] Optional collections and fields
#[serde(other)] Unknown variant for forward compatibility
#[serde(skip)] Derived/internal fields (e.g. video_id on Format)
json_none deserializer Turns "none" strings to Option::None (in model/utils/serde.rs)
#[serde_as(deserialize_as = "DefaultOnNull")] From serde_with, for nullable JSON fields
Custom Deserialize visitor Polymorphic types (e.g. DrmStatus accepts bool or string)
ordered_float::OrderedFloat<f64> Only when f64 needs Hash/Eq

πŸ–¨οΈ Display format

Always use the format TypeName(key=value, key=value):

impl fmt::Display for Video {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "Video(id={}, title={:?}, channel={:?}, formats={})",
            self.id, self.title, self.channel.as_deref().unwrap_or("Unknown"), self.formats.len())
    }
}
Rule Detail
Only essential fields Never full serialization
Option fields as_deref().unwrap_or("none") or unwrap_or("unknown")
Enum constant variants f.write_str("VariantName")
Enum variants with fields write!(f, "Variant(key={})", val)

πŸ”‘ Custom Hash implementations

Hash only identity fields β€” not all struct fields:

impl Hash for Video {
    fn hash<H: Hasher>(&self, state: &mut H) {
        self.id.hash(state);
        self.title.hash(state);
        self.channel.hash(state);
        self.channel_id.hash(state);
    }
}

🧬 Trait Design

Which pattern to use?

Pattern When Example
#[async_trait] Trait used as dyn Trait (trait objects) VideoExtractor, EventHook
RPITIT (impl Future + Send) Dispatched via concrete enum, never dyn Cache backend traits
DynClone + clone_trait_object! Need to clone trait objects EventHook
Downcast + impl_downcast! Runtime downcasting of trait objects VideoExtractor

#[async_trait] example

#[async_trait]
pub trait VideoExtractor: Downcast + Send + Sync + fmt::Debug {
    async fn fetch_video(&self, url: &str) -> Result<Video>;
    fn name(&self) -> ExtractorName;
    fn supports_url(&self, url: &str) -> bool;
}
impl_downcast!(VideoExtractor);

RPITIT example

pub trait VideoBackend: Send + Sync + std::fmt::Debug {
    fn get(&self, url: &str) -> impl Future<Output = Result<Option<Video>>> + Send;
    fn put(&self, url: String, video: Video) -> impl Future<Output = Result<()>> + Send;
}

πŸ“– Trait method declarations carry full rustdoc; implementations may add only a brief clarifying comment.


πŸ”’ Shared State & Concurrency

Primitives used

Primitive Purpose
Arc<reqwest::Client> Shared HTTP client with connection pooling
Arc<Mutex<...>> Mutable shared state (download queues, task maps, next_id counter)
Arc<Semaphore> Concurrency limit for parallel downloads
Arc<AtomicU64> / Arc<AtomicBool> Lock-free counters and flags
Arc<RwLock<...>> Read-heavy shared state (hook registry, stats, webhooks)
Arc<DownloadEvent> Events in broadcast channel (efficient cloning)
Arc<dyn Fn(...) + Send + Sync> Callbacks and filter predicates
tokio_util::sync::CancellationToken Graceful shutdown

⚠️ Important rules

Rule Detail
Async locks Use tokio::sync::Mutex and tokio::sync::RwLock
Sync locks std::sync::Mutex only for progress counters and non-async contexts
Lock safety Never hold a tokio lock across .await points
Simple counters Prefer Arc<AtomicU64> over Arc<Mutex<u64>>
Caches on Downloader Option<Arc<VideoCache>>

⚑ Async Programming

Rule Detail
Runtime tokio (multi-threaded)
Task spawning tokio::spawn for concurrency
Multiple tasks tokio::select! for managing cancellations
Structured concurrency Prefer scoped tasks and clean cancellation paths
Timeouts tokio::time::timeout with kill on timeout
Blocking work Offload to tokio::task::spawn_blocking (used for serde_json::from_reader, CPU-intensive parsing)
Time operations tokio::time::sleep and tokio::time::interval
HTTP reqwest with Arc<Client> connection pooling

Channels

Channel Usage
tokio::sync::mpsc Webhook delivery queue (bounded, backpressure)
tokio::sync::broadcast Event broadcasting to multiple subscribers
tokio::sync::oneshot One-time task communication

πŸ”” Event System

The event system lives in src/events/ and follows a three-phase delivery pattern:

Architecture

Component Role
EventBus Wraps broadcast::Sender<Arc<DownloadEvent>>
DownloadEvent Large enum β€” all variants use named fields (no tuple variants)
EventFilter Predicate-based with Vec<Arc<dyn Fn(&DownloadEvent) -> bool + Send + Sync>>
HookRegistry Arc<RwLock<Vec<Box<dyn EventHook>>>>
simple_hook! Macro to create hooks from closures

Event emission order (in Downloader::emit_event())

  1. πŸͺ Hooks β€” with timeout (#[cfg(feature = "hooks")])
  2. πŸ“‘ Webhooks β€” non-blocking (#[cfg(feature = "webhooks")])
  3. πŸ“’ Broadcast bus β€” always

Adding a new event variant

// In DownloadEvent β€” always use named fields:
// βœ… GOOD
MyNewEvent {
    download_id: u64,
    reason: String,
},

// ❌ BAD β€” No tuple variants
MyNewEvent(u64, String),

🎯 Feature Flags

Available features

Feature Purpose Dependencies
hooks Rust event callbacks None
webhooks HTTP event delivery None
statistics Real-time analytics None
cache-memory (default) In-memory Moka cache moka
cache-json JSON file backend None
cache-redb Embedded redb backend redb
cache-redis Distributed Redis backend redis
live-recording Live stream recording (HLS) m3u8-rs
live-streaming Live fragment streaming (HLS) m3u8-rs
rustls TLS backend reqwest/rustls
hickory-dns Async DNS resolver reqwest/hickory-dns
profiling Heap profiler dhat

βš™οΈ cache cfg is emitted by build.rs

The cache cfg is not a Cargo feature β€” it is a custom cfg emitted by build.rs when any cache backend (cache-memory, cache-json, cache-redb, or cache-redis) is enabled. Users cannot activate it directly, and it is invisible in Cargo.toml. Use #[cfg(cache)] to guard code that requires any cache backend.

Backend selection

build.rs emits persistent_cache when any of cache-json, cache-redb, or cache-redis is enabled. Multiple persistent features may be active simultaneously β€” the multiple_persistent_backends cfg and its associated compile_error! have been removed.

When exactly one persistent feature is compiled in, CacheConfig::persistent_backend is auto-deduced and may be left as None. When more than one is compiled in, persistent_backend must be set explicitly to a PersistentBackendKind variant; leaving it None causes CacheLayer::from_config to return Error::AmbiguousCacheBackend at runtime.

use yt_dlp::prelude::*;

// Multiple backends compiled in β€” pick one at runtime:
let config = CacheConfig::builder()
    .cache_dir("cache")
    .persistent_backend(PersistentBackendKind::Redb) // required when multiple compiled in
    .build();

Conditional compilation patterns

// Module-level guard for all cache code (cfg emitted by build.rs)
#[cfg(cache)]

// Backend-specific modules
#[cfg(feature = "cache-json")]
pub mod json;

// Persistent backend guard (any of json/redb/redis)
#[cfg(persistent_cache)]

// Feature-gated struct fields
#[cfg(feature = "hooks")]
pub(crate) hook_registry: Option<events::HookRegistry>,

❌ Forbidden patterns

  • Never use #[cfg(...)] on function parameters. It makes function signatures unreadable and call sites overly complex. If a parameter is feature-dependent, either feature-gate the entire function, or use a config struct / builder pattern where the specific field is feature-gated.

πŸ“ Tracing & Logging

Tracing is an unconditional dependency β€” every important function must have tracing.

Rules at a glance

Rule Detail
Macro style Always fully-qualified: tracing::debug!(...) β€” never import the macros
No #[instrument] Never use the #[instrument] attribute
Structured fields key = value, key = ?value (Debug), key = %value (Display)
No interpolation Never tracing::debug!("msg {}", var) β€” always structured fields

Log levels

Level Usage Emoji?
trace Hot paths, data transforms (rare β€” prefer deleting) βœ… Yes
debug Function entry/exit, parameters, config, internal ops βœ… Yes
info Key milestones (download start/end, fetch, install, shutdown) βœ… Yes
warn Recoverable failures, retries, fallbacks ❌ No emoji
error Unrecoverable per-item failures ❌ No emoji

🎨 Emoji prefixes

Every trace/debug/info message must start with one domain emoji:

Emoji Domain
πŸ“¦ Install / dependencies
πŸ“‘ Fetch / extract
πŸ“₯ Download
🎬 Combine / mux
βœ‚οΈ Postprocess / ffmpeg
🏷️ Metadata
πŸ’¬ Subtitle
πŸ–ΌοΈ Thumbnail
πŸ“‹ Playlist
βœ… Success / completion
πŸ”„ Retry / update
πŸ”§ Config / setup / builder
πŸ” Cache / lookup
βš™οΈ Internal / utility
πŸ“Š Statistics
πŸ”” Events
🧩 Format selection
πŸ›‘ Shutdown

Example

// βœ… GOOD
tracing::debug!(url = %url, timeout = ?timeout, "πŸ“₯ Starting download");
tracing::info!(video_id = video_id, formats = formats.len(), "πŸ“‘ Video fetched");
tracing::warn!(url = %url, attempt = attempt, "Retry after failure");

// ❌ BAD
tracing::debug!("Starting download for {}", url);  // No interpolation
tracing::info!("Video fetched");                     // No structured fields
tracing::warn!("⚠️ Retry");                          // No emoji on warn

What NOT to trace

  • ❌ Trivial getters/setters that just return or set a field
  • ❌ Pure transforms (to_ffmpeg_name, is_empty, enum-to-string)
  • ❌ Simple constant lookups / match on enum returning a value

πŸ“– Documentation

Every public function, method, and trait method must have a rustdoc comment:

Template

/// Brief one-line description.
///
/// Optional extended description.
///
/// # Arguments
///
/// * `param` - Description
///
/// # Errors
///
/// Returns an error if ...
///
/// # Returns
///
/// Description of return value.
///
/// # Examples
///
/// ```rust,no_run
/// # use yt_dlp::prelude::*;
/// # #[tokio::main]
/// # async fn main() -> Result<(), Box<dyn std::error::Error>> {
/// let downloader = Downloader::builder(libraries, "output").build().await?;
/// # Ok(())
/// # }
/// ```

Section rules

Section When to include
# Arguments Only if params beyond &self/&mut self
# Errors Only if returns Result
# Returns Only if returns a value (not ())
# Examples Main public API entry points (Downloader::new, download, fetch, etc.)

Additional rules

Rule Detail
Trait methods Full rustdoc on the trait declaration; impls may add only a brief comment
Getters Minimum one-liner + # Returns
Setters Minimum one-liner + # Arguments
Builder methods Minimum one-liner + # Arguments
Examples Use no_run or ignore for network/binary-dependent code

βš™οΈ Process Execution

The crate runs external processes (yt-dlp, ffmpeg) through a controlled abstraction:

Component Location Purpose
Executor src/executor/mod.rs Wraps tokio::process::Command with piped I/O and timeout
ProcessOutput src/executor/process.rs { stdout, stderr, code }
FfmpegArgs src/executor/ffmpeg.rs Fluent builder: .input(), .codec_copy(), .args(), .output(), .build()
run_ffmpeg_with_tempfile() src/executor/ffmpeg.rs Temp file + rename pattern for atomic writes

Key patterns

  • ⏱️ Timeout: tokio::time::timeout + process.kill() on timeout
  • πŸͺŸ Windows: command.creation_flags(0x08000000) (CREATE_NO_WINDOW) behind #[cfg(target_os = "windows")]
  • πŸ”„ Temp + rename: FFmpeg writes to a temp file, then renames atomically β€” never write directly to the final output
  • 🧡 CPU-heavy parsing: tokio::task::spawn_blocking for serde_json::from_reader and other CPU-intensive work

🧩 Macros

Defined in src/macros.rs and src/events/hooks.rs:

Macro Purpose
youtube!($yt_dlp, $ffmpeg, $output) Convenience Downloader constructor
ytdlp_args![...] Args builder (string list or key-value pairs)
install_libraries!($dir) Async binary installation
ternary!($cond, $true, $false) Ternary operator
simple_hook! Create an EventHook from a closure

All macros must use $crate:: fully-qualified paths for robustness. The use inside macro_rules! bodies is the only exception to the "imports at module top" rule.


πŸ” Contributing to media-seek

crates/media-seek/ is a standalone crate published independently to crates.io. Changes to it follow the same code conventions as the main crate, with a few important constraints.

Constraints

Rule Detail
No feature flags All formats are always compiled in β€” no conditional compilation inside media-seek
No reqwest The crate is transport-agnostic. Callers implement RangeFetcher.
No serde No serialization β€” pure parsing only
No async_trait RangeFetcher uses RPITIT (impl Future + Send), not #[async_trait]
No tuples ByteRange { start, end } instead of (u64, u64)
Named constants All magic numbers (sync bytes, header sizes, bitrate tables) as const at file top
dedup safety dedup_by_key only after sorting by the same key; re-sort after dedup if needed

Where to make changes

Change Location
Audio format parser crates/media-seek/src/audio/ (mp3.rs, ogg.rs, flac.rs, pcm.rs, adts.rs)
Video format parser crates/media-seek/src/video/ (mp4.rs, webm.rs, flv.rs, avi.rs, ts.rs)
Format detection crates/media-seek/src/detect.rs
Index data types crates/media-seek/src/index.rs
Error handling crates/media-seek/src/error.rs
Public API crates/media-seek/src/lib.rs

Tracing conventions

Every pub(crate) fn parse() / pub(crate) async fn parse() must have entry and success tracing:

// At function start:
tracing::debug!(probe_len = probe.len(), "βš™οΈ Parsing <Format> stream");

// Just before each successful return:
tracing::debug!(segments = result.len(), "βœ… <Format> index parsed");

Use βš™οΈ for internal operations and βœ… for success β€” same as the main crate. No emoji on warn! or error!.

Checking your changes

# media-seek standalone lint
cargo clippy -p media-seek -- -D warnings

# Run media-seek unit + integration tests
cargo test --test unit --all-features -- media_seek
cargo test --test integration --all-features -- media_seek

# Doc-tests (both crates)
cargo test --doc --workspace

βœ… Verification Checklist

Before submitting your PR, make sure:

  • πŸ” cargo clippy --workspace --all-features -- -D warnings β€” zero warnings
  • πŸ’„ cargo +nightly fmt --all -- --check β€” properly formatted
  • πŸ§ͺ cargo test --test unit --all-features β€” all unit tests pass
  • πŸ§ͺ cargo test --test integration --all-features β€” all integration tests pass
  • πŸ§ͺ cargo test --test e2e --all-features -- --test-threads=1 β€” all E2E tests pass
  • πŸ§ͺ cargo test --doc --workspace --all-features β€” all doc-tests pass
  • πŸ” cargo deny check β€” no dependency issues
  • 🧹 cargo machete β€” no unused dependencies
  • πŸ“ All new public items have rustdoc following the template
  • 🎨 All tracing uses structured fields + emoji prefix
  • 🚨 Errors use the existing Error enum with structured fields
  • πŸ“₯ All use imports are at the top of the file
  • πŸ”’ No magic numbers β€” all literals extracted to named const at file top
  • πŸ“¦ No tuple return types β€” use named structs instead
  • πŸ”— No double-qualified paths β€” import types and use short names
  • 🌍 All text (comments, docs, logs) is in English
  • πŸͺ† No function exceeds 2 nesting levels β€” extract deeper logic into private helpers

Thank you for contributing! πŸŽ‰
If you have questions, open a Discussion β€” we're happy to help.