Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README-es.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,9 @@ Cuando no se especifica `type`, DLoad automáticamente usa todos los manejadores
> Usa `type="phar"` para herramientas PHP que deben mantenerse como archivos `.phar`.
> Usar `type="archive"` extraerá incluso archivos `.phar`.

> [!NOTE]
> Cuando hay un `<binary>` configurado, con `type="binary"` o sin `type`, los paquetes del sistema operativo (`.deb`, `.rpm`, `.apk`, `.snap`, `.flatpak`, `.msi`, `.msix`, `.msixbundle`, `.appx`, `.appxbundle`, `.nupkg`, `.dmg`, `.pkg`) nunca se seleccionan: DLoad no puede desempaquetarlos. Un release que solo ofrece un paquete para el host se omite en favor de uno más antiguo, entre los 10 releases más recientes que cumplen la restricción de versión; fija `version` para ir más atrás.

#### Extracción de Archivos (Preservando la Estructura)

`type="archive"` descomprime el asset **completo** en `extract-path`, manteniendo la estructura interna de directorios del archivo en lugar de aplanar los archivos coincidentes en una sola carpeta:
Expand Down
3 changes: 3 additions & 0 deletions README-ru.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,9 @@ DLoad поддерживает три типа загрузки, которые
> Используйте `type="phar"` для PHP-инструментов, которые должны остаться как `.phar` файлы.
> Использование `type="archive"` распакует даже `.phar` архивы.

> [!NOTE]
> Если задан `<binary>` — с `type="binary"` или без `type`, — пакеты ОС (`.deb`, `.rpm`, `.apk`, `.snap`, `.flatpak`, `.msi`, `.msix`, `.msixbundle`, `.appx`, `.appxbundle`, `.nupkg`, `.dmg`, `.pkg`) никогда не выбираются: DLoad не умеет их распаковывать. Релиз, в котором для хоста есть только пакет, пропускается ради более старого — среди 10 новейших релизов, удовлетворяющих ограничению версии; чтобы уйти дальше в прошлое, закрепите `version`.

#### Распаковка архива с сохранением структуры

`type="archive"` распаковывает **весь** ассет в `extract-path`, сохраняя внутреннюю структуру каталогов архива, а не сплющивая совпавшие файлы в одну папку:
Expand Down
3 changes: 3 additions & 0 deletions README-zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,9 @@ DLoad 支持三种下载类型,它们决定了资源的处理方式:
> 对于应该保持为 `.phar` 文件的 PHP 工具,使用 `type="phar"`。
> 使用 `type="archive"` 会解包甚至 `.phar` 压缩包。

> [!NOTE]
> 配置了 `<binary>` 时(无论是 `type="binary"` 还是未指定 `type`),永远不会选择操作系统安装包(`.deb`、`.rpm`、`.apk`、`.snap`、`.flatpak`、`.msi`、`.msix`、`.msixbundle`、`.appx`、`.appxbundle`、`.nupkg`、`.dmg`、`.pkg`):DLoad 无法解包它们。如果某个发布版本只为当前主机提供安装包,则会跳过它,改用更早的版本,范围限于满足版本约束的最新 10 个发布版本;要回溯更早的版本,请固定 `version`。

#### 解包压缩包(保留结构)

`type="archive"` 会将**整个**资源解包到 `extract-path`,保留压缩包内部的目录结构,而不是把匹配的文件平铺到单个目录中:
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,9 @@ When `type` is not specified, DLoad automatically uses all available handlers:
> Use `type="phar"` for PHP tools that should remain as `.phar` files.
> Using `type="archive"` will unpack even `.phar` archives.

> [!NOTE]
> When a `<binary>` is configured, with `type="binary"` or without a `type`, OS packages (`.deb`, `.rpm`, `.apk`, `.snap`, `.flatpak`, `.msi`, `.msix`, `.msixbundle`, `.appx`, `.appxbundle`, `.nupkg`, `.dmg`, `.pkg`) are never selected: DLoad cannot unpack them. A release that offers only a package for the host is skipped for an older one, among the 10 newest releases that satisfy the version constraint; pin `version` to go further back.

#### Archive Extraction (Preserving Structure)

`type="archive"` unpacks the **entire** asset into `extract-path`, keeping the archive's internal directory layout instead of flattening matched files into a single folder:
Expand Down
2 changes: 1 addition & 1 deletion skills/dload-fetch-tool/references/registry-entry.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Filters the release's asset list. dload then runs OS/arch detection on every mat
- match all OS/arch variants of the tool;
- leave out sibling tools, checksums, signatures, source archives.

Build variants stay in the pattern too. Among the assets for the host, dload prefers the host libc (`musl` builds on Alpine and Android, the others elsewhere), then the name with the fewest extra tokens — so `bun-linux-x64.zip` wins over `-baseline`, `-profile` and `-debug` twins — then archives. Android builds are dropped on Linux whenever the entry has a `binary` (without one, every other platform is only ranked lower). On ARM macOS and Windows, x86-64 builds are a fallback: Rosetta 2 and the Windows emulation run them.
Build variants stay in the pattern too. Among the assets for the host, dload prefers the host libc (`musl` builds on Alpine and Android, the others elsewhere), then the name with the fewest extra tokens — so `bun-linux-x64.zip` wins over `-baseline`, `-profile` and `-debug` twins — then archives. Android builds are dropped on Linux whenever the entry has a `binary` (without one, every other platform is only ranked lower), and so are OS packages (`.deb`, `.rpm`, `.apk`, `.snap`, `.flatpak`, `.msi`, `.msix`, `.msixbundle`, `.appx`, `.appxbundle`, `.nupkg`, `.dmg`, `.pkg`) everywhere: dload cannot unpack them, so a release that has only a package for the host gives way to an older one among the 10 newest releases that satisfy `version` (pin `version` to go further back). On ARM macOS and Windows, x86-64 builds are a fallback: Rosetta 2 and the Windows emulation run them.

Tokens the OS/arch matchers recognise (case-insensitive, bounded by `_` or a word boundary):

Expand Down
2 changes: 2 additions & 0 deletions skills/dload-fetch-tool/references/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ A candidate that matches neither is discarded. Common offenders:

Compare with the host: `php -r "echo PHP_OS_FAMILY, ' / ', php_uname('m'), PHP_EOL;"`.

With a `binary`, OS packages (`.deb`, `.msi`… — the list is in [`registry-entry.md`](registry-entry.md#asset-pattern)) are discarded too, since dload cannot unpack them, and a release offering only packages is skipped for an older one among the 10 newest releases that satisfy `version`, with a warning naming both versions. When the package names carry the host OS and arch, the report says "only OS packages match" and lists them under "OS packages are skipped because a binary is expected"; a package without an OS token (`tool.dmg`) is discarded by the OS check above instead. When no older release fits, pin a `version` that ships an archive or a raw binary for the host — pinning also reaches past the 10 newest.

Several candidates left: dload tries them best first — host libc, then the fewest extra name tokens, then archives, then the release order. Run with `-vvv` to see the rank of every asset (`Asset ... ranked: os=0 arch=0 libc=0 extras=1 archive=0`); a wrong pick usually means the wanted build carries an extra token the others lack.

## Stage 4 — binary extraction
Expand Down
33 changes: 31 additions & 2 deletions src/Module/Downloader/Downloader.php
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@
use Internal\DLoad\Module\Downloader\Exception\ReleaseGone;
use Internal\DLoad\Module\Downloader\Internal\Diagnostics\DownloadDiagnostics;
use Internal\DLoad\Module\Downloader\Internal\AssetSelection\AssetSelector;
use Internal\DLoad\Module\Downloader\Internal\AssetSelection\Candidate;
use Internal\DLoad\Module\Downloader\Internal\AssetSelection\Rule\ArchitectureRule;
use Internal\DLoad\Module\Downloader\Internal\AssetSelection\Rule\PackageRule;
use Internal\DLoad\Module\Downloader\Internal\DownloadContext;
use Internal\DLoad\Module\Downloader\Task\DownloadResult;
use Internal\DLoad\Module\Downloader\Task\DownloadTask;
Expand Down Expand Up @@ -63,6 +65,12 @@ final class Downloader
/** Number of release names collected for the failure report. */
private const FETCHED_RELEASES_LIMIT = 10;

/**
* Number of the newest matching releases tried. Iteration stops after this many matches; a higher cap
* reads further release pages whenever the first pages hold fewer matches.
*/
private const TRIED_RELEASES_LIMIT = 10;

public function __construct(
private readonly DownloaderConfig $config,
private readonly Logger $logger,
Expand Down Expand Up @@ -177,6 +185,7 @@ public function download(
private function processRepository(Repository $repository, DownloadContext $context, bool $mayRetry = true): \Closure
{
return function () use ($repository, $context, $mayRetry): ReleaseInterface {
$context->packagesOnlyRelease = null;
// Set when a release turned out to be deleted: the release list is outdated then
$forgotten = false;

Expand All @@ -199,7 +208,7 @@ private function processRepository(Repository $repository, DownloadContext $cont
}

/** @var ReleaseInterface[] $releases */
$releases = $releasesCollection->limit(10)->sortByVersion()->toArray();
$releases = $releasesCollection->limit(self::TRIED_RELEASES_LIMIT)->sortByVersion()->toArray();

$this->logger->debug('%d releases found.', \count($releases));

Expand Down Expand Up @@ -233,6 +242,18 @@ private function processRepository(Repository $repository, DownloadContext $cont

try {
await(coroutine($this->processRelease($context)));

/** @var ReleaseInterface|null $skipped */
$skipped = $context->packagesOnlyRelease;
$skipped === null or $this->logger->warning(
'`%s` %s has only OS packages for `%s`/`%s`, downloaded %s instead.',
$context->software->getId(),
$skipped->getVersion()->string,
$this->operatingSystem->value,
$this->architecture->value,
$context->release->getVersion()->string,
);

return $context->release;
} catch (ReleaseGone $e) {
// The registry must not offer this release again, and the list needs a fresh check
Expand Down Expand Up @@ -299,10 +320,18 @@ private function processRelease(DownloadContext $context): \Closure
);
$this->logger->debug('%d matching assets found.', \count($selection->candidates));

$skippedPackages = \array_map(
static fn(Candidate $candidate): string => $candidate->asset->getName(),
$selection->removed[PackageRule::KEY] ?? [],
);
$context->releaseAttempt->registerSkippedPackages($skippedPackages);

$selection->isEmpty() && $skippedPackages !== [] and $context->packagesOnlyRelease ??= $context->release;
$selection->isEmpty() and throw new NotFound(
$strict
? \sprintf(
'no asset matches OS `%s`, architecture `%s`, name pattern `%s`%s',
'%s OS `%s`, architecture `%s`, name pattern `%s`%s',
$skippedPackages === [] ? 'no asset matches' : 'only OS packages match',
$this->operatingSystem->value,
$this->architecture->value,
$context->repoConfig->assetPattern,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
use Internal\DLoad\Module\Downloader\Internal\AssetSelection\Rule\LibcRule;
use Internal\DLoad\Module\Downloader\Internal\AssetSelection\Rule\NamePatternRule;
use Internal\DLoad\Module\Downloader\Internal\AssetSelection\Rule\OperatingSystemRule;
use Internal\DLoad\Module\Downloader\Internal\AssetSelection\Rule\PackageRule;
use Internal\DLoad\Module\Repository\AssetInterface;
use Psr\Container\ContainerInterface;

Expand Down Expand Up @@ -54,6 +55,9 @@ public function __construct(
new CompanionRule(),
new OperatingSystemRule($operatingSystem),
new ArchitectureRule($architecture, $operatingSystem),
// After the platform rules, so the report names only packages built for the host. A package whose
// name carries no OS token, like `tool.dmg`, is already gone then and stays out of the report.
new PackageRule(),
new LibcRule($container),
new ExtrasRule(),
new ArchiveRule($archiveFactory),
Expand Down
41 changes: 41 additions & 0 deletions src/Module/Downloader/Internal/AssetSelection/Rule/PackageRule.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<?php

declare(strict_types=1);

namespace Internal\DLoad\Module\Downloader\Internal\AssetSelection\Rule;

use Internal\DLoad\Module\Downloader\Internal\AssetSelection\AssetRule;
use Internal\DLoad\Module\Downloader\Internal\AssetSelection\Candidate;
use Internal\DLoad\Module\Downloader\Internal\AssetSelection\Selection;

/**
* Removes OS packages (`.deb`, `.rpm`, `.msi`…) when a binary is expected.
*
* dload cannot unpack a package, so the binary is never found inside one. A selected package
* downloads fine and then fails the installation, which runs after the release loop is over:
* older releases with an archive would never be tried. Removed here, a release with only
* a package for the host has no matching asset, and the downloader moves on to the next one.
*
* @internal
* @psalm-internal Internal\DLoad\Module\Downloader
*/
final class PackageRule implements AssetRule
{
/** Reason the packages are removed under, see {@see Selection::$removed}. */
public const KEY = 'package';

private const EXTENSIONS = [
'deb', 'rpm', 'apk', 'msi', 'dmg', 'pkg',
'snap', 'flatpak', 'msix', 'msixbundle', 'appx', 'appxbundle', 'nupkg',
];

public function select(Selection $selection, callable $next): Selection
{
$selection->strict and $selection = $selection->remove(
static fn(Candidate $candidate): bool => $candidate->hasExtension(self::EXTENSIONS),
self::KEY,
);

return $next($selection);
}
}
30 changes: 23 additions & 7 deletions src/Module/Downloader/Internal/AssetSelection/Selection.php
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,15 @@ final class Selection
* @param non-empty-string $assetPattern Pattern the asset names must match.
* @param Type|null $type Download action type restricting the asset format.
* @param bool $strict Whether assets for another OS or architecture are removed rather than ranked lower.
* @param array<non-empty-string, list<Candidate>> $removed Candidates removed with a reason, by the reason,
* for a failure report to name them.
*/
private function __construct(
public readonly array $candidates,
public readonly string $assetPattern,
public readonly ?Type $type,
public readonly bool $strict,
public readonly array $removed = [],
) {}

/**
Expand All @@ -53,13 +56,25 @@ public static function create(iterable $assets, string $assetPattern, ?Type $typ
* is a rank, see {@see self::rank()}.
*
* @param \Closure(Candidate): bool $predicate
* @param non-empty-string|null $reason Key to keep the removed candidates under, see {@see self::$removed}.
*/
public function remove(\Closure $predicate): self
public function remove(\Closure $predicate, ?string $reason = null): self
{
return $this->withCandidates(\array_values(\array_filter(
$this->candidates,
static fn(Candidate $candidate): bool => !$predicate($candidate),
)));
$kept = $removed = [];
foreach ($this->candidates as $candidate) {
if ($predicate($candidate)) {
$removed[] = $candidate;
} else {
$kept[] = $candidate;
}
}

return $this->withCandidates(
$kept,
$reason === null || $removed === []
? $this->removed
: [...$this->removed, $reason => [...$this->removed[$reason] ?? [], ...$removed]],
);
}

/**
Expand Down Expand Up @@ -121,9 +136,10 @@ public function assets(): array

/**
* @param list<Candidate> $candidates
* @param array<non-empty-string, list<Candidate>>|null $removed
*/
private function withCandidates(array $candidates): self
private function withCandidates(array $candidates, ?array $removed = null): self
{
return new self($candidates, $this->assetPattern, $this->type, $this->strict);
return new self($candidates, $this->assetPattern, $this->type, $this->strict, $removed ?? $this->removed);
}
}
16 changes: 16 additions & 0 deletions src/Module/Downloader/Internal/Diagnostics/ReleaseAttempt.php
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ final class ReleaseAttempt
/** @var list<string> Names of all assets in the release */
private array $assetNames = [];

/** @var list<non-empty-string> Names of the OS packages for the host, skipped because a binary is expected */
private array $skippedPackages = [];

/** @var list<non-empty-string> Errors occurred while downloading matched assets */
private array $failures = [];

Expand All @@ -42,6 +45,14 @@ public function registerAssets(array $names): void
$this->assetNames = $names;
}

/**
* @param list<non-empty-string> $names Names of the OS packages for the host, skipped because a binary is expected
*/
public function registerSkippedPackages(array $names): void
{
$this->skippedPackages = $names;
}

/**
* Registers a failure of a matched asset, e.g. a broken download.
*
Expand Down Expand Up @@ -79,6 +90,11 @@ public function describe(): array
);
}

$this->skippedPackages === [] or $lines[] = \sprintf(
' OS packages are skipped because a binary is expected: %s',
\implode(', ', $this->skippedPackages),
);

foreach ($this->failures as $failure) {
$lines[] = ' Failed asset ' . $failure;
}
Expand Down
7 changes: 7 additions & 0 deletions src/Module/Downloader/Internal/DownloadContext.php
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,13 @@ final class DownloadContext
/** @var ReleaseAttempt Diagnostics of the release being processed */
public ReleaseAttempt $releaseAttempt;

/**
* @var ReleaseInterface|null Newest release of the current repository pass skipped because it has only
* OS packages for the host. Another repository or a refetched release list starts over, so a version
* found elsewhere is not reported against it.
*/
public ?ReleaseInterface $packagesOnlyRelease = null;

/**
* Creates a new download context.
*
Expand Down
Loading
Loading