Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,15 @@ backward compatible.

## [Unreleased]

### Changed

- **`vg build` and `vg scan` do not follow symlinks, and they say so.** A
directory symlink is not entered, so a link to its parent cannot hang the
walk, and a file symlink is not read. When any are skipped, one stderr
notice lists the count and the first few root-relative paths. Nothing is
printed when there are none. Output files, the exit code, and JSON on
stdout stay the same.

### Added

- **Unknown optional lockfile fields warn and continue.** When a lockfile still lists dependencies vg can read, an extra field does not drop that graph or the SBOM rows built from it. The command prints one warning per lockfile path and field name, sorted by path then field: `pnpm-lock.yaml: unknown optional lockfile field "<name>"; continuing with the fields this version understands.` A field name that is not a plain identifier is reported as `(name omitted)`. A truncated lockfile, or one missing the structure its format requires, still exits non-zero.
Expand Down
31 changes: 31 additions & 0 deletions DOCS.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ For a quick overview, see the [README](./README.md). This document covers everyt
- [vibgrate.config.ts](#vibgrateconfigts)
- [Thresholds](#thresholds)
- [Scanner Toggles](#scanner-toggles)
- [Symlinks](#symlinks)
- [Extended Scanners](#extended-scanners)
- [Platform Matrix](#platform-matrix)
- [Dependency Risk](#dependency-risk)
Expand Down Expand Up @@ -2128,6 +2129,8 @@ Switches (flags that take no value, such as `--vulns`, `--offline` or `--no-grap

By default, the scan writes `.vibgrate/scan_result.json`. Use `--no-local-artifacts` or `--max-privacy` to suppress local JSON artifact files.

`vg scan` does not follow symlinks while it indexes the tree. A skipped link is named once on stderr. See [Symlinks](#symlinks).

For offline drift scoring, pass `--package-manifest <file>` with a downloaded manifest bundle such as `https://github.com/vibgrate/manifests/latest-packages.zip`. The manifest shape, the fail-closed errors, and what offline mode skips are in [Offline scan with a package-version manifest](#offline-scan-with-a-package-version-manifest).

### Unknown lockfile fields
Expand Down Expand Up @@ -2718,6 +2721,8 @@ Maps source code into a graph artifact, enabling all downstream queries (`vg sho
| `--attestation <file>` | `.vibgrate/attestation.intoto.jsonl` | Where `--attest` writes, and where `--verify` reads |
| `--pub <path>` | — | Public key PEM that pins the signer for `--verify` |

`vg build` does not follow symlinks while it discovers files. A skipped link is named once on stderr. See [Symlinks](#symlinks). `.gitignore` and `--exclude` still apply.

**Local by default — no git churn.** The first time vg writes into `.vibgrate/` it also creates `.vibgrate/.gitignore`, keeping the graph artifacts (`graph.json`, `graph.html`, `GRAPH_REPORT.md`, `facts.jsonl`, `mcp-navigation.json`) and the cache out of git — so builds, auto-refreshes, and MCP use never leave your branch dirty. Run `vg share` when you want the map committed for your team (it rewrites that ignore file). vg never touches an existing `.vibgrate/.gitignore`, so edit it (or leave it empty) to manage the ignores yourself.

Maven `import` edges, and the Gradle files that stay document nodes, are in [Maven and Gradle manifests](#maven-and-gradle-manifests). How a GitHub Actions `uses:` pin is copied into the map: [Action pins in the code map](./docs/ci/github-actions.md#action-pins-in-the-code-map).
Expand Down Expand Up @@ -5031,6 +5036,32 @@ of observed memory — so identical input still produces an identical
`graph.json`. To give the build more room instead of limiting it, raise the
Node heap: `NODE_OPTIONS=--max-old-space-size=8192`.

### Symlinks

`vg build` and `vg scan` do not follow symlinks. While walking, a directory
symlink is not entered and a file symlink is not read. That includes a link
that points at its own parent, so a cycle cannot make the walk hang. The link
is left out of the map and out of the scan. Output files do not change, and
the exit code stays the same.

The directory you pass as the root is opened even when that path itself is a
symlink. Links found underneath it are not followed. To include the files a
link points at, point the root at that target, or pass a narrower directory
that already contains them. To leave a link out without a notice, list it in
`.gitignore` or pass `--exclude`.

When a walk skips one or more symlinks, vg prints one notice on stderr and
nothing when there are none. The notice is the count, then the first few
root-relative paths in sorted order (at most five; further links are a
`+N more` count). Paths are relative to the root you passed. Absolute paths
are not printed. The line goes to stderr, so `--format json` and
`vg build --json` keep a JSON document on stdout. `--quiet` hides promotional
text only; it does not hide this notice.

```text
notice: skipped 3 symlinks (alias.ts, nested/cycle, via). vg does not follow symlinks. Point the root at the link target, or pass --exclude or a narrower root.
```

---

## Extended Scanners
Expand Down
29 changes: 29 additions & 0 deletions src/core-open/utils/fs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import {
UnsafeRootError,
type WalkBudgetState,
} from './root-safety.js';
import { emitSkippedSymlinkNotice, rememberSkippedSymlink } from './skipped-symlinks.js';


const execFileAsync = promisify(execFile);
Expand Down Expand Up @@ -490,6 +491,10 @@ export class FileCache {
if (budgetError) return;
budgetError = noteWalkEntry(budgetState);
};
// Symlinks are not followed. Dirent.isDirectory() is false for a link, so a
// link to a parent cannot re-enter this walk. One notice is printed after
// the walk finishes. The prelude count stays quiet so a scan says it once.
const skippedSymlinks: string[] = [];

async function walk(dir: string, gitignoreLevels: GitignoreLevel[]) {
if (budgetError) return;
Expand Down Expand Up @@ -542,6 +547,15 @@ export class FileCache {
if (isExcluded && isExcluded(relPath)) continue;
if (budgetError) break;

if (e.isSymbolicLink()) {
// Ignored as a file or as a directory: the operator already asked to
// skip it, so it does not join the notice.
if (!(isGitignored(levels, absPath, false) || isGitignored(levels, absPath, true))) {
rememberSkippedSymlink(skippedSymlinks, relPath);
}
continue;
}

if (e.isDirectory()) {
if (isSkippedDirName(e.name) || extraSkip.has(e.name)) continue;
// Respect .gitignore (root + any nested ones seen so far) so
Expand Down Expand Up @@ -577,6 +591,7 @@ export class FileCache {

await walk(rootDir, []);
if (budgetError) throw budgetError;
emitSkippedSymlinkNotice(skippedSymlinks);

let totalDirs = 0;
const rootNameIndex = new Map<string, string[]>();
Expand Down Expand Up @@ -885,6 +900,10 @@ async function countTreeWithBudget(
const relPath = path.relative(rootDir, absPath);
if (isExcluded && isExcluded(relPath)) continue;

// Not followed. The indexing walk prints the notice; this count does not,
// so vg scan does not say the same thing twice.
if (e.isSymbolicLink()) continue;

if (e.isDirectory()) {
if (isSkippedDirName(e.name) || extraSkip.has(e.name)) continue;
if (isGitignored(levels, absPath, true)) continue;
Expand Down Expand Up @@ -975,6 +994,8 @@ export async function countFilesInDir(dir: string, recursive = true): Promise<nu
}
const subs: Promise<void>[] = [];
for (const e of entries) {
// Not followed. A link to a parent cannot re-enter this count.
if (e.isSymbolicLink()) continue;
if (e.isDirectory()) {
if (!recursive) continue;
if (isSkippedDirName(e.name) || extraSkip.has(e.name)) continue;
Expand Down Expand Up @@ -1009,6 +1030,8 @@ export async function bytesInDir(dir: string, recursive = true): Promise<number>
}
const subs: Promise<void>[] = [];
for (const e of entries) {
// Not followed. A link to a parent cannot re-enter this size walk.
if (e.isSymbolicLink()) continue;
if (e.isDirectory()) {
if (!recursive) continue;
if (isSkippedDirName(e.name) || extraSkip.has(e.name)) continue;
Expand Down Expand Up @@ -1044,6 +1067,7 @@ export async function findFiles(
budgetError = noteWalkEntry(budgetState);
};
const results: string[] = [];
const skippedSymlinks: string[] = [];
const cores = typeof os.availableParallelism === 'function' ? os.availableParallelism() : os.cpus().length || 4;
const maxConcurrentReads = Math.max(8, Math.min(64, cores * 4));
const readDirSemaphore = new Semaphore(maxConcurrentReads);
Expand All @@ -1061,6 +1085,10 @@ export async function findFiles(

for (const e of entries) {
if (budgetError) break;
if (e.isSymbolicLink()) {
rememberSkippedSymlink(skippedSymlinks, path.relative(rootDir, path.join(dir, e.name)));
continue;
}
if (e.isDirectory()) {
if (isSkippedDirName(e.name)) continue;
note();
Expand All @@ -1081,6 +1109,7 @@ export async function findFiles(

await walk(rootDir);
if (budgetError) throw budgetError;
emitSkippedSymlinkNotice(skippedSymlinks);
return results;
}

Expand Down
65 changes: 65 additions & 0 deletions src/core-open/utils/skipped-symlinks.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
// VENDORED from @vibgrate/core-open (packages/vibgrate-core-open) by
// scripts/vendor-core-open.mjs. Do not edit here — change the source package
// and re-run the vendor script. Apache-2.0.
import * as path from 'node:path';

/**
* How many root-relative paths one notice lists. Further links are a count.
* The walk sorts before capping, so the same tree always names the same paths.
*/
export const SKIPPED_SYMLINK_NOTICE_CAP = 5;

function comparePath(a: string, b: string): number {
return a < b ? -1 : a > b ? 1 : 0;
}

/**
* Root-relative POSIX path, or '' when `absPath` is the root or escapes it.
* Absolute paths are never returned.
*/
export function walkRelativePath(root: string, absPath: string): string {
const rel = path.relative(root, absPath);
if (!rel || rel === '.' || rel.startsWith('..')) return '';
return rel.split(path.sep).join('/');
}

/** Record one skipped symlink. Empty and out-of-root paths are dropped. */
export function rememberSkippedSymlink(bucket: string[], relPath: string): void {
const rel = relPath.split(path.sep).join('/');
if (!rel || rel === '.' || rel.startsWith('..') || rel.startsWith('/')) return;
bucket.push(rel);
}

/**
* One stderr line when a walk skips symlinks. `null` when `relPaths` is empty.
* Paths are de-duplicated, sorted, and capped. Absolute paths are omitted.
*/
export function formatSkippedSymlinkNotice(relPaths: readonly string[]): string | null {
const seen = new Set<string>();
const unique: string[] = [];
for (const raw of relPaths) {
const rel = raw.split('\\').join('/');
if (!rel || rel === '.' || rel.startsWith('../') || rel === '..' || rel.startsWith('/')) continue;
if (/^[A-Za-z]:/.test(rel)) continue;
if (seen.has(rel)) continue;
seen.add(rel);
unique.push(rel);
}
unique.sort(comparePath);
if (unique.length === 0) return null;
const shown = unique.slice(0, SKIPPED_SYMLINK_NOTICE_CAP);
const extra = unique.length - shown.length;
const list = extra > 0 ? `${shown.join(', ')}, +${extra} more` : shown.join(', ');
const noun = unique.length === 1 ? 'symlink' : 'symlinks';
return (
`notice: skipped ${unique.length} ${noun} (${list}). ` +
'vg does not follow symlinks. Point the root at the link target, or pass --exclude or a narrower root.'
);
}

/** Write {@link formatSkippedSymlinkNotice} to stderr. No-op when there is nothing to say. */
export function emitSkippedSymlinkNotice(relPaths: readonly string[]): void {
const line = formatSkippedSymlinkNotice(relPaths);
if (!line) return;
process.stderr.write(`${line}\n`);
}
11 changes: 11 additions & 0 deletions src/engine/discover.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { requireDataConfig } from '../core-open/config.js';
import { dropBlankPatterns, gitignoreWithoutBlankLines } from '../core-open/utils/glob.js';
import { assertLockfileFile, lockfileKind } from '../core-open/utils/lockfile-parse.js';
import { assertSafeWalkRoot, createWalkBudget, noteWalkEntry } from '../core-open/utils/root-safety.js';
import { emitSkippedSymlinkNotice } from '../core-open/utils/skipped-symlinks.js';

/**
* Deterministic file discovery.
Expand Down Expand Up @@ -248,6 +249,10 @@ export function discover(options: DiscoverOptions): DiscoveredFile[] {

const found = new Map<string, DiscoveredFile>();
const budget = createWalkBudget(root, options.maxEntries);
// Symlinks are not followed. Dirent.isDirectory() / isFile() are false for a
// link, so a directory link to its parent cannot re-enter this walk. The
// notice lists the ones this walk skipped; ignored links stay quiet.
const skippedSymlinks: string[] = [];

const considerFile = (abs: string): void => {
const rel = toPosix(path.relative(root, abs));
Expand All @@ -274,6 +279,11 @@ export function discover(options: DiscoverOptions): DiscoveredFile[] {
for (const entry of entries) {
const abs = path.join(dir, entry.name);
const rel = toPosix(path.relative(root, abs));
if (entry.isSymbolicLink()) {
if (rel && (rootIg.ignores(rel) || rootIg.ignores(`${rel}/`))) continue;
if (rel) skippedSymlinks.push(rel);
continue;
}
if (entry.isDirectory()) {
if (isSkippedDirName(entry.name)) continue;
if (rel && rootIg.ignores(`${rel}/`)) continue;
Expand All @@ -297,6 +307,7 @@ export function discover(options: DiscoverOptions): DiscoveredFile[] {
} else if (stat.isFile()) considerFile(scope);
}

emitSkippedSymlinkNotice(skippedSymlinks);
return [...found.values()].sort((a, b) => (a.rel < b.rel ? -1 : a.rel > b.rel ? 1 : 0));
}

Expand Down
2 changes: 2 additions & 0 deletions src/engine/docs-ingest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -408,6 +408,8 @@ export function discoverDocs(options: DiscoverDocsOptions): DiscoveredDoc[] {
if (found.size >= maxFiles) return;
const abs = path.join(dir, entry.name);
const rel = toPosix(path.relative(root, abs));
// Not followed. discover() prints the one notice for this tree.
if (entry.isSymbolicLink()) continue;
if (entry.isDirectory()) {
if (isSkippedDirName(entry.name)) continue;
// Workflows / .github must be walked even if other tools ignore them
Expand Down
Loading
Loading