Add unified page and attachment filter

README.md

@@ -1,7 +1,8 @@
 # git-confluence
 
-`git-confluence` is a Git clean/smudge filter that converts between
-Confluence storage XML and Markdown.
+`git-confluence` is a unified Git filter for Confluence pages and attachments.
+It converts page bodies between Confluence storage XML and Markdown and
+materializes attachment pointer files through the Confluence download API.
 
 It keeps Confluence storage XML as the content stored in Git while exposing the
 same files as Markdown in the working tree. This lets Confluence keep the XML
@@ -24,9 +25,11 @@ Git remote. Page bodies are stored in Git as Confluence storage XML.
 5. The clean filter converts Markdown back to Confluence storage XML.
 6. `git-remote-confluence` pushes the committed XML to existing Confluence page bodies.
 
-This repository does not call the Confluence API by itself. It reads storage XML
-or Markdown from standard input, converts it, and writes the result to standard
-output.
+Attachment pointers remain small YAML files when the filter is not configured,
+the PAT is unavailable, or smudge is explicitly skipped. With the filter
+configured, checkout downloads attachment bytes into a local Git-specific cache
+and exposes them at their normal paths. Attachment changes are read-only until
+upload support is implemented.
 
 ## Storage And Working Formats
 
@@ -39,6 +42,16 @@ content shown in the working tree.
 | Working tree after checkout | Markdown |
 | Index after `git add` | Confluence storage XML |
 
+For attachments:
+
+| Location | Format |
+| --- | --- |
+| Git blob | Confluence attachment pointer |
+| Working tree after checkout | Attachment bytes, or pointer when unavailable/skipped |
+| Index after `git add` | Original attachment pointer |
+
+Attachment pointers use the `attachment/v1` YAML schema.
+
 This split preserves storage XML for Confluence synchronization while allowing
 people to edit Markdown.
 
@@ -94,26 +107,32 @@ git-confluence version
 
 ## Filter Configuration
 
-Place this repository wherever you want, then configure the filter in the
-repository that contains Confluence pages:
+Configure the unified page and attachment filter once for the current user:
 
 ```sh
-git config filter.confluence-storage.clean "/path/to/git-confluence/git-confluence clean"
-git config filter.confluence-storage.smudge "/path/to/git-confluence/git-confluence smudge"
-git config filter.confluence-storage.required true
+git confluence install --global
 ```
 
+`install` registers the `filter.confluence` clean/smudge commands in Git
+configuration; it does not install the `git-confluence` executable or download
+attachments.
+
+Use `git confluence install --local` after a `--no-checkout` clone when the
+configuration should apply to only one repository.
+
 Limit the target files with `.gitattributes`:
 
 ```gitattributes
-pages/**/*.md filter=confluence-storage diff=markdown
+*.md filter=confluence diff=markdown
+**/attachments/** filter=confluence -text
 ```
 
 Repositories created by `git-remote-confluence` include this imported
 `.gitattributes` entry:
 
 ```gitattributes
-*.md filter=confluence-storage diff=markdown
+*.md filter=confluence diff=markdown
+**/attachments/** filter=confluence -text
 ```
 
 If you want the first checkout during clone to produce Markdown, configure the
@@ -125,9 +144,7 @@ CONFLUENCE_PAT=... git clone --no-checkout \
   'confluence::https://confluence.example.com/pages/viewpage.action?pageId=123456789' \
   pages
 cd pages
-git config filter.confluence-storage.clean "/path/to/git-confluence/git-confluence clean"
-git config filter.confluence-storage.smudge "/path/to/git-confluence/git-confluence smudge"
-git config filter.confluence-storage.required true
+git confluence install --local
 git checkout
 ```
 
@@ -139,6 +156,46 @@ CONFLUENCE_PAT=... git clone \
   'confluence::https://confluence.example.com/pages/viewpage.action?pageId=123456789'
 ```
 
+Keep attachment pointers while still converting page XML to Markdown:
+
+```sh
+GIT_CONFLUENCE_SKIP_SMUDGE=1 git checkout
+```
+
+Attachment checkout progress is written to stderr as logfmt. Each record names
+the working-tree path, attachment ID, and version, and reports cache hits and
+download progress without exposing the Confluence PAT:
+
+```text
+level=INFO msg="downloading attachment" app=git-confluence path=123/attachments/diagram.png attachment_id=456 attachment_version=2 filter=smudge direction=attachment_pointer_to_bytes purpose=materialize_worktree
+level=INFO msg="downloaded attachment" app=git-confluence path=123/attachments/diagram.png attachment_id=456 attachment_version=2 filter=smudge direction=attachment_pointer_to_bytes purpose=materialize_worktree bytes=42000
+```
+
+Page clean and smudge conversion also reports start and completion records,
+including the path, filter, direction, purpose, and input or output byte count.
+The clean filter normalizes worktree content for Git comparison or storage;
+the smudge filter materializes Git content in the worktree. This makes a slow
+page conversion distinguishable from an attachment download.
+
+Page smudge stores the exact source storage XML below
+`$GIT_DIR/confluence/pages`. When the resulting Markdown is unchanged, clean
+restores those exact bytes instead of regenerating XML. Edited Markdown is
+converted normally. This keeps an untouched checkout clean even when its source
+contains Confluence attributes that Markdown cannot represent. If the filter is
+installed after a normal checkout, an existing working file identical to its
+index blob is also passed through unchanged until it is explicitly checked out
+through smudge.
+
+Materialize all attachments, or selected paths, after checkout:
+
+```sh
+git confluence pull
+git confluence pull 123456789/attachments/diagram.png
+```
+
+`pull` enables the filter locally, refuses to overwrite attachment paths with
+local changes, and reports an error if an attachment could not be materialized.
+
 ## Direct Use
 
 You can also run the converter directly instead of using it as a Git filter:
@@ -151,6 +208,19 @@ go run . clean < page.md > page.xml
 `smudge` converts Confluence storage XML to Markdown. `clean` converts Markdown
 to Confluence storage XML.
 
+## Attachment Authentication And Cache
+
+Attachment download uses the first PAT found in `CONFLUENCE_PAT`,
+`GIT_REMOTE_CONFLUENCE_PAT`, `confluence.pat`, or
+`remote.confluence.pat`. Download URLs are constrained to the Confluence origin
+recorded by the pointer.
+
+Downloaded objects and clean-filter reverse mappings are stored below
+`.git/confluence`. The cache avoids downloading the same pointer again and lets
+the clean filter restore an unchanged attachment to its canonical pointer.
+Unknown attachment bytes are rejected as read-only rather than accidentally
+being committed as large Git blobs.
+
 ## Input Size And Recursion Depth
 
 The filter reads up to 64 MiB by default. Set a byte limit when converting a

filter.go

@@ -0,0 +1,184 @@
+package main
+
+import (
+	"fmt"
+	"io"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+
+	"github.com/hkwi/git-confluence/internal/attachment"
+	"github.com/hkwi/git-confluence/internal/confluence"
+	"github.com/hkwi/git-confluence/internal/logging"
+	"github.com/hkwi/git-confluence/internal/pagecache"
+)
+
+func filterClean(path string, input io.Reader, output, errorOutput io.Writer, maxInput int64, maxDepth int) error {
+	if isAttachmentPath(path) {
+		return attachment.Clean(input, path, output)
+	}
+	data, err := readLimitedInput(input, maxInput)
+	if err != nil {
+		return err
+	}
+	logger := logging.New(errorOutput).With(
+		"app", "git-confluence", "path", path,
+		"filter", "clean", "direction", "markdown_to_confluence_storage",
+		"purpose", "normalize_for_git",
+	)
+	logger.Info("clean filter started", "bytes", len(data))
+	cache, err := pagecache.Open()
+	if err != nil {
+		return err
+	}
+	originalStorage, ok, err := cache.Original(data, path)
+	if err != nil {
+		return fmt.Errorf("read cached original Confluence storage: %w", err)
+	}
+	if ok {
+		logger.Info("clean filter restored original storage", "bytes", len(originalStorage), "source", "page_cache")
+		_, err = output.Write(originalStorage)
+		return err
+	}
+	if pagecache.MatchesIndex(path, data) {
+		logger.Info("clean filter preserved indexed storage", "bytes", len(data), "source", "git_index")
+		_, err = output.Write(data)
+		return err
+	}
+	storage, err := confluence.MarkdownToStorageWithMaxDepth(string(data), maxDepth)
+	if err != nil {
+		return err
+	}
+	_, err = io.WriteString(output, storage)
+	if err == nil {
+		logger.Info("clean filter completed", "bytes", len(storage))
+	}
+	return err
+}
+
+func filterSmudge(path string, input io.Reader, output, errorOutput io.Writer, maxInput int64, maxDepth int) error {
+	data, err := readLimitedInput(input, maxInput)
+	if err != nil {
+		return err
+	}
+	if isAttachmentPath(path) || attachment.IsPointer(data) {
+		return attachment.Smudge(data, path, output, errorOutput)
+	}
+	logger := logging.New(errorOutput).With(
+		"app", "git-confluence", "path", path,
+		"filter", "smudge", "direction", "confluence_storage_to_markdown",
+		"purpose", "materialize_worktree",
+	)
+	logger.Info("smudge filter started", "bytes", len(data))
+	markdown, err := confluence.StorageToMarkdownWithMaxDepth(string(data), maxDepth)
+	if err != nil {
+		return err
+	}
+	cache, err := pagecache.Open()
+	if err != nil {
+		return err
+	}
+	if err := cache.Remember([]byte(markdown), path, data); err != nil {
+		return fmt.Errorf("cache original Confluence storage: %w", err)
+	}
+	_, err = io.WriteString(output, markdown)
+	if err == nil {
+		logger.Info("smudge filter completed", "bytes", len(markdown))
+	}
+	return err
+}
+
+func isAttachmentPath(path string) bool {
+	path = "/" + strings.Trim(filepath.ToSlash(path), "/") + "/"
+	return strings.Contains(path, "/attachments/")
+}
+
+func installFilter(args []string) error {
+	scope := "--global"
+	if len(args) > 1 {
+		return fmt.Errorf("install accepts at most one scope")
+	}
+	if len(args) == 1 {
+		scope = args[0]
+	}
+	if scope != "--global" && scope != "--local" {
+		return fmt.Errorf("install scope must be --global or --local")
+	}
+	settings := [][2]string{
+		{"filter.confluence.clean", "git-confluence filter-clean %f"},
+		{"filter.confluence.smudge", "git-confluence filter-smudge %f"},
+		{"filter.confluence.required", "true"},
+	}
+	for _, setting := range settings {
+		cmd := exec.Command("git", "config", scope, setting[0], setting[1])
+		if output, err := cmd.CombinedOutput(); err != nil {
+			return fmt.Errorf("git config %s: %w: %s", setting[0], err, strings.TrimSpace(string(output)))
+		}
+	}
+	return nil
+}
+
+func pullFiles(paths []string) error {
+	if err := installFilter([]string{"--local"}); err != nil {
+		return err
+	}
+	if len(paths) == 0 {
+		output, err := exec.Command("git", "ls-files", "-z").Output()
+		if err != nil {
+			return fmt.Errorf("list tracked files: %w", err)
+		}
+		for _, path := range strings.Split(string(output), "\x00") {
+			if path != "" && isAttachmentPath(path) {
+				paths = append(paths, path)
+			}
+		}
+	}
+	if len(paths) == 0 {
+		return nil
+	}
+	pathspecs := make([]string, 0, len(paths))
+	for index, path := range paths {
+		clean := filepath.Clean(path)
+		if clean != path || filepath.IsAbs(clean) || clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator)) || !isAttachmentPath(clean) {
+			return fmt.Errorf("%s is not an attachment path", path)
+		}
+		paths[index] = clean
+		pathspecs = append(pathspecs, ":(literal)"+filepath.ToSlash(clean))
+	}
+	diffArgs := append([]string{"diff", "--quiet", "--"}, pathspecs...)
+	if err := exec.Command("git", diffArgs...).Run(); err != nil {
+		return fmt.Errorf("attachment working files have local changes; refusing to overwrite")
+	}
+	cachedDiffArgs := append([]string{"diff", "--cached", "--quiet", "HEAD", "--"}, pathspecs...)
+	if err := exec.Command("git", cachedDiffArgs...).Run(); err != nil {
+		return fmt.Errorf("attachment index entries have staged changes; refusing to overwrite")
+	}
+	for _, path := range paths {
+		if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
+			return fmt.Errorf("prepare attachment %s for materialization: %w", path, err)
+		}
+	}
+	const batchSize = 100
+	for start := 0; start < len(paths); start += batchSize {
+		end := start + batchSize
+		if end > len(paths) {
+			end = len(paths)
+		}
+		args := append([]string{"checkout", "HEAD", "--"}, pathspecs[start:end]...)
+		cmd := exec.Command("git", args...)
+		if output, err := cmd.CombinedOutput(); err != nil {
+			return fmt.Errorf("materialize attachments: %w: %s", err, strings.TrimSpace(string(output)))
+		}
+	}
+	for _, path := range paths {
+		data, err := os.ReadFile(path)
+		if err != nil {
+			return fmt.Errorf("verify materialized attachment %s: %w", path, err)
+		}
+		if attachment.IsPointer(data) {
+			return fmt.Errorf("attachment %s was not materialized; configure a Confluence PAT or check the download error", path)
+		}
+	}
+	return nil
+}