Serve a stored post's attachments as Documents, with their mime

appinfo/info.xml

@@ -31,7 +31,7 @@ What it does:
 - 👋 A short introduction on the first visit: your address, who to follow, and a way to bring the follows you already have
 
 You can pin your own posts to the top of your profile, bookmark any post, and see which hashtags the instance is using most. This is a partial implementation of ActivityPub and of the Mastodon client API. Blocking, muting and reporting (with a moderation panel in the administration settings) are supported, as are locked accounts with approvable follow requests. Polls are fully supported: create your own, and view and vote on federated ones. Profiles carry an avatar, a banner image and up to four profile metadata fields. Posts that link somewhere get a link preview card. Image (JPEG, PNG, GIF, WebP, AVIF, and HEIC from an iPhone where the server can read it), video (MP4, WebM, QuickTime) and audio attachments are supported, up to ten per post, each with alt text and a focal point. It does not offer lists.]]></description>
-	<version>0.19.35</version>
+	<version>0.19.36</version>
 	<licence>agpl</licence>
 	<author mail="benedikt.schaechner@web.de" homepage="https://benedikt.xn--schchner-2za.de">Benedikt Schächner</author>
 	<namespace>Social</namespace>

composer.json

@@ -2,7 +2,7 @@
 	"name": "nextcloud/social",
 	"description": "Social app",
 	"license": "AGPL-3.0-or-later",
-	"version": "0.19.35",
+	"version": "0.19.36",
 	"minimum-stability": "stable",
 	"authors": [
 		{

docs/API.md

@@ -158,7 +158,7 @@ A `Relationship` carries `id` as a **string**, like every other id on the wire:
 
 The response is the status itself in local format.
 
-In that format, `in_reply_to_id` and `in_reply_to_account_id` carry the parent status's numeric id and its author's, resolved from the stored ActivityPub id (memoised per request, so a thread's replies are one lookup). Both are `null` for a post that is not a reply and for a parent this instance has never seen. `tags` is Mastodon's `[{name, url}]`; `Note::jsonSerialize()` still emits the app's own `hashtags: ["foo"]` alongside it when the status is exported with complete details. `edited_at` is when the post was last edited, or `null`. `quote` is Mastodon 4.5's Quote entity — `{state, quoted_status}` — or `null` for a post that quotes nothing. `state` comes from the quoted author's approval, not from whether this instance holds the post: `pending` until they answer, then `accepted`, `rejected`, or `revoked` if they take it back. `quoted_status` carries the quoted post inline only when the quote is accepted **and** the viewer may read it — an accepted quote of a post this reader may not see is `accepted` with a null `quoted_status`, so a quote never becomes a way of reading somebody's followers-only post. Mention ids are strings even when the handle could not be resolved. `attachment` — an ActivityPub-named duplicate of `media_attachments` — is no longer part of the client format.
+In that format, `in_reply_to_id` and `in_reply_to_account_id` carry the parent status's numeric id and its author's, resolved from the stored ActivityPub id (memoised per request, so a thread's replies are one lookup). Both are `null` for a post that is not a reply and for a parent this instance has never seen. `tags` is Mastodon's `[{name, url}]`; `Note::jsonSerialize()` still emits the app's own `hashtags: ["foo"]` alongside it when the status is exported with complete details. `edited_at` is when the post was last edited, or `null`. `quote` is Mastodon 4.5's Quote entity — `{state, quoted_status}` — or `null` for a post that quotes nothing. `state` comes from the quoted author's approval, not from whether this instance holds the post: `pending` until they answer, then `accepted`, `rejected`, or `revoked` if they take it back. `quoted_status` carries the quoted post inline only when the quote is accepted **and** the viewer may read it — an accepted quote of a post this reader may not see is `accepted` with a null `quoted_status`, so a quote never becomes a way of reading somebody's followers-only post. Mention ids are strings even when the handle could not be resolved. `attachment` — an ActivityPub-named duplicate of `media_attachments` — is no longer part of the client format. Each `MediaAttachment` carries one key beyond Mastodon's: `media_type`, the full mime (`image/png`), which is what the ActivityPub `Document` the post is served as states; a Mastodon client ignores it.
 
 ### Scheduled statuses
 

docs/Architecture.md

@@ -7,7 +7,7 @@ Nextcloud Social is a federated social networking app built on the W3C ActivityP
 **App ID:** `social`  
 **Namespace:** `OCA\Social`  
 **License:** AGPL-3.0-or-later  
-**App version:** 0.19.35  
+**App version:** 0.19.36  
 **Supported Nextcloud versions:** 35 – 36  
 **Supported PHP versions:** 8.3 – 8.5  
 

docs/Mastodon-Compatibility.md

@@ -564,10 +564,10 @@ missing. All of them have since landed.
 |---|---|---|---|---|
 | 11 | **Authorized fetch inbound** — verify the HTTP signature on GET and resolve the remote reader | Weeks | Signature verification ran on inbox POSTs only, so a followers-only object could not be served to an authorized remote reader and secure mode was impossible. It failed closed, so nothing leaked | done (`AuthorizedFetchService`) |
 | 12 | **`Add` and `Remove` outbound** for pins | Days | A pin was only visible to a peer that re-polled `featured` | done |
-| 13 | **`mediaType` on attachments** | Hours | See 39: the field exists and the served object does not use it | **reopened** |
+| 13 | **`mediaType` on attachments** | Hours | The stored row (`MediaAttachment::asLocal()`) now carries `media_type`, `import()` reads it back — with a guess from the extension for rows written before it existed — and the Document a post is served as states it | done |
 | 14 | **The WebFinger profile-page link** | Hours | Pointed at the Nextcloud user profile rather than a Social one | done |
 | 15 | **Emoji reactions** | Days | Announcement reactions are stored and served. Reactions to a *status* are a Misskey and Pleroma extension Mastodon does not handle either, and are deliberately not implemented | done (announcements) |
-| 39 | **Serve attachments as ActivityPub `Document`s** | Hours | New, and verified on the wire rather than in a unit test: everything served on request — a single status, the outbox, `featured`, `replies`, and any re-fetch by a peer — carries Mastodon's *client* shape under `attachment` (`"type": "video"`, `preview_url`, `remote_url`, `meta`) instead of `{"type": "Document", "mediaType": "video/mp4", "name": …}`. `MediaAttachment::asDocument()` is correct and `ACore::FORMAT_ACTIVITYPUB` is set on the attachments of a freshly created post, so the original `Create` goes out right; but `StreamRequest::save()` stores `asLocal()` and hydration leaves the objects in the local format, so every later read of the same post is wrong. `WireCompatibilityTest` calls `asDocument()` directly and therefore passes | **open** |
+| 39 | **Serve attachments as ActivityPub `Document`s** | Hours | New, and verified on the wire rather than in a unit test: everything served on request — a single status, the outbox, `featured`, `replies`, and any re-fetch by a peer — carries Mastodon's *client* shape under `attachment` (`"type": "video"`, `preview_url`, `remote_url`, `meta`) instead of `{"type": "Document", "mediaType": "video/mp4", "name": …}`. `MediaAttachment::asDocument()` is correct and `ACore::FORMAT_ACTIVITYPUB` is set on the attachments of a freshly created post, so the original `Create` goes out right; but `StreamRequest::save()` stores `asLocal()` and hydration leaves the objects in the local format, so every later read of the same post is wrong. `WireCompatibilityTest` calls `asDocument()` directly and therefore passes | done — `Stream::jsonSerialize()` maps every attachment through `asDocument()` whatever format it was hydrated in, so a re-read post goes out the same as the original `Create`; `StreamTest::testAHydratedPostServesItsAttachmentsAsDocuments` pins it |
 
 ### Tier 4 — the admin and moderation surface
 

lib/Model/ActivityPub/Stream.php

@@ -1599,12 +1599,34 @@ public function jsonSerialize(): array {
 		// already carries them as `media_attachments`, and a second copy under
 		// a key Mastodon does not define was only ever confusing.
 		if ($this->getExportFormat() !== self::FORMAT_LOCAL) {
-			$result['attachment'] = $this->getAttachments();
+			// as Documents, whatever format the attachments themselves are in:
+			// a post read back from the database has them hydrated in the local
+			// format, and served like that a peer got Mastodon's client entity
+			// under an ActivityPub key -- Pixelfed dropped every picture
+			$result['attachment'] = array_map(
+				static fn (MediaAttachment|array $attachment): array => self::asWireDocument($attachment),
+				$this->getAttachments()
+			);
 		}
 
 		return $result;
 	}
 
+	/**
+	 * One attachment as the wire carries it. A caller may hand the list over
+	 * as arrays already shaped for the wire; those go through as they are.
+	 *
+	 * @param MediaAttachment|array<string, mixed> $attachment
+	 * @return array<string, mixed>
+	 */
+	private static function asWireDocument(MediaAttachment|array $attachment): array {
+		if ($attachment instanceof MediaAttachment) {
+			return $attachment->asDocument();
+		}
+
+		return $attachment;
+	}
+
 	/**
 	 * The client format is a fixed key set, and nothing may be filtered out of
 	 * it.

lib/Model/Client/MediaAttachment.php

@@ -165,6 +165,12 @@ public function import(array $data): self {
 		$this->setId($this->get('id', $data));
 		$this->setType($this->get('type', $data));
 		$this->setUrl($this->get('url', $data));
+		// this app's own key on the stored row (see asLocal()); a row from
+		// before it existed, or a Mastodon entity, has none and gets a guess
+		$this->setMediaType($this->get('media_type', $data, $this->get('mediaType', $data, '')));
+		if ($this->getMediaType() === '') {
+			$this->setMediaType(self::guessMediaType($this->getType(), $this->getUrl()));
+		}
 		$this->setPreviewUrl($this->get('preview_url', $data));
 		$this->setRemoteUrl($this->get('remote_url', $data));
 		$this->setDescription($this->get('description', $data));
@@ -206,6 +212,11 @@ public function asLocal(): array {
 		return [
 			'id' => $this->getId(),
 			'type' => $this->getType(),
+			// not Mastodon's: the full mime, kept so that the ActivityPub
+			// Document this row is served as later can state it. Pixelfed
+			// refuses an attachment without one; Mastodon sniffs the URL and
+			// hid for weeks that every re-served post had lost it.
+			'media_type' => $this->getMediaType(),
 			'url' => $this->onThisInstance($this->getUrl()),
 			'preview_url' => ($preview === null || $preview === '') ? null : $preview,
 			'remote_url' => ($remote === null || $remote === '') ? null : $remote,
@@ -218,6 +229,32 @@ public function asLocal(): array {
 		];
 	}
 
+	/**
+	 * The mime a row written before `media_type` was stored most likely has:
+	 * from the file extension where there is one, else the commonest type of
+	 * its kind. A guess, but a guess Pixelfed accepts; '' it refuses.
+	 */
+	public static function guessMediaType(string $type, string $url): string {
+		$extension = strtolower(pathinfo(parse_url($url, PHP_URL_PATH) ?? '', PATHINFO_EXTENSION));
+		$byExtension = [
+			'jpg' => 'image/jpeg', 'jpeg' => 'image/jpeg', 'png' => 'image/png', 'gif' => 'image/gif',
+			'webp' => 'image/webp', 'avif' => 'image/avif', 'heic' => 'image/heic', 'heif' => 'image/heif',
+			'mp4' => 'video/mp4', 'm4v' => 'video/mp4', 'webm' => 'video/webm', 'mov' => 'video/quicktime',
+			'mp3' => 'audio/mpeg', 'm4a' => 'audio/mp4', 'ogg' => 'audio/ogg', 'oga' => 'audio/ogg',
+			'opus' => 'audio/opus', 'wav' => 'audio/wav', 'flac' => 'audio/flac', 'aac' => 'audio/aac',
+		];
+		if (isset($byExtension[$extension])) {
+			return $byExtension[$extension];
+		}
+
+		return match ($type) {
+			'image' => 'image/jpeg',
+			'video', 'gifv' => 'video/mp4',
+			'audio' => 'audio/mpeg',
+			default => '',
+		};
+	}
+
 	/**
 	 * A media link pointed at the address this instance answers on now.
 	 *

package.json

@@ -1,7 +1,7 @@
 {
 	"name": "social",
 	"description": "Nextcloud becomes part of the federated social networks!",
-	"version": "0.19.35",
+	"version": "0.19.36",
 	"authors": [
 		{
 			"name": "Benedikt Schächner",

tests/Model/ActivityPub/StreamTest.php

@@ -168,6 +168,35 @@ public function testImportBuildsMediaAttachmentsThroughTheDocumentAndImageInterf
 		$this->assertStringEndsWith('.png', $attachments[1]->getUrl());
 	}
 
+	/**
+	 * A post read back from the database has its attachments hydrated in the
+	 * local format. Served to a peer, they went out as Mastodon's client
+	 * entity under the ActivityPub key -- no `mediaType`, no Document -- so
+	 * every re-served post lost its pictures on Pixelfed.
+	 */
+	public function testAHydratedPostServesItsAttachmentsAsDocuments(): void {
+		$stream = new Stream();
+		$stream->setId('https://cloud.example/apps/social/@alice/1');
+		$attachment = (new MediaAttachment())->import([
+			'id' => '7', 'type' => 'image', 'media_type' => 'image/png',
+			'url' => 'https://cloud.example/apps/social/media/abc', 'preview_url' => null, 'remote_url' => null,
+			'meta' => ['original' => ['width' => 640, 'height' => 480]], 'description' => 'a cat', 'blurhash' => 'LEHV6n',
+		]);
+		$stream->setAttachments([$attachment]);
+		$this->assertSame(ACore::FORMAT_LOCAL, $attachment->getExportFormat(), 'hydrated in the local format, as importFromDatabase() leaves it');
+
+		$stream->setExportFormat(ACore::FORMAT_ACTIVITYPUB);
+		$served = $stream->jsonSerialize()['attachment'];
+
+		$this->assertSame('Document', $served[0]['type']);
+		$this->assertSame('image/png', $served[0]['mediaType']);
+		$this->assertSame('a cat', $served[0]['name']);
+		$this->assertArrayNotHasKey('preview_url', $served[0], 'not the client entity');
+
+		$stream->setExportFormat(ACore::FORMAT_LOCAL);
+		$this->assertArrayNotHasKey('attachment', $stream->jsonSerialize());
+	}
+
 	public function testAnAbsurdAttachmentListIsCappedRatherThanImported(): void {
 		// a signed Create is authenticated, not trusted: each entry is a row
 		// written and a file queued inside the inbox request
@@ -499,7 +528,8 @@ public function testJsonSerializeExposesTheAttachments(): void {
 		$stream = new Note();
 		$stream->setAttachments([$media]);
 
-		$this->assertSame([$media], $stream->jsonSerialize()['attachment']);
+		// as Documents, whatever format the attachment objects are in
+		$this->assertSame([$media->asDocument()], $stream->jsonSerialize()['attachment']);
 	}
 
 	/**

tests/Model/Client/MediaAttachmentTest.php

@@ -90,11 +90,41 @@ public function testAsLocalNullsEmptyFieldsRatherThanDroppingThem(): void {
 		$this->assertArrayHasKey('description', $local);
 		$this->assertNull($local['description']);
 		$this->assertSame(
-			['id', 'type', 'url', 'preview_url', 'remote_url', 'meta', 'description', 'blurhash'],
+			['id', 'type', 'media_type', 'url', 'preview_url', 'remote_url', 'meta', 'description', 'blurhash'],
 			array_keys($local)
 		);
 	}
 
+	/**
+	 * The stored row is what a post is served from for ever after, and the
+	 * Document it is served as has to state its mime: Pixelfed refuses an
+	 * attachment without one. Mastodon's entity has no such key, so it is
+	 * this app's own, and a row from before it was written gets a guess.
+	 */
+	public function testTheMimeSurvivesTheStoredRow(): void {
+		$media = (new MediaAttachment())->import($this->mastodonAttachment());
+		$media->setMediaType('image/png');
+
+		$stored = $media->asLocal();
+		$this->assertSame('image/png', $stored['media_type']);
+
+		$again = (new MediaAttachment())->import($stored);
+		$this->assertSame('image/png', $again->getMediaType());
+		$this->assertSame('image/png', $again->asDocument()['mediaType']);
+	}
+
+	public function testARowWithoutTheMimeGetsAGuessRatherThanNothing(): void {
+		// a Mastodon entity, or a row written before media_type was stored
+		$media = (new MediaAttachment())->import($this->mastodonAttachment());
+		$this->assertSame('image/jpeg', $media->getMediaType(), 'from the .jpg');
+
+		$this->assertSame('video/quicktime', MediaAttachment::guessMediaType('video', 'https://x.example/a/b.MOV?x=1'));
+		$this->assertSame('image/jpeg', MediaAttachment::guessMediaType('image', 'https://x.example/media/uuid'));
+		$this->assertSame('video/mp4', MediaAttachment::guessMediaType('gifv', 'https://x.example/media/uuid'));
+		$this->assertSame('audio/mpeg', MediaAttachment::guessMediaType('audio', ''));
+		$this->assertSame('', MediaAttachment::guessMediaType('unknown', ''));
+	}
+
 	public function testTheMetaIsAnObjectOnTheWire(): void {
 		$media = new MediaAttachment();
 		$media->import($this->mastodonAttachment());
@@ -124,7 +154,8 @@ public function testAsDocumentBuildsAnActivityPubDocument(): void {
 
 		$this->assertSame([
 			'type' => 'Document',
-			'mediaType' => '',
+			// Mastodon's entity carries no mime; the .jpg says what it is
+			'mediaType' => 'image/jpeg',
 			'url' => 'https://files.mastodon.social/media/cat.jpg',
 			// the wire carries the alt text as `name`
 			'name' => 'A cat',