Skip to content

Sync EMBER-DANDI production - #206

Merged
NEStock merged 321 commits into
prodfrom
apl-setup
Oct 27, 2026
Merged

NEStock merged 321 commits into
prodfrom
apl-setup

Conversation

@NEStock

@NEStock NEStock commented Oct 27, 2026

Copy link
Copy Markdown
Member

Closes #201

Update EMBER-DANDI production to https://github.com/dandi/dandi-archive/releases/tag/v1.1.0

bendichter and others added 30 commits May 5, 2026 16:55
- Drop the `\"` claim from the unbalanced-quote message; the parser
  doesn't support that escape (no_R2#2).
- Empty operator value (e.g. `species:" "`) now raises 400 instead of
  being silently dropped (no_R2#6).
- Override OpenAPI help text on the `search` parameter to document the
  operator vocabulary instead of "Search terms to filter the results"
  (no_R2#5).
- Acknowledge sequential-scan trade-off on jsonpath operators with a
  comment, and document cross-key AND-on-same-asset semantics in the
  apply_search_filters body (no_R2#3, #4 — comment-only, indexes
  deferred until needed).
- Pin same-key AND semantics with a new test
  test_advanced_search_repeated_same_key_operator_combines_with_and.
- Nits: drop no-op re.DOTALL on _BARE_OP_RE; fix "incompatible
  previously generated" → "incompatible with previously generated"
  comment in views/dandiset.py; trim __all__ in services.search to
  the public surface (parse_search, SearchSyntaxError); add a clarifying
  comment to the cross-key intersect test.
Replace the v-row + v-list-subheader with a v-card-title so the
heading matches the "Owners" and "Dandiset Actions" cards on the
dandiset landing page.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix "This Version" header style on DLP
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
Per @yarikoptic: it's a one-time-use helper that returns None just so
the caller can raise. Drop the indirection — strptime + try/except in
place, raise SearchSyntaxError directly from the except.
Gmail-style advanced search for dandiset listing
The popover was hard-capped at max-width: 420px, which forced the
description column to wrap awkwardly even when there was plenty of
viewport room.

Use a responsive cap (min-width: 420px; max-width: min(80vw, 720px))
so the popover comfortably fits the longest operator example on
ordinary monitors and shrinks gracefully on small screens. Also pin
the example column with white-space: nowrap so operator strings stay
readable as code.
…opover

Wider, responsive advanced-search help popover (closes dandi#2818)
Filters dandisets to those owned by a given user. The value is matched
case-insensitively against User.username OR User.email. The special form
`owner:me` resolves to the requesting user (consistent with the existing
?user=me query parameter) and returns 400 if the request is anonymous.

Implementation reuses the existing `get_owned_dandisets()` permission
helper. We pass `with_superuser=False` so `owner:admin` returns only
what admin explicitly owns — guardian's default would otherwise inflate
to the entire archive for any superuser.

Unknown users return zero results (not an error): a search for a
nonexistent owner is a valid 0-hit query.

Tests cover username/email lookup, case-insensitivity, unknown user,
`owner:me` for an authenticated user, anonymous `owner:me` → 400, the
superuser non-inflation guarantee, and combination with other operators.

OpenAPI help text and the frontend operator popover updated.
Real users encounter the dandiset list with owners shown by display
name (e.g. "Super User"), not by username. Searching that string was
returning 0 because the lookup only matched username/email.

Now matches case-insensitively against username, email, first_name,
last_name, OR "first_name last_name" — so owner:"Super User" works
the same as owner:ben.dichter@gmail.com.

Multiple users may match (e.g. shared last name); we union dandisets
owned by any of them via a direct DandisetUserObjectPermission query.
Updated OpenAPI help text and the frontend popover example to
`owner:"Jane Doe"` so users discover the new shape.
Round-2 review feedback on dandi#2821:

- @yarikoptic flagged that owner:me silently shadows a real user named "Me".
  Fix: distinguish quoted vs unquoted at the parser level. Unquoted
  owner:me → magic alias for the requesting user. Quoted owner:"me" →
  literal lookup (matches a user whose first/last name is "Me"). Same
  pattern lets owner:"Me Someoneyou" reach the literal full-name match
  while keeping the convenient owner:me shortcut.

  Implementation: ParsedSearch.operators is now a list of `Operator`
  dataclasses (key, value, quoted) instead of bare tuples. Filters
  consume the new shape and the owner filter switches on the quoted
  flag.

- Replaced personal email (ben.dichter@gmail.com) in the full-name test
  fixture with a generic example user.

- Consolidated 10 small owner-tests into 3 denser ones that share setup
  per @yarikoptic's "make each test matter more" feedback. Coverage is
  unchanged (every documented lookup path is asserted; cross-key AND
  with another operator; multi-user union via shared last name; unknown
  user → 0; superuser non-inflation; owner:me magic; owner:"me"
  literal-escape; anonymous owner:me → 400). DB setup runs ~3x instead
  of ~10x.

Updated OpenAPI help text and the search popover to mention the
owner:me alias and the quoted-escape.
The unquoted owner:me → current-user shortcut required threading a
`quoted` flag through the parser and a `request_user` arg through the
filter dispatch — non-trivial machinery to support one alias.

Per dandi#2822 review discussion, removing it from this PR keeps the owner
operator focused on literal lookup-by-value (username / email / first /
last / "first last") and avoids the design debate about the right escape
mechanism for "I literally want a user named Me." The alias can come
back in a focused follow-up PR if/when there's appetite for it.

Concrete drops:
- owner:me magic + 400-on-anonymous in `_apply_owner_filter`
- `Operator.quoted` field on the parser dataclass
- `quoted` and `request_user` parameters on `_apply_owner_filter`
- `get_owned_dandisets` import (no longer used here)
- `test_advanced_search_owner_me_magic_and_literal_escape` test
- The two `owner-me-quoted` / `owner-me-unquoted` parser test cases
- "owner:me" mentions in OpenAPI help text and the popover entry
Each citation formatter now early-returns when there are no authors,
with a dedicated template per format. The previous approach stuffed
the title into authorString and then blanked out the title slot
downstream, which conflated two variables (authorString held the
title) and forced six near-identical workarounds.

Cleanups in the no-author path:
- MLA, Chicago: period inside the closing quote ("Title.")
- IEEE: comma inside the closing quote ("Title,")
- authorString is now typed as string and only holds authors

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…lter

[codex] Fix citation author filtering
Fix documentation link in email template
…bitmq-4.3

Fix native Celery instructions for RabbitMQ 4.3
…tions/add-to-project-2.0.0

[gh-actions](deps): Bump actions/add-to-project from 1.0.2 to 2.0.0
Add support to open NIfTI assets in Neurosift
* Design doc for S3 backup

* Fix outdated info

We are no longer considering the use of a bucket in a different region.

* Update cost analysis with accurate inputs

We are expecting a bulk of 6PB over the next 5 years, not 30PB.

* Fix typo in bulk retrieval cost

* Add worst-case disaster recovery cost estimate

* Add missing periods

Co-authored-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com>

* Improve readability of LaTeX and fix conversion units

Co-authored-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com>

* Fix conversion units

When the AWS docs say "GB", they mean 10^9 bytes, not 2^30.

Co-authored-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com>

* Fix dollar amount formatting

Co-authored-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com>

* Improve readability and fix conversion factors

* Fix cost analysis

Clarify purpose of calculating the expected bucket storage cost covered
by AWS already.

* Merge pull request dandi#2829 from CodyCBakerPhD/patch-1

Update status for initial s3 backup doc

---------

Co-authored-by: Roni Choudhury <roni.choudhury@kitware.com>
Co-authored-by: Roni Choudhury <2903332+waxlamp@users.noreply.github.com>
Co-authored-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com>
Display total data size on the homepage with one decimal place
Show the embargo end date next to the access status in the
landing page's Access Information section, so owners, curators,
and reviewers can see when an embargo lifts without querying the
API. Falls back to the plain status when embargoedUntil is absent.

Closes dandi#2832

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e-snippet

Update code snippet to show correct instance
Updated the S3 backup documentation to include detailed sections on proposed solutions, requirements, advantages, limitations, and cost analysis for various backup strategies.
Searching from the "My Dandisets" page silently fell through to the
public search, omitting the user's embargoed dandisets (dandi#2834).

- DandisetSearchField: keep the search in context on the myDandisets
  route instead of always pushing to the public searchDandisets route.
- DandisetsPage: send the `search` param whenever a search is active
  (not only on the dedicated search view). Combined with the existing
  user=me and embargoed=true params, search now spans owned + embargoed
  dandisets. Result count and search-error alert use the same flag.
- MyDandisetsView: show a search-aware empty state instead of the
  misleading "You haven't created any dandisets yet".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface the available GitHub-style search operators (species:, approach:,
created_after:, etc.) in a dropdown beneath the search field to make them
discoverable, as requested in dandi#2828.

- Suggestions are derived from the existing operatorHelp list, so the help
  table and the dropdown stay in sync.
- Filters by substring as the user types the operator token; matches the
  token under the cursor, so operators work mid-query too.
- Keyboard support: up/down to move, Enter/Tab to insert the highlighted
  operator, Esc to dismiss. When browsing the full list (empty token)
  nothing is highlighted, so Enter still submits the search.
- The list is teleported to <body> and positioned manually so the
  toolbar's overflow:hidden doesn't clip it; focus stays in the input.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Display the example value (e.g. the 2024-01-01 date format for date
operators) as a muted hint after each operator key, so users can see the
expected value format. Derived from the same operatorHelp examples.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `standard:` operator matched `$.dataStandard[*].name` against asset
metadata, but `dataStandard` is reported on the Version's assetsSummary,
not on individual assets — so it never matched and always returned 0
results in production.

Match `standard:` against the Version metadata
(`$.assetsSummary.dataStandard[*].name`) instead, restricting the
dandiset queryset to dandisets with a matching version. The existing
test was rewritten to seed `dataStandard` on the version assetsSummary
(where it actually lives) rather than on the asset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
waxlamp and others added 24 commits October 6, 2026 11:25
Use correct environment variable to configure Sentry
…nents) (dandi#2929)

* Add vuetify-component-api skill from PR dandi#2926 review

Encodes the practice applied in review of PR dandi#2926: prefer Vuetify's own
component props and utility classes over custom scoped CSS, wrapper
links, and imperative click handlers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPduPSnJUhEFREA6FSPkeQ

* Add AGENTS.md pointing at the vuetify-component-api skill

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPduPSnJUhEFREA6FSPkeQ

---------

Co-authored-by: Claude <noreply@anthropic.com>
* Add table visualizer for .tsv/.csv assets in the file browser (#15)

* Add table viewer for .tsv/.csv assets in the file browser

Clicking a .tsv or .csv asset in the file browser previously navigated away
to the raw file. Instead, open it in a dialog that renders the file as a
sortable, searchable table.

- `utils/tabular.ts`: extension-to-delimiter detection plus an RFC 4180 style
  parser handling quoted fields, embedded delimiters/newlines and escaped
  quotes.
- `TableViewerDialog.vue`: fetches the asset via its inline URI, renders it in
  a `v-data-table`, and offers a toggle for whether the first row is a header.
  Files over 50 MB are not fetched and tables are capped at 5000 rows; the
  dialog always links out to the raw file and the download endpoint.
- The file browser gains a table icon for tabular assets, so the viewer is
  reachable without changing which row is clicked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018cHVjL6MP4kgJLH3NtSm7c

* Fetch table contents from the direct S3 URL when not embargoed

The viewer fetched the asset through the API's inline-download endpoint,
which 302s to a presigned S3 URL. That makes the browser fetch depend on
CORS at both hops.

Use the same URL selection the external-service integrations already make:
the direct S3 URL (`asset.url`), which third-party viewers such as
Neurosift already fetch cross-origin today, falling back to the API URL for
embargoed dandisets where the presigned S3 link isn't usable. The raw-view
and download buttons still point at the API endpoints.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018cHVjL6MP4kgJLH3NtSm7c

* Always treat row 1 as header, linkify URLs, share view via URL

Four changes from review feedback:

- Drop the "first row is a header" toggle. A header row is a safe assumption
  for the .tsv/.csv files in a dandiset, and the checkbox was clutter.
- Render cell values that are http(s) URLs as links, opening in a new tab.
  `isUrl` parses with the URL constructor and accepts only http/https, so
  cell contents can't produce `javascript:` or `data:` links.
- Replace the "View raw file" link with an in-dialog raw-text toggle. The
  link pointed at the API's inline-download endpoint, which serves .tsv/.csv
  with a tabular content type, so browsers downloaded the file instead of
  displaying it -- making that button a duplicate of the download button
  beside it. The toggle shows the already-fetched text in the dialog.
- Record the open table in a `table` query parameter, so the URL can be
  shared and reopens the viewer. The route watcher only refetches the
  listing when the location or page changes, so opening and closing the
  viewer doesn't re-request it, and a parameter naming a non-tabular or
  missing asset is ignored rather than opening an empty dialog.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018cHVjL6MP4kgJLH3NtSm7c

* Keep the table's horizontal scrollbar in view

The dialog body scrolled as a whole, so a wide table put its horizontal
scrollbar below the last row: reaching it meant scrolling all the way down
first, and scrolling sideways then lost the rows you were reading.

Give the table body and the raw-text block their own bounded scroll areas
and stop the dialog body itself from scrolling. The horizontal scrollbar now
sits at the bottom of the visible rows, the sticky header stays put, and the
table footer remains on screen.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018cHVjL6MP4kgJLH3NtSm7c

---------

Co-authored-by: Claude <noreply@anthropic.com>

* Stop row-level row link from hijacking the download button

On a tabular row the list item is a router link to the table viewer, so a
click on the download or open-in-browser button bubbled up and the router
navigation won: the viewer opened instead of the file downloading. Stop the
click on those buttons, as the metadata button already does, without
preventing their own href.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

* Open tabular assets in the table viewer from the row's open button

Opening a .tsv/.csv asset raw only makes the browser download it, which
duplicated the download button. The open button now carries the same target
as the row itself: the raw asset for other files, the table viewer for
tabular ones, shown with a table icon. That folds the separate table button
into it, so each row has one view action.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

* Keep the open button uniform across file types

The open button needs no separate icon or wording for tabular assets: it
opens the asset, in the viewer that can show it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

* Skip the viewer for embargoed assets and cap the raw-text view

Embargoed assets are only reachable through the authenticated API, which the
viewer does not use, so they open raw as before: rowHref, rowRoute and the
route sync now share one viewableAsTable check. The API fetch itself goes
through the rest.ts client so its auth interceptor applies, while the public
S3 URL stays on bare axios, which S3 requires.

The raw-text view also no longer holds a whole 50 MB file in one <pre>: it
is clamped to the same MAX_ROWS lines, or 1 MB, whichever comes first, with a
truncation notice like the table's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

* Cap the viewer's rows-per-page at 1000 instead of All

Rendering every row of a 5000-row table at once is what the row cap exists to
avoid, so the page-size menu tops out at 1000.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

* Close the viewer by going back, and strip a leading BOM

Opening the viewer pushes a history entry, so closing it now goes back instead
of replacing: the back button lands where the user came from rather than
reopening the viewer. A link straight into the viewer has no entry to return
to, so it still drops the query parameter.

A byte order mark at the start of the file also no longer ends up inside the
first column's header name.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

* Warn when a quoted field is never closed

An unclosed quote makes the rest of the file one field, so the table silently
looks nothing like the file. The parser now reports the unterminated quote
along with its rows, and the viewer shows a warning pointing at the raw text
view.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

* Widen the page-size select so 1000 isn't clipped

Vuetify sizes that select at 90px, which fits its own two- and three-digit
options but ellipsises the 1000 this table offers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

* Cap cell width so one huge value can't stretch its column

A serialized object in a cell made its column hundreds of thousands of pixels
wide, leaving the rest of the table stranded off to the left. Cells are now
capped at 24em and ellipsised, with the value (itself capped) on hover.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

* Enable the viewer for embargoed assets

The viewer already fetches the API URL through the rest.ts client, whose
interceptor supplies the auth headers, and an embargoed asset already routes
to that URL rather than its ready-made S3 one, so the skip added for embargoed
dandisets is no longer needed. A rejected fetch now says so rather than
reporting a generic failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

* Confine the viewer's integration to its own files

The file browser carried the viewer's query parameter, route sync, open and
close state, and link helpers, which made this feature a 126-line change to a
file it has nothing to do with. All of it moves into TableViewer.vue, which
reads the URL itself, and tableViewer.ts, which says which items it can open.

What stays in the file browser is the link: tabular rows and the open button
point at tableViewerRoute(), the component is mounted, and the two buttons
with their own href stop the click so the row link isn't followed. Removing
the feature now means deleting four files and reverting those lines.

One behavior changes with it: opening or closing the viewer refetches the
listing, since the file browser refetches on any query change and no longer
carries the guard that made an exception for this parameter.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HifAb4uLDR696kwyXrYQva

---------

Co-authored-by: Claude <noreply@anthropic.com>
Skip test workflows for changes only to doc/
`user` is a ref, so `!user` was always false; check `user.value` instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Do not bump draft `modified` when assetsSummary is unchanged
Even though the button is `disabled`, the underlying `<a>` element was still there and could be selected and triggered via e.g. keyboard.

Co-authored-by: Michael Nagler <mike.nagler@kitware.com>
…shed

Disable AI meditor on published versions
@netlify

netlify Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for ember-dandi-archive ready!

Name Link
🔨 Latest commit 7614f30
🔍 Latest deploy log https://app.netlify.com/projects/ember-dandi-archive/deploys/6ae0c49008d40763c56a77e8
😎 Deploy Preview https://deploy-preview-206--ember-dandi-archive.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@NEStock NEStock self-assigned this Oct 27, 2026
@NEStock
NEStock merged commit 9e46068 into prod Oct 27, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.