Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
196 commits
Select commit Hold shift + click to select a range
c6e298a
fix: cleanup ingest code
akhileshh Aug 6, 2023
b053e96
add ttl column family
akhileshh Aug 6, 2023
d78d01d
fix: new l2 cx edge attribute
akhileshh Aug 6, 2023
17d4d6b
feat: post process sv cross edges
akhileshh Aug 6, 2023
bf3f502
fix: use longer expiry for debugging
akhileshh Aug 11, 2023
e751782
feat(ingest): read l2 cross edges
akhileshh Aug 11, 2023
0a8a0b3
feat(ingest): postprocess job handling
akhileshh Aug 12, 2023
aa9a93d
fix(ingest): status
akhileshh Aug 12, 2023
11992ad
fix: timedelta import
akhileshh Aug 12, 2023
a0d3efe
fix(ingest): status
akhileshh Aug 12, 2023
a199c5a
fix(ingest): use hypenated names for valid dns
akhileshh Aug 12, 2023
50a344b
fix: rename attr; better var names
akhileshh Aug 20, 2023
9c5e827
fix: rename attr; better var names
akhileshh Aug 20, 2023
f44c75a
fix: add more docs; better var names
akhileshh Aug 20, 2023
68dd790
fix: move cross_edges module to ingest module; only used in ingest
akhileshh Aug 20, 2023
7a27263
fix: reduce mem use; var names; remove unused code
akhileshh Aug 20, 2023
68e2f8e
fix: adds cg typehint
akhileshh Aug 20, 2023
3ceee80
fix: reduce loc
akhileshh Aug 20, 2023
ef995cd
fix: use shorter name
akhileshh Aug 20, 2023
c9633a4
feat: cache cx edges at each layer
akhileshh Aug 20, 2023
c5664fe
fix: convert array type
akhileshh Aug 20, 2023
6eb5a70
fix: use atomic edges during ingest
akhileshh Aug 20, 2023
e28a382
fix: tests
akhileshh Aug 20, 2023
e0f8390
fix: remove postprocess step
akhileshh Aug 20, 2023
5178e93
fix: raises specific error
akhileshh Aug 20, 2023
0034d0c
fix: removes dangerous default value
akhileshh Aug 21, 2023
965aa61
wip: read from cached edges
akhileshh Aug 21, 2023
8b4d9d7
wip: edits refactor
akhileshh Aug 21, 2023
bc6fbd0
wip: edits refactor
akhileshh Aug 21, 2023
ebd3537
fix(ingest): cache cross chunk edges from children
akhileshh Aug 22, 2023
c19e918
feat: add unique flag
akhileshh Aug 22, 2023
eaf98fa
feat: cross edges column family gcversionrule
akhileshh Aug 22, 2023
0595c17
fix: convert input to np arrays
akhileshh Aug 22, 2023
5b1144d
fix: linting issues
akhileshh Aug 22, 2023
4545f5e
wip: edits refactor
akhileshh Aug 22, 2023
a75a094
fix: undo gcrule changes
akhileshh Aug 23, 2023
bd971cc
fix: add mock_edges; linting issues
akhileshh Aug 23, 2023
99bb3da
feat: edits using cached cross edges
akhileshh Aug 23, 2023
cc5455b
fix: use function for dry code
akhileshh Aug 24, 2023
240ad53
fix: mask skipped nodes
akhileshh Aug 28, 2023
cebbadc
fix: use the correct layer variable
akhileshh Aug 28, 2023
ddefd4d
fix: redis pipeline for lower latency
akhileshh Aug 29, 2023
787b01f
fix: pass redis connection
akhileshh Aug 29, 2023
36d8514
fix: version update for deployment
akhileshh Aug 29, 2023
39c585d
fix: status print padding
akhileshh Aug 29, 2023
23eae9c
fix: filter active edges for split, add timestamp for reading cross c…
akhileshh Aug 30, 2023
b31e7db
fix: get roots no cache flag
akhileshh Aug 30, 2023
2c2148e
fix: parent and roots no cache
akhileshh Aug 31, 2023
40d0228
fix: out edges here dont refer to edges crossing chunk
akhileshh Aug 31, 2023
ecaaa14
fix: missing timestamps
akhileshh Sep 2, 2023
2e238c8
fix: consolidate neighbor nodes cx edge updates
akhileshh Sep 8, 2023
9d66c3c
fix: set to list for np.array
akhileshh Sep 8, 2023
18b294f
fix: use copy=False where possible; some cleanup
akhileshh Sep 8, 2023
15d8c9a
fix: attribute type must be np.array
akhileshh Sep 8, 2023
5af56fe
fix(ingest): worker details in status
akhileshh Sep 9, 2023
85d4f81
fix: handle empty input
akhileshh Sep 9, 2023
e5e53e1
fix: use empty array instead
akhileshh Sep 9, 2023
4bbc6a8
fix: missed time_stamp
akhileshh Sep 10, 2023
b72aee4
fix: only consolidate cx_edge writes; update per new_id
akhileshh Sep 10, 2023
8894149
fix: reset parent layer in loop
akhileshh Sep 11, 2023
1838ea5
fix(ingest): use get_roots with ceil=False instead of get_parents
akhileshh Sep 11, 2023
59961bc
fix(ingest): incorrect stop_layer
akhileshh Sep 11, 2023
37672c3
fix: add safeguard to against data corruption
akhileshh Sep 12, 2023
0064332
add another safeguard
akhileshh Sep 12, 2023
005a51f
feat: log operation_id in errors
akhileshh Sep 12, 2023
bbd735d
fix: remove temp error
akhileshh Sep 12, 2023
3623a29
add more safeguards
akhileshh Sep 12, 2023
3663b6e
fix: circular import
akhileshh Sep 12, 2023
d768f8b
fix: consider layer 2 as well
akhileshh Sep 12, 2023
44ea8b9
fix(edits): incorrect order of opeartions; documentation
akhileshh Sep 13, 2023
dc4b5e1
feat(ingest): add tests command
akhileshh Sep 15, 2023
5f69c98
fix(edits): make sure to add reverse edges
akhileshh Sep 26, 2023
1895e22
fix(edits): read neighbor cx edges from cache
akhileshh Sep 26, 2023
312148b
fix(edits): check for no cx edges; comments
akhileshh Sep 27, 2023
40ab9d3
fix(edits): update neighbor cx edges in a skipped layer
akhileshh Oct 3, 2023
ea797cb
fix(edits): make sure to update all skipped neighbors
akhileshh Oct 11, 2023
5fe4f0a
fix(edits): ignore new ids in neighbor update
akhileshh Oct 11, 2023
7fb0775
add docs
akhileshh Oct 12, 2023
ea5ca2e
fix: resolve column filter ambiguity
akhileshh Jan 14, 2024
0c4a54a
fix: resolve column filter ambiguity(2)
akhileshh Jan 14, 2024
8bc23c2
V3 migration (#484)
akhileshh May 12, 2024
e9f46e4
reset version v3
akhileshh May 12, 2024
956033b
breakup long fn
akhileshh May 12, 2024
a15f6f3
gh actions for pcgv3
akhileshh May 15, 2024
a86e1a7
update split tests (#497)
akhileshh May 25, 2024
2d2096b
segregate update nodes logic
akhileshh Jun 10, 2024
55e8897
fix(edits): overwrite children partners when superseded by parents
akhileshh Jun 28, 2024
8fc4c4b
fix: unique edges always, predecing edit ts, allow same segment merge
akhileshh Jul 4, 2024
4b62dc2
Bump version: 3.0.0 → 3.0.1
akhileshh Jul 4, 2024
72c6cf2
fix(edits): mask all descendants when updating cx edges
akhileshh Jul 6, 2024
5a57b61
Bump version: 3.0.1 → 3.0.2
akhileshh Jul 6, 2024
74a6e69
fix(edits): use supervoxels to get the correct cross edge parents
akhileshh Jul 7, 2024
751869b
Bump version: 3.0.2 → 3.0.3
akhileshh Jul 7, 2024
d63328a
fix(edits/split): filter out inactive cross edges
akhileshh Jul 16, 2024
f4b07a4
fix(edits/split): filter out inactive cross edges AT EACH LAYER
akhileshh Jul 17, 2024
aaef244
migration debug code
akhileshh Aug 30, 2024
50360da
use parent timestamps to lift cx edges
akhileshh Sep 22, 2024
0240011
make dynamic mesh dir graph specific
akhileshh Sep 23, 2024
1e0ab1b
fix(upgrade): use hierarchy from supervoxels
akhileshh Sep 26, 2024
205224d
fix(upgrade): include cx edges at node_ts explicitly
akhileshh Sep 26, 2024
9519b4c
adds job type guard, flush_redis prompts, improved status output
akhileshh Sep 29, 2024
528b60f
fix(upgrade): include timestamps for partner supervoxel parents
akhileshh Nov 10, 2024
8ec81a6
fix(upgrade): use timestamps of partners at layers > 2
akhileshh Nov 21, 2024
fcd152c
version 3.0.9
akhileshh Dec 5, 2024
7f13deb
feat: use mesh dir and dynamic dir from metadata
akhileshh Dec 9, 2024
3c963ee
ingest: change job batch size, more logging
akhileshh Jul 15, 2025
d5986db
Bump version: 3.0.10 → 3.0.11
akhileshh Jul 15, 2025
d2b80f8
ingest: add socket_timeout for redis connections
akhileshh Jul 15, 2025
56629f8
Bump version: 3.0.11 → 3.0.12
akhileshh Jul 15, 2025
14580c7
fix(edits): descriptive error message
akhileshh Jul 17, 2025
964f6ee
Bump version: 3.0.12 → 3.0.13
akhileshh Jul 17, 2025
b1f715f
fix(edits): find stale edges and their latest nodes
akhileshh Jul 29, 2025
10fc1ac
fix(edits): more precise filter for latest edges; error on chunk_id m…
akhileshh Aug 3, 2025
c1812aa
fix(edits): account for fake edges when finding latest edges
akhileshh Aug 5, 2025
5022ec6
fix(upgrade): use start and end timestamps to filter out irreleveant …
akhileshh Oct 21, 2025
8829d94
feat(upgrade): cache stale timestamp info; remove unnecessary checks …
akhileshh Oct 24, 2025
2f72d1a
fix(edits): include padded region to search for edges; not just L2 IDs
akhileshh Oct 27, 2025
0153652
Bump version: 3.1.0 → 3.1.1
akhileshh Oct 27, 2025
139d59d
fix(migration): filter out stale source nodes for latest edges
akhileshh Oct 28, 2025
393d536
fix(migration): handle cases where children IDs have already had thei…
akhileshh Oct 30, 2025
1f8b096
fix(migration, edits): replace stale edges with new edges before pers…
akhileshh Oct 31, 2025
e477077
Bump version: 3.1.1 → 3.1.2
akhileshh Oct 31, 2025
444d192
fix(migration): use None for non stale node end ts, performance impro…
akhileshh Nov 6, 2025
0cd87d4
perf(upgrade): reduce latency for atomic layer chunks
akhileshh Nov 6, 2025
c9ee0c3
fix(migration): erase corrupt ids from failed edits
akhileshh Nov 8, 2025
9bd35a4
fix(edits/migration): pass edge layer to helper function; handle stal…
akhileshh Nov 12, 2025
53d78e7
Bump version: 3.1.2 → 3.1.3
akhileshh Nov 12, 2025
0a7ed8b
fix(edits/migration): don't use lru parents cache with edits
akhileshh Nov 13, 2025
ef258a3
fix(migration): use larger cache, faster parent search with bisect, c…
akhileshh Nov 13, 2025
254ed86
Bump version: 3.1.3 → 3.1.4
akhileshh Nov 13, 2025
c7ea47a
fix(migration): add env for cache size
akhileshh Nov 13, 2025
49ab255
fix(edits): dont use children cache for edits
akhileshh Nov 13, 2025
120fd9c
Bump version: 3.1.4 → 3.1.5
akhileshh Nov 13, 2025
66c3293
feat(migration): split chunks tasks for faster completion
akhileshh Nov 15, 2025
b27fcc2
fix(migration): fallback to atomic edges when everything else fails, …
akhileshh Nov 16, 2025
a579a39
fix(upgrade_cli): check for completion when splits>0
akhileshh Jan 10, 2026
ebe4f21
fix(locks): improve locking latency
akhileshh Jan 19, 2026
ea2f675
feat(upgrade): cache earliest ts, live status refresh
akhileshh Jan 21, 2026
218de1e
Bump version: 3.1.5 → 3.1.6
akhileshh Jan 21, 2026
e69b0b2
fix(upgrade): avoid reading the entire chunk for split tasks
akhileshh Jan 22, 2026
e228cb1
fix(edits): search entire hierarchy upto stop to layer to find new edges
akhileshh Jan 25, 2026
c9bb1cf
fix(copy): add shim to replace graph_id in meta for copied graphs
akhileshh Jan 25, 2026
cb18972
fix(edits): stale edges - search hierarchies from both sides for pote…
akhileshh Jan 26, 2026
c130507
fix(edits): look for new edges by dilating lifted edges down to l2
akhileshh Jan 28, 2026
30e51ab
edits: batch parents dict and neighbor cx edges updates to reduce io …
akhileshh Jan 28, 2026
b634c1e
test: batch update cx edges to use the batched stale edge resolution
akhileshh Jan 29, 2026
f1dd4ff
fix(edits): search for children when edges are dilated
akhileshh Feb 4, 2026
43e69b8
fix(edits): handle simple case when stale edge is at l2; persist reso…
akhileshh Feb 6, 2026
6fa1c26
feat(edits): batch get descendents, make sanity checks optional
akhileshh Feb 8, 2026
3b01659
frontend: enable sanity checks for edits
akhileshh Feb 8, 2026
9f11702
Bump version: 3.1.6 → 3.1.7
akhileshh Feb 8, 2026
1e41f18
fix(edits): batch create new l2 ids for better performance
akhileshh Feb 11, 2026
a45a4d5
fix(edits): use a valid timestamp when locks are overridden, instead …
akhileshh Feb 11, 2026
6f38418
fix(edits): batch create higher layer ids, reuse future roots from lo…
akhileshh Feb 15, 2026
f68a4f0
fix: use 8 threads at most for locks, refactor stale edge code
akhileshh Feb 19, 2026
eaa24bc
tests: reorganize, new tests, fix codecov integration
akhileshh Feb 19, 2026
9bbc816
migrate to p3.12 and numpy 2, consolidate dockerfile
akhileshh Feb 19, 2026
930706b
try to fix codecov
akhileshh Feb 20, 2026
c3ec2a4
add more tests to improve coverage
akhileshh Feb 20, 2026
03a0422
feat: add stitching changes
akhileshh Feb 25, 2026
d163513
fix: initialize stitch mode attr
akhileshh Feb 25, 2026
df96222
add buildkit env for cloudbuild
akhileshh Feb 26, 2026
45dddf5
remove bigtable client, reorganize tests, improve redis+rq ops
akhileshh Feb 28, 2026
803acd6
upgrade kvdbclient
akhileshh Mar 4, 2026
e758d68
remove multiwrapper from ingest module
akhileshh Mar 18, 2026
90e8258
feat(sv_split): track max sv id to create new ids; convert ws seg to …
akhileshh Sep 10, 2025
a2a51f9
feat(sv_split): metadata changes to support ocdbt seg
akhileshh Sep 10, 2025
ea78867
feat(sv_split): split sv, update seg and edges, read and write new ed…
akhileshh Sep 10, 2025
14338c6
feat(sv_split): sv split in frontend
akhileshh Sep 10, 2025
fb50497
fix(sv_split): update multicut tests, add other tests
akhileshh Feb 26, 2026
17c5577
use kvdbclient, organize tests
akhileshh Mar 4, 2026
0c1cd1f
regenrate requirements
akhileshh Mar 4, 2026
deff2c0
fix: use registered attributes module
akhileshh Mar 5, 2026
328f98e
feat(ingest): make ocdbt seg optional
akhileshh Mar 5, 2026
9c0778a
add ocdbt flag to ingest cli
akhileshh Mar 18, 2026
18d3198
bugfix: use labels for cx edges after split with inf affinity
akhileshh Mar 18, 2026
5e82105
optimize whole sv lookup, use rep sv from source
akhileshh Mar 19, 2026
b659906
fix: use relative ocdbt path in info
akhileshh Mar 19, 2026
7b27599
fix: inf edges to only closest partner; mesh worker pubsub version, a…
akhileshh Mar 20, 2026
621a811
remove multiwrapper usage and references
akhileshh Mar 20, 2026
174156b
use a common logger, cleanup old mess
akhileshh Mar 20, 2026
bf1bd18
fix: mincut vertex index mismatch, source/sink overlap detection, sv_…
akhileshh Mar 21, 2026
f13f0b5
refactor: extract edge routing to edges_sv.py, root-based active/inac…
akhileshh Mar 21, 2026
919ca73
chore: remove trivial and redundant tests
akhileshh Mar 21, 2026
6d688f3
test the right error is raised for ocdbt vs precomputed seg
akhileshh Mar 21, 2026
48d5667
fix: _expand_partners set type crash, validation checks labels not fr…
akhileshh Mar 21, 2026
497cef0
add pcg logger with timing for sv split flow: whole sv, seg read, spl…
akhileshh Mar 21, 2026
6af123b
add sv_split debug utilities: subgraph edges, inf bridges, L2 childre…
akhileshh Mar 21, 2026
632ae4d
perf: lazy kdtree distances, single-process chunk updates, add timing…
akhileshh Mar 21, 2026
f54f87a
fix: app logger propagation blocking pcg logger in common.py
akhileshh Mar 21, 2026
7e778bf
test: all edge routing tests use multi-SV old_new_map to match produc…
akhileshh Mar 21, 2026
5bfae6d
fix: use sv_remapping for whole SV lookup, filter stale split edges v…
akhileshh Mar 22, 2026
7fbe2ce
refactor: extract split_with_sv_splits, return user message when retr…
akhileshh Mar 22, 2026
3a938b2
add split lineage inspection, unsplit SV bridge detection
akhileshh Mar 22, 2026
8ba60ff
add compacted split edges for O(1) current-time reads, suppress glog …
akhileshh Mar 22, 2026
b65e754
ingest/upgrade: add ocdbt option to upgrade cli
akhileshh Mar 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 2.21.1
current_version = 3.1.7
commit = True
tag = True

Expand Down
7 changes: 7 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,13 @@ source = pychunkedgraph
omit =
*test*
*benchmarking/*
pychunkedgraph/debug/*
pychunkedgraph/export/*
pychunkedgraph/jobs/*
pychunkedgraph/logging/*
pychunkedgraph/repair/*
pychunkedgraph/meshing/*
pychunkedgraph/app/*

[report]
# Regexes for lines to exclude from consideration
Expand Down
38 changes: 34 additions & 4 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,49 @@ on:
push:
branches:
- "main"
- "pcgv3"
pull_request:
branches:
- "main"
- "pcgv3"

jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Build image and run tests
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Build image
uses: docker/build-push-action@v6
with:
context: .
load: true
tags: seunglab/pychunkedgraph:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Run tests with coverage
run: |
docker build --tag seunglab/pychunkedgraph:$GITHUB_SHA .
docker run --rm seunglab/pychunkedgraph:$GITHUB_SHA /bin/sh -c "pytest --cov-config .coveragerc --cov=pychunkedgraph ./pychunkedgraph/tests && codecov"
docker run --name pcg-tests seunglab/pychunkedgraph:${{ github.sha }} \
/bin/sh -c "pytest --cov-config .coveragerc --cov=pychunkedgraph --cov-report=xml:/app/coverage.xml ./pychunkedgraph/tests"

- name: Copy coverage report from container
if: always()
run: docker cp pcg-tests:/app/coverage.xml ./coverage.xml

- name: Upload coverage to Codecov
if: always()
uses: codecov/codecov-action@v5
with:
files: ./coverage.xml
token: ${{ secrets.CODECOV_TOKEN }}
slug: CAVEconnectome/PyChunkedGraph
fail_ci_if_error: true

- name: Cleanup
if: always()
run: docker rm pcg-tests || true
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
- name: Install Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.12"
- name: Install bumpversion
run: pip install bumpversion
- name: Bump version with bumpversion
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ venv.bak/


# local dev stuff
.claude/
.devcontainer/
*.ipynb
*.rdb
Expand Down
5 changes: 5 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
repos:
- repo: https://github.com/psf/black
rev: 26.1.0
hooks:
- id: black
60 changes: 0 additions & 60 deletions .travis.yml

This file was deleted.

59 changes: 57 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,11 +1,66 @@
FROM caveconnectome/pychunkedgraph:base_042124
# syntax=docker/dockerfile:1
ARG PYTHON_VERSION=3.12
ARG BASE_IMAGE=tiangolo/uwsgi-nginx-flask:python${PYTHON_VERSION}


######################################################
# Stage 1: Conda environment
######################################################
FROM ${BASE_IMAGE} AS conda-deps
ENV PATH="/root/miniconda3/bin:${PATH}"

RUN apt-get update && apt-get install build-essential wget -y \
&& wget -q https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm Miniconda3-latest-Linux-x86_64.sh \
&& conda config --add channels conda-forge \
&& conda update -y --override-channels -c conda-forge conda \
&& conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main \
&& conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r \
&& conda install -y --override-channels -c conda-forge conda-pack

COPY requirements.yml requirements.txt requirements-dev.txt ./

RUN --mount=type=cache,target=/opt/conda/pkgs \
conda env create -n pcg -f requirements.yml

RUN conda-pack -n pcg --ignore-missing-files -o /tmp/env.tar \
&& mkdir -p /app/venv && cd /app/venv \
&& tar xf /tmp/env.tar && rm /tmp/env.tar \
&& /app/venv/bin/conda-unpack


######################################################
# Stage 2: Bigtable emulator
######################################################
FROM golang:bullseye AS bigtable-emulator
ARG GOOGLE_CLOUD_GO_VERSION=bigtable/v1.19.0
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
git clone --depth=1 --branch="$GOOGLE_CLOUD_GO_VERSION" \
https://github.com/googleapis/google-cloud-go.git /usr/src \
&& cd /usr/src/bigtable && go install -v ./cmd/emulator


######################################################
# Stage 3: Production
######################################################
FROM ${BASE_IMAGE}
ENV VIRTUAL_ENV=/app/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

COPY --from=conda-deps /app/venv /app/venv
COPY --from=bigtable-emulator /go/bin/emulator /app/venv/bin/cbtemulator
COPY override/gcloud /app/venv/bin/gcloud
COPY override/timeout.conf /etc/nginx/conf.d/timeout.conf
COPY override/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
RUN pip install --no-cache-dir --no-deps --force-reinstall zstandard>=0.23.0 \
&& mkdir -p /home/nginx/.cloudvolume/secrets \
&& chown -R nginx /home/nginx \
&& usermod -d /home/nginx -s /bin/bash nginx

COPY requirements.txt .
RUN pip install --upgrade -r requirements.txt
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade -r requirements.txt

COPY . /app
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# PyChunkedGraph


[![Build Status](https://travis-ci.org/seung-lab/PyChunkedGraph.svg?branch=master)](https://travis-ci.org/seung-lab/PyChunkedGraph)
[![codecov](https://codecov.io/gh/seung-lab/PyChunkedGraph/branch/master/graph/badge.svg)](https://codecov.io/gh/seung-lab/PyChunkedGraph)
[![Tests](https://github.com/CAVEconnectome/PyChunkedGraph/actions/workflows/main.yml/badge.svg)](https://github.com/CAVEconnectome/PyChunkedGraph/actions/workflows/main.yml)
[![codecov](https://codecov.io/gh/CAVEconnectome/PyChunkedGraph/branch/main/graph/badge.svg)](https://codecov.io/gh/CAVEconnectome/PyChunkedGraph)

The PyChunkedGraph is a proofreading and segmentation data management backend powering FlyWire and other proofreading platforms. It builds on an initial agglomeration of supervoxels and facilitates fast and parallel editing of connected components in the agglomeration graph by many users.

Expand Down
70 changes: 0 additions & 70 deletions base.Dockerfile

This file was deleted.

4 changes: 0 additions & 4 deletions build_pypi.sh

This file was deleted.

2 changes: 1 addition & 1 deletion cloudbuild.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ steps:
args:
- "-c"
- |
docker build -t $$USERNAME/pychunkedgraph:$TAG_NAME .
DOCKER_BUILDKIT=1 docker build -t $$USERNAME/pychunkedgraph:$TAG_NAME .
timeout: 600s
secretEnv: ["USERNAME"]

Expand Down
19 changes: 19 additions & 0 deletions codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
codecov:
require_ci_to_pass: true

coverage:
status:
project:
default:
target: auto
threshold: 1%
informational: true
patch:
default:
target: 1%
informational: true

comment:
layout: "reach,diff,flags,files"
behavior: default
require_changes: false
1 change: 0 additions & 1 deletion compile_reqs.sh

This file was deleted.

4 changes: 2 additions & 2 deletions docs/Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ pip install -r requirements.txt

## Multiprocessing

Check out [multiprocessing.md](https://github.com/seung-lab/PyChunkedGraph/blob/master/src/pychunkedgraph/multiprocessing.md) for how to use the multiprocessing tools implemented for the ChunkedGraph
Check out [multiprocessing.md](https://github.com/CAVEconnectome/PyChunkedGraph/blob/master/src/pychunkedgraph/multiprocessing.md) for how to use the multiprocessing tools implemented for the ChunkedGraph


## Credentials
Expand All @@ -30,7 +30,7 @@ The current version of the ChunkedGraph contains supervoxels from `gs://nkem/bas

### Building the graph

[buildgraph.md](https://github.com/seung-lab/PyChunkedGraph/blob/master/src/pychunkedgraph/buildgraph.md) explains how to build a graph from scratch.
[buildgraph.md](https://github.com/CAVEconnectome/PyChunkedGraph/blob/master/src/pychunkedgraph/buildgraph.md) explains how to build a graph from scratch.


### Initialization
Expand Down
2 changes: 1 addition & 1 deletion docs/edges.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

PyChunkedgraph uses protobuf for serialization and zstandard for compression.

Edges and connected components per chunk are stored using the protobuf definitions in [`pychunkedgraph.io.protobuf`](https://github.com/seung-lab/PyChunkedGraph/pychunkedgraph/io/protobuf/chunkEdges.proto).
Edges and connected components per chunk are stored using the protobuf definitions in [`pychunkedgraph.io.protobuf`](https://github.com/CAVEconnectome/PyChunkedGraph/pychunkedgraph/io/protobuf/chunkEdges.proto).
This format is a result of performance tests.
It provided the best tradeoff between deserialzation speed and storage size.

Expand Down
2 changes: 1 addition & 1 deletion docs/edges_and_components.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

PyChunkedgraph uses protobuf for serialization and zstandard for compression.

Edges and connected components per chunk are stored using the protobuf definitions in [`pychunkedgraph.io.protobuf`](https://github.com/seung-lab/PyChunkedGraph/pychunkedgraph/io/protobuf/chunkEdges.proto).
Edges and connected components per chunk are stored using the protobuf definitions in [`pychunkedgraph.io.protobuf`](https://github.com/CAVEconnectome/PyChunkedGraph/pychunkedgraph/io/protobuf/chunkEdges.proto).
This format is a result of performance tests.
It provided the best tradeoff between deserialzation speed and storage size.

Expand Down
4 changes: 2 additions & 2 deletions docs/segmentation_preprocessing.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,10 @@ There are three types of edges:
2. `cross_chunk`: edges between parts of "the same" supervoxel in the unchunked segmentation that has been split across chunk boundary
3. `between_chunk`: edges between supervoxels across chunks

Every pair of touching supervoxels has an edge between them. All edges are stored using [protobuf](https://github.com/seung-lab/PyChunkedGraph/blob/pcgv2/pychunkedgraph/io/protobuf/chunkEdges.proto). During ingest only edges of type 2. and 3. are copied into BigTable, whereas edges of type 1. are always read from storage to reduce cost. Similar to the supervoxel segmentation, we recommed storing these on GCloud in the same zone the ChunkedGraph server will be deployed in to reduce latency.
Every pair of touching supervoxels has an edge between them. All edges are stored using [protobuf](https://github.com/CAVEconnectome/PyChunkedGraph/blob/pcgv2/pychunkedgraph/io/protobuf/chunkEdges.proto). During ingest only edges of type 2. and 3. are copied into BigTable, whereas edges of type 1. are always read from storage to reduce cost. Similar to the supervoxel segmentation, we recommed storing these on GCloud in the same zone the ChunkedGraph server will be deployed in to reduce latency.

To denote which edges form a connected component within a chunk, a component mapping needs to be created. This mapping is only used during ingest.

More details on how to create these protobuf files can be found [here](https://github.com/seung-lab/PyChunkedGraph/blob/pcgv2/docs/storage.md).
More details on how to create these protobuf files can be found [here](https://github.com/CAVEconnectome/PyChunkedGraph/blob/pcgv2/docs/storage.md).


Loading
Loading