0.7.1-beta

This minor release is primarily to test the updated CI and zenodo webhook.

CI

• Check if the version number in pyproject.toml is strictly newer than the latest release on PRs with master, and matches CITATION.cff
• Test zenodo webhook
• Cache barrel cortex model across OS's
• Cache NEURON installation on Windows

Fixes

• Extend swc support to udvary2022
• Improve docs and tutorials.

Other

• Add CITATION.cff

v0.7.0-beta

New features:

• swc support
• Network Parameter Builder

Other improvements:

• improved generalizability in network mapper and l5pt-specific functionality
• improved tutorials and docs

swc support

We now support swc. ISF pipelines are agnostic to whether a morphology file is hoc or swc. Reading in and building the morphology is simply handled by single_cell_parser.io.morphology.read_morphology. Big thanks to @su-saka and @fkkastrinakis for writing the initial converters, that save me a bunch of time for further integration.

We have converters from swc to hoc and hoc to swc. Converting hoc to swc can slightly change connectivity of some sections to the soma, but this is of no impact in ISF as we connect them by default at
x=0.5.

Simulations on both yield almost identical results. I implemented a test for this. variable time step simulations are not fully identical, fixed timesteps simulations are. I am not sure yet why this is the case. Variable time step simulations for hoc vs swc are virtually indistinguishable, but the time steps are ever so slightly different.

#482 and #476

Network mapping generalizibility

I changed the network mapping procedure so it does not depend anymore on any particular label (e.g. "apical"). Instead, labels are inferred from the morphology file. Additional empirical data should then match these labels (e.g. connections spreadsheet) case-insensitively #478 #467

singlecell_input_mapper has been refactored extensively. It now uses a builder pattern to add network parameters. this makes it way easier to modularize and understand which data is being added, and how (see #465). In addition, the default network embedding strategy can now be configured on a user-level (see #466 )

I extended the network parameters with a NetworkParamBuilder class that makes it way more convenient to build network parameters, and makes it explicit which data you add, when, and where. This is explored
extensively in the tutorials. #465

Network mapping does not rely anymore on a NrOfConnections.csv. This used to be default output of our network mapper. However, the same information is contained in a .con file. We now parse it from a .con file. This way, ISF is fully independent of any particular network mapping procedure, and simply relies on a syn and con file.

I augmented our dataset for activity data. The correct activity data file + key used to be inferred at runtime, i.e. depending on your recording site and stimulus, it would redirect you to the correct activity data file on the fly. This was very barrel cortex specific. I augmented the activity data by simply generating datasets for all stimuli. The augmentation method is saved as a notebook in getting_started/functional_constraints/evoked_activity/empirical_recordings. Now, the key you pass under "whisker" is instead interpreted as a globstring that fetches the correct data file in a preconfigured data directory. The NetworkParamBuilder of course also allows you to directly add activity data, independent of any directory configuration. #465

A tutorial now explores all the data requirements for Udvary 2022 #480

The network mapper used in ISF is Udvary 2022, but it should be clear that ISF does not rely on this particular network mapping. I refactored the top-level pipeline so it infers a specific network mapping procedure, and refactored Udvary 2022 as a separate subpackage in singlecell_input_mapper. #466

L5PT mentions

I removed all mentions of "L5PT" in the code, except for those methods that are actually specific to L5PTs. In the latter case, it should be clear that this methods provides an example, and should not live in top-level pipelines or modules #469

Tutorials

Updated the intro tutorial so be more targeted at exactly what ISF does. Removed configuration setup. Added more headings, intro and explanation on what we do, how, and why. #484

Updated network mapping tutorial to add API for the SynapseMapper and visualizations. Added auxiliary tutorials. They are hidden from direct view in the webpage, but reflinks work. This is e.g. used in tutorial 2.1 #485

Config

Added config for morphology labels. You can extend a map so labels identified in morphology files are renamed to something else. This is useful for having a consistent internal representation. I had to do this to rename "BasalDendrite" (in network mapping methods) to "Dendrite"

Added mechanisms to user config, since the mechanisms are user-specific,
technically.

Docs

Added documentation on swc and mod files. Wrote a custom lexer for
NMODL :)
Extended installation instructions for more info on how ISF handles environments and env variables.
Extended documentation on file formats, added data sources to paramfile overview
Extended tutorials

v0.6.0-beta

Features

• Added windows support (experimental) (#355)
• Refactored and improved RW class (#457)
• Databases are now fully self-contained (#425, #420)
• Added plot_hoc to plot out a .hoc file directly
• .param files are now read and written using the json package. Parameterfiles that are written are from now on fully JSON-compliant. Older .param files are sanitized and then read using json
• Added swc exporter for cells (caa795c)
• Added a check_health keyword arg in db_init_simrun_general in case there is some missing data (#420)

Performance

• Vectorized singlecell_input_mapper.map_singlecell_inputs (#408)
• Improved reader for axon traces in singlecell_input_mapper
• Full pipeline now takes about ~10 minutes for reading and generating 1 embedding, rather than ~45 minutes.

Various

• Remove barrel_cortex as a package. It is now fully optional, and can be downloaded with pixi r download_bc_model
• Moved dendrite_thickness to ibs_projects
• Drop deprecated sumatra dependency in favor of built-in NTParameterSet

Docs

• Reflinking to parameters now works in documentation (if the FQN checks out)
• Improved documentation everywhere

Bug fixes

• Fix: initializing other people's register. Logical bug in assuming the database register is always your own (it's not)(7bc4baf)
• Fix: allow fetching ModelDataBases by unique ID (b5d95d2)
• Fix faulty sim_trial_index for dendritic voltage traces (81723e1)
• Fix: Only filter paramfiles during simrun_init_db when hash_rename is True (afc4968)
• Fix: resolve relative .syn and .con files (81723e1)

v0.5.0-beta

New features

• Hay (2011) has been fully translated to Python instead of NEURON (see #301)
• Databases are now self-contained (see #302)
• database "backend switching" (between ISFDataBase and ModelDataBase) is more convenient
• db_initializers are database backend agnostic (see #326)
• databases use msgpack again as an optimized format. See https://pypi.org/project/isf-pandas-msgpack/ for the un-deprecated version of pandas_msgpack (see #320)
• File locking configuration is evaluated lazily (see PR #297)

Other improvements

• CDK data has been published on https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/21W1HR, backed up in ibs_projects and removed from ISF
• Example parameter files are renamed and restructured for better readability
• Tutorials on activity data have a better explanation on the pipeline from empirical activity data to network activation simulations.
• Various documentation improvements (installation instructions, video, scrollable notebook cell output)
• refactored db_initializers into separate modules for easier maintenance

API changes

hay has been refactored since the Python translation. For backwards compatibility in the IBS group, a compatibility patch is available on the latest version of ibs_projects. You can apply compatibility patches by:

from ibs_projects import compatibility

v0.4.0-beta

Prerelease 0.4.0-beta

Most important features to look out for in this release: MacOS support, 100% documentation coverage

Warning

MOEA_EH will be removed in a future version in favor of the Python translation of Hay
Databases will be fully self-contained in a future version, rather than simlinking to parameter files.

Changelog

Installer:

• use pixi installation by default
• add MacOS installation. Note that this usess a different version of numpy and NEURON, and introduces some roundoff errors compared to the linux version.
• barrel cortex model can now be downloaded from harvard dataverse, and doesnt need to be hosted on github anymore (058ea1b)
• download anaconda with wget without checking for a certificate (it expired) (058ea1b)
• No more semicolons after commands in installer, otherwise errors are hard to catch (058ea1b)
• add cmasher

biophysics

• evaluation can now be done with a full python translation of Hay 2011, including the step currents (d994f73)
• hay specification values (empirical mean and std) are moved to a json config file

Documentation:

• 100% Documentation coverage

Improve visualize

• CMV with joinstyle argument to avoid jagged lines
• CMV: Allow overwriting frames to re-create animations
• CMV: allow visualizing non-simulated cell objects and export to vtk
• VTK: Added converter for Amira lattice files (structured grid) to VTK, allowing to visualize PST density files.

data_base

• use pandas_to_parquet dumper for the parameterfiles key in init_simrun_general (only for isf_data_base, not for model_data_base) (9edb139)
• move dbopen to the parent package data_base to remove dependency on specific data base version
• fix assimilate_remote_register after data_base rename

Bugfixes and improvements

• moved cluster configuration to cluster_control repo instead of ISF
• Synapse IDs are maintained in rerun_mdb (52a0c21)
• Update compatibility: now does not import modules, but uses modulespec to register them
• no more internal Interface imports
• rename sim_trail to sim_trial
• lazy evaluate the locking config (53606b4) to show logger output only when actually needed

v0.3.0-beta

This release adds a new way of installing ISF: through pixi.
Pixi is a much more convenient way (in my opinion) to manage and solve Python environments/projects. It's absurdly fast, handles the installation of ourpandas-msgpack with ease, provides cross-platform support (todo), and actually solves the environment for PyPI and conda repos at the same time.

This release also significantly increases documentation coverage. 83% of all members are documented. 100% documentation coverage is coming soon TM.

ISF v0.2.0-alpha

Data base is py2 compatible in test suite

ISF v0.1.0-alpha

About versions

Major changes have been introduced since the last version on master. Version 1.x.x is reserved for the first publication-ready version of ISF. Until then, changes will have a 0.x.x tag, even when the changes may be major.

The syntax as it appears now in master is intended to be the publication version. Minor changes and patches may still apply, changing the version number to 0.1.x+1. the alpha keyword indicates that the current version of master is still untested by fellow developers.

Changelog

project_specific_ipynb_code

• has been moved to a separate repo

cell_morphology_visualizer

• Updated syntax for plotting functions. There is now plot, write_gif, write_video, animation and write_vtk_frames.
• Depending on what you want as a colorscale, and if you want to show synapses, you can pass these in as function arguments
• t_end has been renamed to t_stop to be more compatible with NEURON ('tStop')

data_base

• model_data_base has become a subpackage of the wrapper class data_base. Only when explicitly opening an old model_data_base will it use model_data_base, otherwise, it will use the new isf_data_base with json format for dumpers and metadata.
• To mitigate pickling issues, Interface registers the subpackage data_base.model_data_base under an additional anme model_data_base to redirect pickle to the correct package location.

simrun

• simrun2 and simrun3 have been merged into a single simrun package. Same pickling trick as before was applied.

mechanisms

• the mechanisms for our beloved L5PT have been bundled in a subfolder in mechanisms, to highlight that these mechanisms are L5PT specific. While these channels may very well appear in other cells too, this seemed to be the most convenient way of categorizing them.

Bug fixes

• Completed #191
• Completed #189
• Completed #162. This can always be achieved by cmv.morphology['diameter'] = ...
• Completed #145 for now. The workflows are outlined in the tutorials (see getting_started/tutorials)

Improvements

• Added tutorials (WIP: ANN and reduced modeling)
• Added documentation (WIP)

TODO

• Finalize tutorials
• Finalize documentation
• Test functionality given the recent API changes