From d83a2e538ca40eae8c8d83769089d9dd22b4217a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Semid=C3=A1n=20Robaina=20Est=C3=A9vez?= Date: Thu, 18 Jun 2026 17:56:55 +0100 Subject: [PATCH 1/4] Drop conda dependency: replace hmmer/prodigal/seqkit with pyhmmer/pyrodigal/Python Pynteny shelled out to three conda-only binaries (hmmsearch/hmmscan, prodigal and seqkit). This replaces all of them with pure-Python, pip-installable equivalents so Pynteny can be installed via pip without conda. - wrappers.run_HMM_search: run searches with pyhmmer and write a standard HMMER3 tabular (--tblout) file, so the existing Bio.SearchIO parser and result reuse are unchanged. Adds a translator for common hmmsearch_args (--cut_ga/_nc/_tc, -E/-T/--domE/-Z, ...). - wrappers.run_seqkit_nodup: reimplement seqkit rmdup -s with pyfastx. - GeneAnnotator.annotate: predict genes in-process with pyrodigal (threaded; GIL released during find_genes), writing prodigal-format FASTA so from_prodigal_output is unchanged. Removes run_prodigal and the split-by-contigs + multiprocessing dance. - FASTA.filter_by_IDs: replace seqkit grep with a case-insensitive pyfastx ID filter. - FASTAmerger.merge: replace the `printf | xargs cat` shell-out with Python file concatenation. Packaging: declare runtime deps in pyproject.toml (previously only in the conda recipe); drop hmmer/prodigal/seqkit from both conda recipes and the dev env (pyhmmer/pyrodigal via pip); convert the Dockerfile to a pure-pip build; update README install and dependency sections. Also fixes a pre-existing bug in FASTA.add_prefix_to_records: it opened output_file with "w+" while pyfastx read the same path lazily, truncating the source whenever output_file == self.file_path (so `pynteny build --prefix` produced an empty database). It now writes to a temp file and moves it into place. test_build_fasta is strengthened to assert non-empty, prefixed output. Closes #97 Co-Authored-By: Claude Opus 4.8 --- .bioconda/meta.yaml | 5 +- .conda/meta.yaml | 5 +- Dockerfile | 25 ++--- README.md | 39 ++++---- envs/pynteny-dev.yml | 5 +- pyproject.toml | 9 ++ src/pynteny/preprocessing.py | 106 +++++++++++++------- src/pynteny/wrappers.py | 185 ++++++++++++++++++++++------------- tests/test_database_build.py | 18 ++++ 9 files changed, 248 insertions(+), 149 deletions(-) diff --git a/.bioconda/meta.yaml b/.bioconda/meta.yaml index 2ac9913..08bb1f7 100644 --- a/.bioconda/meta.yaml +++ b/.bioconda/meta.yaml @@ -24,14 +24,13 @@ requirements: - poetry - pip run: - - hmmer - - prodigal + - pyhmmer >=0.10 + - pyrodigal >=3.0 - pyfastx >=0.8 - biopython - numpy - pandas - psutil - - seqkit - requests - tqdm diff --git a/.conda/meta.yaml b/.conda/meta.yaml index d4abe9f..b682176 100644 --- a/.conda/meta.yaml +++ b/.conda/meta.yaml @@ -22,13 +22,12 @@ requirements: - poetry - pip run: - - hmmer >=3.3 - - prodigal >=2.6 + - pyhmmer >=0.10 + - pyrodigal >=3.0 - pyfastx >=0.8 - biopython >=1.8 - numpy - pandas - - seqkit - psutil >=5.9 - requests - tqdm diff --git a/Dockerfile b/Dockerfile index 920b415..37746c0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,27 +1,16 @@ -# FROM mcr.microsoft.com/devcontainers/miniconda:0-3 -FROM mambaorg/micromamba:latest -USER root +FROM python:3.11-slim WORKDIR /Pynteny # Copy repo to docker container COPY src/pynteny src/pynteny/ COPY tests tests/ -COPY envs envs/ COPY README.md . COPY pyproject.toml . COPY LICENSE . -# Make conda environment and activate -# RUN conda install mamba -n base -c conda-forge -RUN micromamba env create -f envs/pynteny-dev.yml -SHELL ["micromamba", "run", "-n", "pynteny-dev", "/bin/bash", "-c"] -# Build and install Pynteny -RUN poetry build && pip install dist/pynteny*.whl && pynteny --help -# Give read/write permissions to install directory (needed to make config.json) -RUN chmod ugo+rw /opt/conda/envs/pynteny-dev/lib/python3.10/site-packages - -# Initialize conda for default user -RUN micromamba shell init --shell=bash -# # Activate pynteny environment by default -RUN echo "micromamba activate pynteny-dev" >> ../root/.bashrc -RUN source ../root/.bashrc \ No newline at end of file +# Build and install Pynteny (pure pip, no conda required). +# pyhmmer and pyrodigal ship manylinux wheels, so no compiler toolchain is needed. +RUN pip install --no-cache-dir poetry \ + && poetry build \ + && pip install --no-cache-dir dist/pynteny*.whl \ + && pynteny --help diff --git a/README.md b/README.md index 00506f0..8752ebe 100644 --- a/README.md +++ b/README.md @@ -36,35 +36,37 @@ Check out the [Pynteny paper](https://doi.org/10.21105/joss.05289) in the Journa ## 2. :wrench: Setup -Install with conda: +Pynteny is a pure-Python package (it no longer requires conda or any external +binaries). All dependencies, including the HMMER and Prodigal engines, are +provided by the pip packages [pyhmmer](https://github.com/althonos/pyhmmer) and +[pyrodigal](https://github.com/althonos/pyrodigal). -1. Pynteny requires Python 3.10. The easiest way to handle dependencies is by creating a dedicated conda environment: +Install with pip: ```bash -conda create -n pynteny -c bioconda -c conda-forge python=3.10 pynteny -conda activate pynteny +pip install pynteny ``` -2. Check that installation worked fine: +Or with conda (from bioconda): ```bash -(pynteny) pynteny --help +conda create -n pynteny -c bioconda -c conda-forge python=3.10 pynteny +conda activate pynteny ``` -### 2.1. Installing on Windows -Pynteny is designed to run on Linux machines. However, it can be installed within the [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install) via conda. - -### 2.2. Installing on MacOS with the latest ARM64 architecture - -Pynteny doesn't currently support the latest ARM64 architecture of silicon processors (e.g. MacBook M1 and M2). If that is your case, you can install Pynteny using the workaround below (based on [this post](https://towardsdatascience.com/how-to-manage-conda-environments-on-an-apple-silicon-m1-mac-1e29cb3bad12)): +Check that the installation worked fine: ```bash -CONDA_SUBDIR=osx-64 conda create -n pynteny_x86 python=3.10 -conda activate pynteny_x86 -conda config --env --set subdir osx-64 -conda install -c bioconda pynteny +pynteny --help ``` +### 2.1. Installing on Windows + +Pynteny is developed and tested on Linux, but since it is now a pure-Python +package, `pip install pynteny` also works on Windows and macOS (including Apple +Silicon / ARM64), provided wheels are available for pyhmmer and pyrodigal on +your platform. + ## 3. :rocket: Usage Consider the following toy example of a syntenic block: @@ -126,11 +128,10 @@ You can find more notebooks in the [examples directory](docs/examples/). Find mo ## 5. :arrows_counterclockwise: Dependencies Pynteny would not work without these awesome projects: -- [hmmer](https://github.com/EddyRivasLab/hmmer) -- [prodigal](https://github.com/hyattpd/Prodigal) +- [pyhmmer](https://github.com/althonos/pyhmmer) +- [pyrodigal](https://github.com/althonos/pyrodigal) - [pyfastx](https://github.com/lmdu/pyfastx) - [biopython](https://github.com/biopython/biopython) -- [seqkit](https://bioinf.shenwei.me/seqkit/) - [numpy](https://github.com/numpy/numpy) - [pandas](https://github.com/pandas-dev/pandas) - [psutil](https://github.com/giampaolo/psutil) diff --git a/envs/pynteny-dev.yml b/envs/pynteny-dev.yml index faf9f52..b8dd3a6 100644 --- a/envs/pynteny-dev.yml +++ b/envs/pynteny-dev.yml @@ -7,17 +7,16 @@ dependencies: - python >= 3.8 - poetry >= 1.3 - pip - - hmmer >= 3.3 - - prodigal >= 2.6 - pyfastx >=0.8 - biopython >= 1.8 - numpy - pandas - - seqkit - psutil >= 5.9 - requests - tqdm - pip: + - pyhmmer >=0.10 + - pyrodigal >=3.0 - mkdocs - mkdocs-gen-files - pymdown-extensions diff --git a/pyproject.toml b/pyproject.toml index 3cffca9..e225886 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -24,6 +24,15 @@ packages = [{ include = "pynteny", from = "src" }] include = ["src/pynteny/config.json"] [tool.poetry.dependencies] python = "^3.8" +pyhmmer = ">=0.10" +pyrodigal = ">=3.0" +pyfastx = ">=0.8" +biopython = ">=1.8" +numpy = "*" +pandas = "*" +psutil = ">=5.9" +requests = "*" +tqdm = "*" [tool.poetry.scripts] pynteny = "pynteny.cli:main" diff --git a/src/pynteny/preprocessing.py b/src/pynteny/preprocessing.py index 3b4f5b0..ac9fe9e 100644 --- a/src/pynteny/preprocessing.py +++ b/src/pynteny/preprocessing.py @@ -13,15 +13,17 @@ import logging import os +import shutil import sys import tempfile +from concurrent.futures import ThreadPoolExecutor from pathlib import Path from typing import TextIO import pyfastx +import pyrodigal from Bio import SeqFeature, SeqIO, SeqRecord -import pynteny.utils as utils import pynteny.wrappers as wrappers logger = logging.getLogger(__name__) @@ -124,13 +126,21 @@ def merge(self, output_file: Path = None, prepend_file_name: bool = False) -> No else: output_file = Path(output_file) logger.info("Merging FASTA files in input directory") - cmd_str = f"printf '%s\\0' * | xargs -0 cat > {output_file}" + + def _concatenate(source_dir: Path) -> None: + source_dir = Path(source_dir) + with open(output_file, "wb") as outfile: + for file in sorted(source_dir.iterdir()): + if file.is_file(): + with open(file, "rb") as infile: + shutil.copyfileobj(infile, outfile) + if prepend_file_name: with tempfile.TemporaryDirectory() as tempdir: self.prepend_filename_to_record_names(output_dir=tempdir) - utils.terminal_execute(cmd_str, work_dir=tempdir) + _concatenate(tempdir) else: - utils.terminal_execute(cmd_str, work_dir=self.input_dir) + _concatenate(self.input_dir) logging.shutdown() @@ -270,14 +280,15 @@ def filter_by_IDs( ) else: output_file = Path(output_file) - with tempfile.NamedTemporaryFile(mode="w+t") as tmp_ids: - tmp_ids.writelines("\n".join(record_ids)) - tmp_ids.flush() - tmp_ids_path = tmp_ids.name - cmd_str = ( - f"seqkit grep -i -f {tmp_ids_path} {self._input_file} -o {output_file}" - ) - utils.terminal_execute(cmd_str, suppress_shell_output=True) + wanted_ids = {record_id.lower() for record_id in record_ids} + fasta = pyfastx.Fasta( + self.file_path.as_posix(), build_index=False, full_name=True + ) + with open(output_file, "w+", encoding="UTF-8") as outfile: + for record_name, record_seq in fasta: + record_id = record_name.split()[0] + if record_id.lower() in wanted_ids: + outfile.write(f">{record_name}\n{record_seq}\n") if point_to_new_file: self.file_path = output_file @@ -353,9 +364,20 @@ def add_prefix_to_records( self.file_path.as_posix(), build_index=False, full_name=True ) prefix = prefix.strip("_") - with open(output_file, "w+", encoding="UTF-8") as outfile: + # Write to a temporary file first and then move it into place. pyfastx + # reads the input lazily, so writing directly to output_file would + # truncate the source whenever output_file == self.file_path. + with tempfile.NamedTemporaryFile( + mode="w+t", + encoding="UTF-8", + dir=output_file.parent, + suffix=output_file.suffix, + delete=False, + ) as outfile: + tmp_path = Path(outfile.name) for record_name, record_seq in fasta: outfile.write(f">{prefix}_{record_name}\n{record_seq}\n") + shutil.move(tmp_path.as_posix(), output_file.as_posix()) if point_to_new_file: self.file_path = output_file @@ -549,32 +571,48 @@ def annotate( ) else: output_file = Path(output_file) + if prodigal_args: + logger.warning( + "prodigal_args are ignored when predicting genes with pyrodigal: " + f"'{prodigal_args}'" + ) if tempdir is None: tempdir = Path(tempfile.gettempdir()) else: tempdir = Path(tempdir).resolve() - with tempfile.TemporaryDirectory( - dir=tempdir - ) as contigs_dir, tempfile.TemporaryDirectory( - dir=tempdir - ) as prodigal_dir, tempfile.NamedTemporaryFile() as temp_fasta: - contigs_dir = Path(contigs_dir) - prodigal_dir = Path(prodigal_dir) - logger.info("Running prodigal on assembly data") - self._assembly.split_by_contigs(contigs_dir) - utils.parallelize_over_input_files( - wrappers.run_prodigal, - input_list=list(contigs_dir.iterdir()), - processes=processes, - output_dir=prodigal_dir, - output_format="fasta", - metagenome=metagenome, - additional_args=prodigal_args, - ) + + logger.info("Predicting genes with pyrodigal") + assembly_path = self._assembly.file_path + gene_finder = pyrodigal.GeneFinder(meta=metagenome) + if not metagenome: + training_seqs = [ + seq.encode() + for _, seq in pyfastx.Fasta( + assembly_path.as_posix(), build_index=False, full_name=True + ) + ] + gene_finder.train(*training_seqs) + + contigs = pyfastx.Fasta( + assembly_path.as_posix(), build_index=False, full_name=True + ) + + # pyrodigal releases the GIL during find_genes, so we can predict genes + # for several contigs concurrently while keeping input order for the + # output (and thus reproducible gene labels). + def _find(record): + record_name, record_seq = record + seq_id = record_name.split()[0] + return seq_id, gene_finder.find_genes(record_seq.encode()) + + with tempfile.NamedTemporaryFile( + mode="w+t", dir=tempdir, suffix=".faa" + ) as temp_fasta: + with ThreadPoolExecutor(max_workers=max(processes, 1)) as executor: + for seq_id, genes in executor.map(_find, contigs): + genes.write_translations(temp_fasta, sequence_id=seq_id) + temp_fasta.flush() logging.shutdown() - FASTAmerger(prodigal_dir).merge( - Path(temp_fasta.name), prepend_file_name=False - ) return LabelledFASTA.from_prodigal_output( Path(temp_fasta.name), output_file ) diff --git a/src/pynteny/wrappers.py b/src/pynteny/wrappers.py index f55dbae..757620a 100644 --- a/src/pynteny/wrappers.py +++ b/src/pynteny/wrappers.py @@ -2,91 +2,120 @@ # -*- coding: utf-8 -*- """ -Simple CLI wrappers to several tools +In-process replacements for the external CLI tools that Pynteny used to shell +out to (HMMER, Prodigal, seqkit). These now rely on the pure-Python, +pip-installable packages pyhmmer, pyrodigal and pyfastx so that Pynteny can be +installed without conda. """ +from __future__ import annotations + +import logging import os +import shlex from pathlib import Path -from pynteny.utils import set_default_output_path, terminal_execute +import pyfastx +import pyhmmer + +from pynteny.utils import set_default_output_path + +logger = logging.getLogger(__name__) def run_seqkit_nodup( input_fasta: Path, output_fasta: Path = None, export_duplicates: bool = False ): - """Simpe CLI wrapper to seqkit rmdup to remove sequence duplicates - in fasta file. + """Remove duplicated sequences from a fasta file (by sequence content). + + Pure-Python replacement for ``seqkit rmdup -s``. Args: input_fasta (Path): path to input fasta. output_fasta (Path, optional): path to output fasta. Defaults to None. export_duplicates (bool, optional): whether to export a file containing - duplicated sequences. Defaults to False. + the IDs of duplicated sequences. Defaults to False. """ input_fasta = Path(input_fasta) if output_fasta is None: output_fasta = set_default_output_path(input_fasta, tag="_no_duplicates") else: output_fasta = Path(output_fasta) + + seen_sequences = set() + duplicated_ids = [] + fasta = pyfastx.Fasta(input_fasta.as_posix(), build_index=False, full_name=True) + with open(output_fasta, "w+", encoding="UTF-8") as outfile: + for record_name, record_seq in fasta: + seq_key = record_seq.upper() + if seq_key in seen_sequences: + duplicated_ids.append(record_name.split()[0]) + continue + seen_sequences.add(seq_key) + outfile.write(f">{record_name}\n{record_seq}\n") + if export_duplicates: dup_file = set_default_output_path( input_fasta, tag="_duplicates", extension=".txt" ) - dup_str = f"-D {dup_file}" - else: - dup_str = "" - cmd_str = ( - f"seqkit rmdup {input_fasta} -s {dup_str} -o {output_fasta.as_posix()} --quiet" - ) - terminal_execute(cmd_str, suppress_shell_output=True) + with open(dup_file, "w+", encoding="UTF-8") as f: + f.write("\n".join(duplicated_ids)) -def run_prodigal( - input_file: Path, - output_file: Path = None, - output_dir: Path = None, - output_format: str = "fasta", - metagenome: bool = True, - additional_args: str = None, -): - """Simple CLI wrapper to prodigal. +def _parse_hmmsearch_args(additional_args: str) -> dict: + """Translate a (subset of) hmmsearch/hmmscan CLI argument strings into + keyword options understood by :func:`pyhmmer.hmmsearch`. + + Only the options commonly used with Pynteny are supported. Unknown flags + (and ``--cpu``, which is handled separately) are ignored with a warning. Args: - input_file (Path): path to input fasta file with nucleotide sequences. - output_file (Path, optional): path to output file containing translated peptides. - Defaults to None. - output_dir (Path, optional): path to output directory (all prodigal output files). - Defaults to None. - output_format (str, optional): either 'gbk' or 'fasta'. Defaults to 'fasta'. - metagenome (bool, optional): whether input fasta correspond to a metagenomic sample. - Defaults to False. - additional_args (str, optional): a string containing additional arguments to prodigal. - Defaults to None. + additional_args (str): CLI-style argument string, e.g. "--cut_ga" or + "-E 1e-10". + + Returns: + dict: keyword options for the pyhmmer Pipeline. """ - input_file = Path(input_file) - if metagenome: - procedure = "meta" - else: - procedure = "single" - if output_dir is None: - output_dir = Path(input_file.parent) - else: - output_dir = Path(output_dir) - if "fasta" in output_format.lower(): - if output_file is None: - output_file = output_dir / f"{input_file.stem}prodigal_output.faa" - out_str = f"-a {output_file}" - else: - if output_file is None: - output_file = output_dir / f"{input_file.stem}prodigal_output.gbk" - out_str = f"-o {output_file}" - output_file = Path(output_file) - if additional_args is not None: - args_str = additional_args - else: - args_str = "" - cmd_str = f"prodigal -i {input_file} -p {procedure} " f"-q {out_str} {args_str}" - terminal_execute(cmd_str, suppress_shell_output=True) + if not additional_args: + return {} + bit_cutoff_flags = { + "--cut_ga": "gathering", + "--cut_nc": "noise", + "--cut_tc": "trusted", + } + float_flags = { + "-E": "E", + "--incE": "incE", + "--domE": "domE", + "-T": "T", + "--incT": "incT", + "--domT": "domT", + "-Z": "Z", + "--domZ": "domZ", + } + options = {} + tokens = shlex.split(additional_args) + i = 0 + while i < len(tokens): + token = tokens[i] + if token in bit_cutoff_flags: + options["bit_cutoffs"] = bit_cutoff_flags[token] + i += 1 + elif token in float_flags: + if i + 1 >= len(tokens): + logger.warning(f"Missing value for hmmsearch argument '{token}'") + break + options[float_flags[token]] = float(tokens[i + 1]) + i += 2 + elif token in ("--cpu",): + # number of threads is handled through the 'processes' argument + i += 2 + else: + logger.warning( + f"Ignoring unsupported hmmsearch argument for pyhmmer: '{token}'" + ) + i += 1 + return options def run_HMM_search( @@ -97,27 +126,45 @@ def run_HMM_search( processes: int = None, additional_args: str = None, ) -> None: - """Simple CLI wrapper to hmmsearch or hmmscan. + """Run an HMM search with pyhmmer and write the results as an HMMER3 + tabular (``--tblout``) output file. + + In-process replacement for the ``hmmsearch``/``hmmscan`` CLI tools. The + on-disk format is unchanged (standard HMMER3 tabular output), so downstream + parsing and result reuse keep working as before. Args: hmm_model (Path): path to profile HMM to be used. - input_fasta (Path): path to fasta containing sequence database to be searched. - output_file (Path, optional): path to prodigal output table file. Defaults to None. + input_fasta (Path): path to fasta containing the sequence database to be searched. + output_file (Path, optional): path to HMMER tabular output file. Defaults to None. method (str, optional): either 'hmmsearch' or 'hmmscan'. Defaults to 'hmmsearch'. - n_processes (int, optional): maximum number of threads. Defaults to all minus one. - additional_args (str, optional): a string containing additional arguments to - hmmsearch/scan. Defaults to None. + processes (int, optional): maximum number of threads. Defaults to all minus one. + additional_args (str, optional): a CLI-style string with additional + arguments for hmmsearch/hmmscan. Defaults to None. """ + hmm_model = Path(hmm_model) + input_fasta = Path(input_fasta) if processes is None: processes = os.cpu_count() - 1 if output_file is None: output_file = set_default_output_path(input_fasta, "_hmmer_hits", ".txt") - if additional_args is not None: - args_str = additional_args else: - args_str = "" - cmd_str = ( - f"{method} --tblout {output_file} {args_str} --cpu {processes} " - f"{hmm_model} {input_fasta}" - ) - terminal_execute(cmd_str, suppress_shell_output=True) + output_file = Path(output_file) + + options = _parse_hmmsearch_args(additional_args) + + with pyhmmer.easel.SequenceFile(input_fasta.as_posix(), digital=True) as seq_file: + sequences = seq_file.read_block() + with pyhmmer.plan7.HMMFile(hmm_model.as_posix()) as hmm_file: + hmms = list(hmm_file) + + search = pyhmmer.hmmscan if method == "hmmscan" else pyhmmer.hmmsearch + if method == "hmmscan": + # hmmscan queries sequences against the HMM database + all_hits = list(search(sequences, hmms, cpus=processes, **options)) + else: + all_hits = list(search(hmms, sequences, cpus=processes, **options)) + + with open(output_file, "wb") as fh: + for i, hits in enumerate(all_hits): + hits.write(fh, format="targets", header=(i == 0)) diff --git a/tests/test_database_build.py b/tests/test_database_build.py index 0689e7e..ebd8be9 100644 --- a/tests/test_database_build.py +++ b/tests/test_database_build.py @@ -9,6 +9,8 @@ import unittest from pathlib import Path +import pyfastx + from pynteny.api import Build from pynteny.preprocessing import Database, LabelledFASTA @@ -31,6 +33,22 @@ def test_build_fasta(self): LabelledFASTA, "Failed to build database from assembly data", ) + records = list( + pyfastx.Fasta( + Path(labelled_database.file_path).as_posix(), + build_index=False, + full_name=True, + ) + ) + self.assertGreater( + len(records), + 0, + "Built database from assembly data is empty", + ) + self.assertTrue( + all(name.startswith("test_") for name, _ in records), + "Sequence prefix was not added to built database records", + ) def test_build(self): with tempfile.NamedTemporaryFile() as outfile: From 00a6d816716b3ec91cfbdbb96c5c87cfaf6bad00 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Semid=C3=A1n=20Robaina=20Est=C3=A9vez?= Date: Thu, 18 Jun 2026 18:24:11 +0100 Subject: [PATCH 2/4] CI: switch tests to pure pip; fix black + biopython 1.87 compatibility - tests.yml: replace the conda env (conda-incubator/setup-miniconda + bioconda) with a pure-pip matrix (actions/setup-python, `pip install .`) over Python 3.9/3.12 on ubuntu + macos. Conda is no longer required after the pyhmmer/pyrodigal migration, and this also fixes the macOS job, which failed creating a py3.8/bioconda env on the ARM64 runner. Add `synchronize` to the pull_request triggers so pushes to a PR re-run the tests. - filter.py: reformat with black 24.8.0 to satisfy the quality check (pre-existing drift; file otherwise unchanged). - preprocessing.py: make Database.is_fasta/is_gbk tolerant of parse errors. Biopython >=1.85 raises (instead of returning no records) when a file is not valid FASTA (e.g. a GenBank file with leading comments), which broke build() on GenBank input once biopython is pulled from PyPI. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/tests.yml | 36 ++++++++++++++++++------------------ src/pynteny/filter.py | 6 +++--- src/pynteny/preprocessing.py | 21 +++++++++++++-------- 3 files changed, 34 insertions(+), 29 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index d6da3e6..737b5ce 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -8,7 +8,7 @@ on: - '**.bib' - 'ms/*' pull_request: - types: [opened, reopened, edited] + types: [opened, reopened, edited, synchronize] paths-ignore: - '**.md' - '**.ipynb' @@ -17,39 +17,39 @@ on: jobs: - create-env: - name: ${{ matrix.os }} + test: + name: ${{ matrix.os }} / py${{ matrix.python-version }} runs-on: ${{ matrix.os }} - defaults: - run: - shell: bash -l {0} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest] + python-version: ["3.9", "3.12"] steps: - name: checkout repository - uses: actions/checkout@v3 + uses: actions/checkout@v4 with: fetch-depth: 0 - - name: create environment - uses: conda-incubator/setup-miniconda@v2 + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v5 with: - python-version: 3.8 - # mamba-version: "*" - channels: conda-forge,bioconda,defaults - auto-activate-base: false - activate-environment: tests_pynteny - environment-file: envs/pynteny-dev.yml + python-version: ${{ matrix.python-version }} - - name: Build & Install Pynteny - run: poetry build && pip install dist/pynteny*.whl + # Pynteny is a pure-pip package (no conda / external binaries required). + - name: Install Pynteny + run: | + python -m pip install --upgrade pip + python -m pip install . + python -m pip install coverage - name: Run tests and collect coverage - run: pynteny --help && coverage run --omit pynteny/wrappers.py,pynteny/cli.py,pynteny/subcommands.py,pynteny/utils.py -m unittest discover tests && coverage xml + run: | + pynteny --help + coverage run --omit pynteny/wrappers.py,pynteny/cli.py,pynteny/subcommands.py,pynteny/utils.py -m unittest discover tests + coverage xml - name: Upload coverage to Codecov uses: codecov/codecov-action@v3 diff --git a/src/pynteny/filter.py b/src/pynteny/filter.py index 9b36697..5f5f5eb 100644 --- a/src/pynteny/filter.py +++ b/src/pynteny/filter.py @@ -220,9 +220,9 @@ def _assign_code_to_HMM(self, hmm_group_name: str) -> int: def _add_meta_codes_to_HMM_hits(self, all_hit_labels: pd.Dataframe) -> pd.Dataframe: """Add numeric codes for each hmm and strand, compute distance between genes""" all_hit_labels["gene_pos_diff"] = all_hit_labels.gene_pos.diff() - all_hit_labels.loc[ - 0, "gene_pos_diff" - ] = 1 # required for rolling (skips first nan) + all_hit_labels.loc[0, "gene_pos_diff"] = ( + 1 # required for rolling (skips first nan) + ) all_hit_labels["hmm_code"] = all_hit_labels.hmm.apply(self._assign_code_to_HMM) all_hit_labels["strand"] = all_hit_labels.strand.apply( lambda strand: -1 if strand == "neg" else (1 if strand == "pos" else 0) diff --git a/src/pynteny/preprocessing.py b/src/pynteny/preprocessing.py index ac9fe9e..d293c68 100644 --- a/src/pynteny/preprocessing.py +++ b/src/pynteny/preprocessing.py @@ -652,10 +652,14 @@ def is_fasta(filename: Path) -> bool: bool: whether the file is in fasta format """ filename = Path(filename) - if filename.exists(): - fasta = list(SeqIO.parse(str(filename), "fasta")) - return any(fasta) - else: + if not filename.exists(): + return False + # Recent Biopython versions raise (instead of returning no records) when + # the file is not valid FASTA (e.g. a GenBank file with leading + # comments), so treat any parse error as "not a FASTA file". + try: + return any(SeqIO.parse(str(filename), "fasta")) + except ValueError: return False @staticmethod @@ -669,10 +673,11 @@ def is_gbk(filename: Path) -> bool: _bool: whether the file is in genbank format """ filename = Path(filename) - if filename.exists(): - gbk = list(SeqIO.parse(str(filename), "genbank")) - return any(gbk) - else: + if not filename.exists(): + return False + try: + return any(SeqIO.parse(str(filename), "genbank")) + except ValueError: return False def build( From b9c4c5bcea07694f9d038479454ca83dad99fa40 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Semid=C3=A1n=20Robaina=20Est=C3=A9vez?= Date: Thu, 18 Jun 2026 18:27:54 +0100 Subject: [PATCH 3/4] Fix pandas 3.0 compatibility: replace DataFrame.applymap with fillna DataFrame.applymap was deprecated in pandas 2.1 and removed in pandas 3.0, breaking get_meta_info_for_HMM on Python 3.12 (which pulls pandas 3.x). The applymap call only replaced NaN with "", which fillna("") does directly and works across all supported pandas versions. Verified: full suite passes in a clean Python 3.12 venv (pandas 3.0.3, biopython 1.87, pyfastx 2.3.1, pyhmmer 0.12.1, pyrodigal 3.7.1). Co-Authored-By: Claude Opus 4.8 --- src/pynteny/hmm.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/pynteny/hmm.py b/src/pynteny/hmm.py index 00cefe5..4184e89 100644 --- a/src/pynteny/hmm.py +++ b/src/pynteny/hmm.py @@ -222,9 +222,7 @@ def get_meta_info_for_HMM(self, hmm_name: str) -> dict: Returns: dict: metadata of provided HMM. """ - meta = self._meta.dropna(subset=["accession"], axis=0).applymap( - lambda x: x if not pd.isna(x) else "" - ) + meta = self._meta.dropna(subset=["accession"], axis=0).fillna("") metadata = { k: list(v.values())[0] if list(v.values())[0] else "undef" for k, v in meta[meta["accession"] == hmm_name].to_dict().items() From c94f58f9776abf6357ba1394076500888be90000 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Semid=C3=A1n=20Robaina=20Est=C3=A9vez?= Date: Thu, 18 Jun 2026 23:50:44 +0100 Subject: [PATCH 4/4] docs: update install instructions for pip-only Pynteny - docs/index.md: lead with `pip install pynteny`; drop the conda-required framing and the obsolete macOS ARM64 conda workaround (pip wheels now work on Apple Silicon). - CONTRIBUTING.md: remove the "HMMER and Prodigal are not available in pip" claim; switch the developer setup to a venv + `pip install -e .` (conda env file kept as optional). - docs/examples/example_api_colab.ipynb: replace the miniconda bootstrap cell with a plain `pip install pynteny`. Also drop paths-ignore from the tests workflow's pull_request trigger: the matrix jobs are required status checks, so a docs-only PR would otherwise skip them and deadlock waiting for a status that never reports. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/tests.yml | 8 +++---- CONTRIBUTING.md | 32 +++++++++++---------------- docs/examples/example_api_colab.ipynb | 31 +++++--------------------- docs/index.md | 32 ++++++++++++++------------- 4 files changed, 39 insertions(+), 64 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 737b5ce..14dc1a2 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -7,13 +7,11 @@ on: - '**.ipynb' - '**.bib' - 'ms/*' + # No paths-ignore here: these jobs are required status checks, and a PR whose + # only changes match an ignore pattern would otherwise skip them and deadlock + # at "Expected - waiting for status to be reported". pull_request: types: [opened, reopened, edited, synchronize] - paths-ignore: - - '**.md' - - '**.ipynb' - - '**.bib' - - 'ms/*' jobs: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9345753..1c4c599 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,36 +32,30 @@ Unsure where to begin contributing to Pynteny? You can start by looking for issu ## Setting up a local developer environment -Pynteny depends on packages that are not available in pip, namely [HMMER](https://github.com/EddyRivasLab/hmmer) and [Prodigal](https://github.com/hyattpd/Prodigal). These can be installed from the bioconda channel. Hence, to setup up a developer environment for Pynteny: +Pynteny is a pure-Python package. Its dependencies (including the HMMER and +Prodigal engines, provided by [pyhmmer](https://github.com/althonos/pyhmmer) +and [pyrodigal](https://github.com/althonos/pyrodigal)) are all installable +from pip, so no conda or external binaries are required. To set up a developer +environment for Pynteny: 1. Fork and download repo, cd to downloaded directory. You should create a new branch to work on your issue. -2. Create conda environment with required dependencies: - -The file `envs/pynteny-dev.yml` contains all dependencies required to use Pynteny. Conda is very slow solving the environment. It is recommended to use [mamba](https://github.com/mamba-org/mamba) instead: - -```bash -mamba env create -n pynteny-dev -f envs/pynteny-dev.yml -conda activate pynteny-dev -``` - -3. Build package +2. Create a virtual environment and install Pynteny in editable mode: ```bash -(pynteny-dev) poetry build +python -m venv .venv +source .venv/bin/activate +pip install -e . ``` -4. Install Pynteny +3. Run tests ```bash -(pynteny-dev) pip install dist/pynteny*.whl +python -m unittest discover tests ``` -5. Run tests - -```bash -(pynteny-dev) python -m unittest discover tests -``` +A conda environment file (`envs/pynteny-dev.yml`) is still provided for +contributors who prefer conda, but it is no longer required. ## Using GitHub codespaces diff --git a/docs/examples/example_api_colab.ipynb b/docs/examples/example_api_colab.ipynb index 5ac80f3..6308683 100644 --- a/docs/examples/example_api_colab.ipynb +++ b/docs/examples/example_api_colab.ipynb @@ -41,33 +41,14 @@ "outputs": [], "source": [ "%%capture\n", - "#@title Install Pynteny (can take a few minutes)\n", + "#@title Install Pynteny\n", "try:\n", " import google.colab\n", - " import sys\n", - " # Install miniconda\n", - " !wget -qnc https://repo.anaconda.com/miniconda/Miniconda3-py38_22.11.1-1-Linux-x86_64.sh\n", - " !chmod +x Miniconda3-py38_22.11.1-1-Linux-x86_64.sh\n", - " !bash ./Miniconda3-py38_22.11.1-1-Linux-x86_64.sh -bfp /usr/local/ 2>&1 1>/dev/null\n", - " !rm ./Miniconda3-py38_22.11.1-1-Linux-x86_64.sh\n", - " # Install Pynteny and dependencies\n", - " !conda install -q -y -c conda-forge -c -c bioconda \\\n", - " python=3.8 \\\n", - " poetry=1.3 \\\n", - " pip \\\n", - " hmmer=3.3 \\\n", - " prodigal=2.6 \\\n", - " pyfastx=0.8 \\\n", - " biopython=1.80 \\\n", - " psutil=5.9 \\\n", - " tqdm \\\n", - " requests 2>&1 1>/dev/null\n", - " !pip install --upgrade numpy==1.21.6\n", - " !pip install --upgrade pandas==1.3.5\n", - " !git clone https://github.com/Robaina/Pynteny.git\n", - " !cd Pynteny && poetry build && pip install -q dist/pynteny*.whl\n", - " !cp -r Pynteny/tests/ . && rm -r Pynteny\n", - " sys.path.insert(0, f\"/usr/local/lib/python3.8/site-packages/\")\n", + " # Pynteny is a pure-pip package (no conda / external binaries required).\n", + " !pip install -q pynteny\n", + " # Grab the bundled test data used throughout this example.\n", + " !git clone -q https://github.com/Robaina/Pynteny.git\n", + " !cp -r Pynteny/tests/ . && rm -rf Pynteny\n", "except ImportError:\n", " pass" ] diff --git a/docs/index.md b/docs/index.md index e6c34e9..03034bf 100644 --- a/docs/index.md +++ b/docs/index.md @@ -13,9 +13,18 @@ These are the available subcommands in the CLI, run as ```pynteny < ## 2. Setup -Install with conda: +Pynteny is a pure-Python package (it no longer requires conda or any external +binaries). The HMMER and Prodigal engines are provided by the pip packages +[pyhmmer](https://github.com/althonos/pyhmmer) and +[pyrodigal](https://github.com/althonos/pyrodigal). -Pynteny requires Python 3.10. The easiest way to handle dependencies is by creating a dedicated conda environment: +Install with pip: + +```bash +pip install pynteny +``` + +Or with conda (from bioconda): ```bash conda create -n pynteny -c bioconda -c conda-forge python=3.10 pynteny @@ -25,22 +34,15 @@ conda activate pynteny Check that installation worked fine: ```bash -(pynteny) pynteny --help +pynteny --help ``` -### 2.1. Installing on Windows -Pynteny is designed to run on Linux machines. However, it can be installed within the [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install) via conda. +### 2.1. Installing on Windows and macOS -### 2.2. Installing on MacOS with the latest ARM64 architecture - -Pynteny doesn't currently support the latest ARM64 architecture of silicon processors (e.g. MacBook M1 and M2). If that is your case, you can install Pynteny using the workaround below (based on [this post](https://towardsdatascience.com/how-to-manage-conda-environments-on-an-apple-silicon-m1-mac-1e29cb3bad12)): - -```bash -CONDA_SUBDIR=osx-64 conda create -n pynteny_x86 python=3.10 -conda activate pynteny_x86 -conda config --env --set subdir osx-64 -conda install -c bioconda pynteny -``` +Pynteny is developed and tested on Linux, but since it is now a pure-Python +package, `pip install pynteny` also works on Windows and macOS (including Apple +Silicon / ARM64), provided wheels are available for pyhmmer and pyrodigal on +your platform. ## 3. General usage