Installing xarray in Linux Mint

[DanRRRR]

Hi author and users,

Was excited with sdf-xarray detailed description and many very nice examples and decided to install and try it. I use Linux Mint ( the one which is the most user-friendly among all Linux distributions, closely resembling Windows and having safety measures against bad installations. It is based on Ubuntu. We run EPOCH on it and via WINE simultaneously Windows too) and Python I already installed before which I checked using command python --version and got: Python3.12.3

Then I have cloned xarray exactly how Getting Started https://sdf-xarray.readthedocs.io/en/latest/getting_started.html suggests

1. git clone --recursive https://github.com/epochpic/sdf-xarray.git
2. got into sdf-xarray directory cd sdf-xarray
3. and tried to install it using command pip install
4. Got the error message
   Command 'pip' not found, but can be installed with:
   sudo apt install python3-pip
5. Ok, I installed python3-pip . Installation message is long but the key part is here (might be not needed but let it be just in case because many will probably experience the same problem)

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
libjs-sphinxdoc libjs-underscore libpython3-dev libpython3-stdlib
libpython3.12-dev libpython3.12-minimal libpython3.12-stdlib
libpython3.12t64 python3 python3-dev python3-minimal
python3-setuptools python3-wheel python3.12 python3.12-dev
python3.12-minimal
Suggested packages:
python3-doc python3-tk python3-venv python-setuptools-doc
python3.12-venv python3.12-doc binfmt-support
The following NEW packages will be installed:
libjs-sphinxdoc libjs-underscore libpython3-dev libpython3.12-dev
python3-dev python3-pip python3-setuptools python3-wheel
python3.12-dev
The following packages will be upgraded:
libpython3-stdlib libpython3.12-minimal libpython3.12-stdlib
libpython3.12t64 python3 python3-minimal python3.12
python3.12-minimal
8 upgraded, 9 newly installed, 0 to remove and 402 not upgraded.
Need to get 16.5 MB of archives.
After this operation, 40.5 MB of additional disk space will be used.

6. Then I tried to install sdf-xarray with command pip install and got the following error

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

7. On that point all further installation attempts like apt install python3-sdf-xarray etc failed with the error messages Unable to locate package sdf-xarray etc

Any suggestions would be greatly appreciated !

[LiamPattinson]

Hi @DanRRRR, I can't say I'm familiar with Mint, but it looks like the issue is around Python packages being installed system-wide. You might find it helpful to use a virtual environment:

$ sudo apt install python3-venv
$ cd your/current/project
$ python3 -m venv venv  # Creates a virtual environment called 'venv' in the current directory
$ source venv/bin/activate  # Activate the virtual environment
$ pip install .  # Will install to the venv, not to your system environment
$ # ... do whatever work you want ...
$ deactivate  # Exit the virtual environment 

Using virtual environments means you don't end up accumulating packages from every project you've ever worked on, you avoid version conflicts, and you can safely install/uninstall things without risk of breaking your system libraries.

You might also want to look into the tool uv, which can be used as a drop-in replacement for both pip and venv, can install different Python versions, and allows you to capture your environment in a 'lockfile' so others can exactly reproduce your Python environment.

If you still want to install system-wide, you may also have more luck using pip install --user ., which directs pip to use the environment at $HOME/.local/ or somewhere similar. I personally wouldn't recommend it, and I definitely wouldn't use sudo pip install with anything, as that can cause some serious issues if you make a mistake.

[DanRRRR]

Thanks Liam for the useful suggestions,

The reason for the error seems is Linux Mint's safety measure which prevented pip from installing packages directly into the system's Python environment. It is possible to override this restriction but you are right the accumulation of packages is not good, it is an invitation for inevitable disaster. Will proceed with the virtual environment way and then may be try your other suggestions.

Can sdf-xarray handle cylindrical EPOCH or can it transform EPOCH PIC variables into other non-SDF file formats?

[JoelLucaAdams]

Hi Dan,

As Liam has already suggested, the easiest way I find to work with this is to simply install the uv command line tool. Then run the following:

Important

You'll need to activate the environment every time you open a new terminal in order for sdf-xarray to be picked up. If you use VScode this should work automatically when you open the sdf_xarray folder in it

# in the sdf_xarray folder
uv sync # This installs all the packages to a hidden folder in sdf_xarray called .venv
source .venv/bin/activate # This "activates" the virtual environment
# ... run your python files as usual ...
# Once done with it run the following to leave the virtual environment
deactivate

We haven't tested sdf-xarray with cylindrical EPOCH in the past but I've run a quick test now and sadly it doesn't seem to support it out of the box and would require some work on our end. I've added it to the backlog (see #112).

sdf-xarray converts the SDF file's contents to an xarray dataset which can easily convert variables to numpy, pandas and several other formats through external packages.

[JoelLucaAdams]

I've updated the documentation to include a section that you might find helpful https://sdf-xarray.readthedocs.io/en/latest/installation.html#new-to-python