Instamatic's modular design

[Baharis]

Introduction

Instamatic as a whole is designed to be a very modular. Several "areas" of the program easily accept new additions that either expand program capabilities or allow replacing existing ones. We can list many modular areas of Instamatic, including:

• microscope interfaces: individual files @ microscope/interface/ directory or external (e.g. instamatic_tecnai_server);
• camera interfaces: sets of files @ camera directory or external (e.g. instamatic_TEM_emulator);
• controller jobs: one file @ gui/jobs.py (e.g. collect_flatfield.py);
• experiment routines: individual sub-directories @ experiments/ directory (e.g. cred/);
• GUI modules: individual files @ gui/ directory (e.g. cred_frame.py);
• calibration routines: individual files @ calibrate/ directory (e.g. cred_frame.py).

Multiple areas that are not "modular" per-se are designed in a clear way that allows easy introduction of modularity:

• server directory: stores individual files @ servers/ (that are for some reason mostly independent);
• image formats: individual files @ formats/ collect tools for diffraction image handling;
• image processing: individual files @ processing/ handles saving and modifying images;

Litany of issues

The overall structure looks fine at the first glance, but the closer you look, the more cracks you start to see. The microscope and camera interface design is, in my opinion, the best part of the code, largely thanks to @viljarjf's refactoring in #91. However, their internal structure is different, with the camera parts being much less organized. Until recently, the camera server would not even work properly, and the communication protocols used by camera and microscope differ (e.g. func_name vs attr_name). More on that later. Adding new microscopes and cameras is mostly straightforward but require some tricks e.g. the GUI and server config interface must differ.

instamatic.microscope structure	instamatic.camera structure	Whatever is going on between experiments and GUI

Small controller jobs are held in gui/jobs.py which is completely fine by itself. However, to register more jobs, they need to be added in particular Instamatic directory with a fixed format, which starts being odd. This becomes even more problematic, when we remember about GUI modules. Most of the GUI modules are there to offer interface to some experiments i.e. job – adding an experiment type is always associated with adding GUI window. If that is the case, then why are they located in two completely different places? Why are experiments accessed via gui modules (they import experiments to define commands themselves)?

Microscopes and cameras can be defined outside Instamatic and developed under a separate license which is great, as it allows their development in private repositories. However: jobs, experimental routines, GUI modules, calibrations, image formaters/processors are fixed in store and MUST be developed as part of Instamatic. The only way to have these hidden from them community is creating a private fork that will rarely or never be updated due to conflict issues. This does raise the barriers for new features to be re-implemented into the main branch; and we know some companies and researches are unwilling to dedicate time to that.

instamatic.microscope.client	instamatic.camera.camera_client	instamatic.goniotool

Servers (and calibrations, to some degree) are another can of worms. Despite being structurally very similar, the amount of repetition between individual scripts is massive. Microscope, camera, and goniotool servers use mostly the same architecture but NO inheritance. Improvements and bug-fixes that make their way into one of them further increase the disparity between their mechanisms instead of addressing common issues. The communication mechanism is also simple but finicky: instructions and results are sent using two different protocols and do not allow for asynchronous evaluation. Because they send and receive directly using socket.send and socket.recv they are bare-bones and super-fast, but also super-easy to break. For a nice example, neither of these commands assumes that the message must be passed in a single piece: it just happens to, and so Instamatic communication relies on a series of happy accidents.

Identifying improvement areas

I did not spend 2 hours writing this text just to complain or bash Instamatic – au contraire, I find its design philosophy highly appealing, to the point where I decided to make it a centerpoint of my everyday work. I would rather use this issue as a discussion, where anyone can list loose ideas for development. I intend to update it as needed, starting from the issues I have listed above:

• TEM/cam interface structure is incomplete and inconsistent, may be generalized for more devices (precession/temp control?);
• All client-server communication uses inheritance and simplistic protocols (prevent yielding, asynchronicity);
• Jobs, experiments, and GUI elements are scattered and can't be implemented and imported from outside Instamatic;
• Calibration routines include no clear inheritance or defined design philosophy, not obvious which are still used/usable;
• Image processing and output saving is only partially generalized and does not easily allow for extensions;

Given time, I would personally like to focus on the jobs/experiments structure. In my opinion it is the lowest-hanging and best-defined fruit that could be easily addressed using importlib entry points. For example, what bothers me right now is the existence of 3 different classes designed to run the same cRED experiment. None of them has an active maintainer, and from my personal perspective 2 of them contribute nothing but maintenance burden. Having them in a separate repository with dedicated goal and maintainer would allow its use to the groups they were originally designed for while having a known maintainer and keeping the GUI clear for the core functionality.

Motivation and some thoughts

I wrote about all of these issues as a way to conceptualize, collect, and put on paper all the issues or ideas that I've had with/for the code in the last year. I would like to give others an option to do the same, as well as create a little space where we can declare some abstract ideas about the direction we may want to push the development of Instamatic in the future. This is not expected to be a TODO list: in fact, some of the things I suggest here are likely moronic and out-of-touch. But I would love to have a discussion whether that is the case and which ideas reverberate with others, developers, users, or future-myself-that-found-some-time-for-coding the most.

[stefsmeets]

The overall structure looks fine at the first glance, but the closer you look, the more cracks you start to see.

🫣

However: jobs, experimental routines, GUI modules, calibrations, image formaters/processors are fixed in store and MUST be developed as part of Instamatic.

This is indeed a big issue, and it's also how the software has grown over the years. Initially, the GUI was nothing but a tkinter windows showing the current frame, a 'save frame' button and some stats on the top. This grew over time as we found it convenient to have experiments be controlled via the gui.

Microscope, camera, and goniotool servers use mostly the same architecture but NO inheritance.

The servers are ducttape to get stuff to work. Speed was my main concern here and to offload processes from the main thread. They turned out to be quite useful, but I found them very difficult to modularize because of small nuances in the interfaces. Definitely ripe for refactoring with the right design (e.g. RPC).

[stefsmeets]

Great write-up btw! Hit the nail on the head with all the issues you listed. Most of it grew organically. It's a pretty big undertaking to try to sort it out and refactor to a better model, but definitely a nice spot on the horizon.

Personally, I thinks splitting the 'core' stuff from the experimental code would be a great step forward.

Core being:

• microscope / camera interfaces
• TEM Controller
• measurement utils
• Grid montaging
• Calibrations
• Simulation

This could make a very nice (and light) standalone package. The rest can be shedded and put in a seperate repo or user repos.

There is also a lot of code that can be deleted:

• Some of the undocumented camera interfaces
• Machine learning bits
• Unmaintained and/or broken experiments (SerialED)

[magnunor]

This discussion is very timely! We at NTNU (myself and @kritvei) have started doing automated TEM experiments for studying magnetic properties in materials, this requires tilting + moving + rotating the sample, changing the objective lens, controlling digiscan and getting data from a MerlinEM detector. Talking to the microscope is done through a lightweight Python library inspired by the JEOL TEM Controller module. This works really well, and it would be nice make this code available to the wider community.

One of our problems is that Instamatic has quite a large dependency list, so I think having a lean "backend" library whose job is talking to the hardware itself would be a nice step forward. Kind of like a "wrapper" for talking to the TEMs. Then the libraries for doing specific experiments would utilize this "backend" library. For example instamatic for doing diffraction experiments, and a new library for doing in-situ magnetic experiments.

[stefsmeets]

One of our problems is that Instamatic has quite a large dependency list, so I think having a lean "backend" library whose job is talking to the hardware itself would be a nice step forward.

Yeah, there are lots of dependencies (some of which quite heavy) that are used for one or two functions in the code, and not at all necessary for the tem / camera interface to work, e.g. Virtualbox, IPython, matplotlib, diffpy.structure.

Or, they are used for the gui layer (e.g. Pillow). Which somewhat duplicates the functionality from scikit_image.

[stefsmeets]

A first start could be to seperate the code on a package level:

• src/instamatic-core (tem interface, cam interface, calibrations, utils)
• src/instamatic-experiments (gui + experimental code)

[Baharis]

@magnunor, @kritvei thank you very much for this feedback! Until someone comments on whether this could be potentially useful or not, it is difficult to judge whether spending time on my random ideas has any real benefit. If you could spare a few moments, it would be incredibly useful to know if there are any particular unique microscope capabilities are you using, any specific libraries or design decisions that specifically (would) prevent you from implementing in Instamatic (other than listed) etc. Maybe even open another issue linking to this one, but a comment with any more details would work great as well!

@stefsmeets I have actually not thought about splitting Instamatic core from GUI, I was thinking more along the lines of splitting off experiments, but I think y'all are raising a really valid point, as I myself know of other groups that use Instamatic as a back-end only.

[magnunor]

Some context: For JEOL TEMs the communication to the microscope is done via TemExt software. This is typically installed on a detector computer (for example a Gatan computer), to allow for the detector software to communicate with the microscope. We can interface to this software using the python library comtypes. A problem is that for some labs, some of their TEMs are stuck with 32-bit Windows, which can not be upgraded.

The easiest way of running Python in such an environment is through Anaconda3 2019.10 Windows x86 (32 bit), which (I think) is the most recent easily installable 32-bit Python environment. Getting any recent NumPy to run in this is really painful. Thus, we made our own little library for talking to the microscope, with only one dependency: comtypes.

For our TEMs with 64-bit computers with TemExt where we can run more modern Python, we also use this library.

I'm not sure how the situation is for old Thermo Fisher Scientific TEMs (FEI). I imagine there are similar issues with old 32-bit computers.

---

So in my ideal world, there would be the "backend" library whose only job is talking to the TEM hardware. With a very minimal set of dependencies so it can run on the 32-bit computers. This could for example also include code for communicating with the MerlinEM detector software: it only requires socket, which is in the Python standard libraries.

For more advanced functionality like receiving data (for example NumPy arrays), it could either have optional dependencies for handling this, or be in a separate library.

Then libraries like Instamatic or other experiment focused libraries could utilize this "backend" library to talk to the hardware.

[Baharis]

@magnunor Thank you a lot for the insight! I do not have experience with Jeol machines, but I believe the TemExt communication mechanism as well as the "backend" library that you are suggesting is already implemented in instamatic. Apart from the rotation speed control, the communication with a Jeol microscope is implemented using TemExt/comtypes. However, because the file is indeed a module of Instamatic, using this interface still requires additional dependencies.

Nevertheless, for FEI microscopes, we have a separate repository called instamatic-tecnai-server that serves as a solution to the problem you posed for FEI machines. It is a lightweight "fragment" of Instamatic that implements only TEM control + server communication and requires only a few light dependencies (comtypes, PyYAML, typing). I use it everyday: the server runs on the microscope PC and I use instamatic from another PC in the same LAN. There is little to nothing that prevents us from having a similar implementation for JEOL machines, other than perceived lack of interested parties.

Regarding advanced functionality and communicating with detectors, in Instamatic we typically split the microscope and detector functionality onto two separate servers. Still, I have recently modified instamatic so that remote cameras can do almost all the same things as local ones, and in my new code I am developing both server and camera servers in the same repository.

I have recently re-implemented the instamatic-tecnai-server for FEI Titan machines and would be happy to make another repository for JEOL computers in this style that enables remote communication with a machine that can handle python 3.4 at most. @magnunor, if you or anyone in your team would have time to spare to test this implementation (I have a test suite ready), I would be more than willing to spend a few days making a standalone server for instamatic users. I understand this is not your highest probability (and neither is mine), but with this small change anyone could use this part as the "backend" you just described.

[magnunor]

The instamatic-tecnai-server is indeed interesting! It should be easy to have a combined one for the JEOL-TemExt interface as well. Being able to remotely control this library through socket is nice, as having the "main" Python running on a more modern system adds a lot more potential functionality.

I have recently re-implemented the instamatic-tecnai-server for FEI Titan machines and would be happy to make another repository for JEOL computers in this style that enables remote communication with a machine that can handle python 3.4 at most. @magnunor, if you or anyone in your team would have time to spare to test this implementation

We can definitely test such a library, and probably also contribute to it in the future.

My one worry with having two implementations, one in Instamatic proper and one in a lightweight library (like instamatic-tecnai-server), is the extra maintenance burden with having two libraries with similar TEM-control functionalities.

[Baharis]

@magnunor Indeed, but sometimes it is just needed, and on the flip side the backend IMO needs less maintanance than GUI / processing / experiment code. It would be nicer to have some modular design with common parts abstracted to unique repositories, inheritence between repositories, but I'd need to have a good discussion with GPT or some experienced software engineer what is the proper way to do it. For the time being I will have you in mind and try to scramble up some instamatic-jeol-server, and if you do have any scripts that you use to control the TEM for an additional reference, I will gladly accept them at tchon-at-fzu.cz.