PyReweighting test example is outdated, disagrees with latest version of python2 and python3 scripts

[sastrys1]

On the Miao Lab website that discusses reweighting, there is a set of scripts under the header "Test Example", this provides a tar.xz archive of files from 2013-2014. This test example runs through reweighting using an example Phi_Psi dat file and then shows the expected output XVG and PNG heatmap files from the reweighted Phi/Psi angles.

This issue is specifically about the 2nd order cumulant expansion, not sure if this applies to the other options

However, the Py-Reweighting-2D.py file used to make these XVG and PNG script is outdated, and thus provides completely different results if we try the latest version of the python2 or python3 PyReweighting-2D.py scripts on the same Phi_Psi dat file.

Here is the output given in the test example, if you use the old python script given in the tar archive:

python PyReweighting-2D.py -cutoff 10 -input Phi_Psi -Xdim -180 180 -discX 6 -Ydim -180 180 -discY 6 -Emax 20 -job amdweight_CE -weight weights.dat | tee -a reweight_variable.log

DATA LOADED: Phi_Psi
dV all: avg = 4.16023073679 std = 2.13785379582
pmf_min-c1 = -8.44361460879
pmf_min-c2 = -15.5787296211
pmf_min-c3 = -304.297343506

Here is the output, with the latest version of the python2 reweighting script. (here, we run into a bug if we try to explicitly set the dimY variable, so I just left that out of the command and let it use the default value of [-180, 180] since that was exactly what I was trying to explicitly set).

python PyReweighting-2D.py -cutoff 10 -input Phi_Psi -Xdim -180 180 -discX 6 -discY 6 -Emax 20 -job amdweight_CE -weight weights.dat | tee -a reweight_variable.log

DATA LOADED: Phi_Psi
bigdata: False
Reweighting with big matrix
hist_max: 684
dV all: avg = 4.16023073679 std = 2.13785379582
pmf_min-c1 = -8.44361460879
pmf_min-c2 = -20.7355467231
pmf_min-c3 = -48.1616340098

Here is the output, with the latest version of the python3 reweighting script. (Ran into a bug "AttributeError: module 'scipy.misc' has no attribute 'factorial'", so changed that scipy.misc.factorial to scipy.math.factorial)

python PyReweighting-2D.py -cutoff 10 -input Phi_Psi -Xdim -180 180 -discX 6 -Ydim -180 180 -discY 6 -Emax 20 -job amdweight_CE -weight weights.dat | tee -a reweight_variable.log

DATA LOADED: Phi_Psi
dV all: avg = 4.160230736794477 std = 2.1378537958222306
pmf_min-c1 = -8.443614608787296
pmf_min-c2 = -20.73554672305713
pmf_min-c3 = -48.16163400979573

I have also attached the XVG files for the 2nd order cumulant expansion reweighting. The latest python2 and python3 XVG files are pretty much identical except the python3 is rounded to 3 decimal places, and the test example script XVG output is completely different from the python2/python3 XVG output.

pmf-c2-Phi_Psi-PYTHON-3-VERSION.txt
pmf-c2-Phi_Psi-PYTHON-2-VERSION.txt
pmf-c2-Phi_Psi-TEST_EXAMPLE.txt

TL;DR: Looks like the latest versions of the python2 and python3 reweighting scripts are identical, but they both are different from the test example script output, on the same exact data. I suspect that the reweighting algorithm was changed sometime between when the test example was uploaded to the website and when the latest versions of the python2 and python3 scripts were uploaded.

I think it would be good to update that test example folder with the latest Python3 code (and update the latest python3 code to use scipy.math.factorial instead of scipy.misc.factorial). My mentor and I were trying to follow that test example and fully re-capitulate the reweighted heatmap from the Phi/Psi data with the new scripts, and I was very puzzled as to why my results were so different. Updating the test example may help new GaMD users understand the reweighting better with the more modern version of the code.

P.S. I appreciate from the bottom of my heart the hard work that has gone into the PPI-GaMD, Pep-GaMD, Lig-GaMD, and GaMD innovations. It is an incredibly beautiful idea to get around the earlier inconveniences of aMD reweighting. I've learned so much about MD and enhanced sampling from the Miao Lab papers, I'm truly thankful for all your hard work.

[MiaoLab20]

@sastrys1 : Thank you very much for your notes, and nice words about our work on GaMD method developments!

Could you possibly try the latest scripts we posted to PyReweighting website and see how they work:
http://miaolab.org/PyReweighting/

Please let us know if you continue to have the issue. We will try to look into it and hope to get it resolved soon!

[sastrys1]

Apologies for the late reply, I was on vacation the past couple of weeks!

The latest scripts posted to the PyReweighting website work great, and both the Python2 and Python3 scripts agree with each other. My comment was specifically referring to the files under the test example header on the PyReweighting website. That test example in the tar archive is from 2014, and it appears to be pretty outdated.

The PyReweighting-2D.py and PyReweighting-3D.py provide different results than the script provided in the Test Example section.

Since that Test Example section provides a good place to start for people new to GaMD reweighting, it may be worthwhile to update both the scripts and the results files to the latest versions of the PyReweighting-1D.py and PyReweighting-2D.py scripts. I feel this way especially because the test example code is Python2 code, while these days most systems run Python3 as a default.

I have created a new folder, in which I have copied the same starting data files that are in the test example folder, and replaced the scripts with the most updated versions. I have created a pull request - perhaps this folder could be made into a tar archive and uploaded to the website to replace the one from 2014 if you deem it suitable.

[MiaoLab20]

Thanks for your updates of the Test Example!

We will try to double check and upload it to the GaMD website.

[groponp]

Dear the link : http://miaolab.org/PyReweighting/ isn't working where can found the right tutorial?

[MiaoLab20]

It's unfortunate that the miaolab.org webiste is down for the moment. You can access the pyreweighting tutorial at: https://github.com/MiaoLab20/pyreweighting (see the Tutorial section).