Is DreamCoder only run on the recent GLIBC version? (2.28 or 2.29?) The compression part seems to require GLIBC_2.28 or above. Most forum articles do not suggest updating glibc, so I would like to maintain the environment. Is this compatibility issue stemming from ocaml? Any workaround can you suggest?
In our setup, we observed the GLIBC version issue as below.
The compression part seems to require GLIBC_2.28 or above.
$ singularity exec container.img python -u bin/official_arc_symmetry.py -t 10 -g -i 2
singularity exec container.img python -u bin/official_arc_symmetry.py -t 10 -g -i 2
CUDA is available?: False
using cuda?: False
Running EC on node03 @ 2022-04-28 23:22:10.069179 with 1 CPUs and parameters:
noConsolidation = False
iterations = 2
enumerationTimeout = 10
useRecognitionModel = False
topk_use_only_likelihood = False
pseudoCounts = 30.0
aic = 0.1
structurePenalty = 0.1
arity = 3
taskReranker = default
storeTaskMetrics = True
rewriteTaskMetrics = False
maximumFrontier = 10
solver = python
topK = 5
evaluationTimeout = 1.0
cuda = False
Currently using this much memory: 139558912
Currently using this much memory: 139558912
Using a waking task batch of size: 36
Disabling parallelism on the Python side because we only have one job.
If you are using ocaml or bottom, there could still be parallelism.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 0.000000 <= MDL < 1.500000. Timeout 10.000000.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 1.500000 <= MDL < 3.000000. Timeout 9.884336.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 3.000000 <= MDL < 4.500000. Timeout 9.748159.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 4.500000 <= MDL < 6.000000. Timeout 9.627584.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 6.000000 <= MDL < 7.500000. Timeout 9.466882.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 7.500000 <= MDL < 9.000000. Timeout 9.331560.
(frontend) Launching input -> grid (35 tasks) w/ 1 CPUs. 9.000000 <= MDL < 10.500000. Timeout 8.932637.
(frontend) Launching input -> grid (30 tasks) w/ 1 CPUs. 10.500000 <= MDL < 12.000000. Timeout 7.100658.
We enumerated this many programs, for each task:
[567, 2472, 2472, 2472, 2472, 2472, 2472, 567, 2472, 2472, 2472, 2472, 2472, 2472, 2472, 567, 2472, 111, 2472, 2472, 2472, 567, 2472, 2472, 2472, 567, 2472, 2472, 2472, 2472, 2472, 2472, 2472, 2472, 2472, 2472]
Generative model enumeration results:
HIT 30 w/ (lambda (object (input $0))) ; log prior = -3.891820 ; log likelihood = 0.000000
HIT 38 w/ (lambda (left_half (rotate_cw (left_half (object (input $0)))))) ; log prior = -9.729551 ; log likelihood = 0.000000
MISS 52
MISS 56
MISS 66
MISS 70
MISS 82
HIT 86 w/ (lambda (rotate_cw (rotate_cw (input $0)))) ; log prior = -5.837730 ; log likelihood = 0.000000
MISS 105
MISS 108
HIT 112 w/ (lambda (overlay (input $0) (x_mirror (input $0)))) ; log prior = -7.783641 ; log likelihood = 0.000000
HIT 115 w/ (lambda (combine_grids_vertically (x_mirror (input $0)) (input $0))) ; log prior = -7.783641 ; log likelihood = 0.000000
MISS 116
HIT 139 w/ (lambda (rotate_cw (rotate_cw (input $0)))) ; log prior = -5.837730 ; log likelihood = 0.000000
MISS 141
HIT 149 w/ (lambda (rotate_cw (rotate_cw (x_mirror (input $0))))) ; log prior = -7.783641 ; log likelihood = 0.000000
MISS 151
HIT 154 w/ (lambda (x_mirror (input $0))) ; log prior = -3.891820 ; log likelihood = 0.000000
MISS 163
HIT 171 w/ (lambda (combine_grids_vertically (input $0) (x_mirror (input $0)))) ; log prior = -7.783641 ; log likelihood = 0.000000
HIT 176 w/ (lambda (object (rotate_cw (rotate_cw (x_mirror (input $0)))))) ; log prior = -9.729551 ; log likelihood = 0.000000
HIT 178 w/ (lambda (rotate_cw (x_mirror (input $0)))) ; log prior = -5.837730 ; log likelihood = 0.000000
MISS 179
HIT 209 w/ (lambda (combine_grids_vertically (input $0) (x_mirror (input $0)))) ; log prior = -7.783641 ; log likelihood = 0.000000
MISS 210
HIT 240 w/ (lambda (rotate_cw (x_mirror (input $0)))) ; log prior = -5.837730 ; log likelihood = 0.000000
MISS 241
MISS 243
MISS 248
MISS 310
MISS 345
MISS 359
MISS 360
HIT 379 w/ (lambda (rotate_cw (rotate_cw (rotate_cw (input $0))))) ; log prior = -7.783641 ; log likelihood = 0.000000
MISS 371
HIT 384 w/ (lambda (overlay (input $0) (x_mirror (input $0)))) ; log prior = -7.783641 ; log likelihood = 0.000000
Hits 15/36 tasks
Average description length of a program solving a task: 7.005277 nats
Generative model average: 1 sec. median: 1 max: 2 standard deviation 0
Currently using this much memory: 148295680
Frontiers discovered top down: 15
Total frontiers: 15
Currently using this much memory: 148295680
Showing the top 5 programs in each frontier being sent to the compressor:
30
-0.23 (lambda (object (input $0)))
-2.17 (lambda (object (object (input $0))))
-4.12 (lambda (object (object (object (input $0)))))
-4.12 (lambda (object (overlay (input $0) (input $0))))
-4.12 (lambda (object (x_mirror (x_mirror (input $0)))))
38
-0.36 (lambda (left_half (rotate_cw (left_half (object (input $0))))))
-2.30 (lambda (left_half (rotate_cw (left_half (object (object (input $0)))))))
-2.30 (lambda (left_half (rotate_cw (left_half (object (rotate_cw (input $0)))))))
-2.30 (lambda (left_half (rotate_cw (left_half (rotate_cw (object (input $0)))))))
86
-0.44 (lambda (rotate_cw (rotate_cw (input $0))))
-2.38 (lambda (object (rotate_cw (rotate_cw (input $0)))))
-2.38 (lambda (rotate_cw (object (rotate_cw (input $0)))))
-2.38 (lambda (rotate_cw (rotate_cw (object (input $0)))))
-4.33 (lambda (overlay (rotate_cw (rotate_cw (input $0))) (input $0)))
112
-1.06 (lambda (overlay (input $0) (x_mirror (input $0))))
-1.06 (lambda (overlay (x_mirror (input $0)) (input $0)))
-3.01 (lambda (object (overlay (input $0) (x_mirror (input $0)))))
-3.01 (lambda (object (overlay (x_mirror (input $0)) (input $0))))
-3.01 (lambda (overlay (input $0) (rotate_cw (rotate_cw (input $0)))))
115
-0.59 (lambda (combine_grids_vertically (x_mirror (input $0)) (input $0)))
-2.53 (lambda (combine_grids_vertically (object (x_mirror (input $0))) (input $0)))
-2.53 (lambda (combine_grids_vertically (x_mirror (input $0)) (object (input $0))))
-2.53 (lambda (combine_grids_vertically (x_mirror (object (input $0))) (input $0)))
-2.53 (lambda (object (combine_grids_vertically (x_mirror (input $0)) (input $0))))
139
-0.11 (lambda (rotate_cw (rotate_cw (input $0))))
-4.00 (lambda (rotate_cw (rotate_cw (overlay (input $0) (input $0)))))
-4.00 (lambda (rotate_cw (rotate_cw (x_mirror (x_mirror (input $0))))))
-4.00 (lambda (rotate_cw (x_mirror (x_mirror (rotate_cw (input $0))))))
-4.00 (lambda (x_mirror (rotate_cw (rotate_cw (x_mirror (input $0))))))
149
-1.15 (lambda (rotate_cw (rotate_cw (x_mirror (input $0)))))
-1.15 (lambda (x_mirror (rotate_cw (rotate_cw (input $0)))))
-3.09 (lambda (object (rotate_cw (rotate_cw (x_mirror (input $0))))))
-3.09 (lambda (object (x_mirror (rotate_cw (rotate_cw (input $0))))))
-3.09 (lambda (rotate_cw (object (rotate_cw (x_mirror (input $0))))))
154
-0.36 (lambda (x_mirror (input $0)))
-2.30 (lambda (object (x_mirror (input $0))))
-2.30 (lambda (x_mirror (object (input $0))))
-4.25 (lambda (object (object (x_mirror (input $0)))))
-4.25 (lambda (object (x_mirror (object (input $0)))))
171
-0.59 (lambda (combine_grids_vertically (input $0) (x_mirror (input $0))))
-2.53 (lambda (combine_grids_vertically (input $0) (object (x_mirror (input $0)))))
-2.53 (lambda (combine_grids_vertically (input $0) (x_mirror (object (input $0)))))
-2.53 (lambda (combine_grids_vertically (object (input $0)) (x_mirror (input $0))))
-2.53 (lambda (object (combine_grids_vertically (input $0) (x_mirror (input $0)))))
176
-2.11 (lambda (object (rotate_cw (rotate_cw (x_mirror (input $0))))))
-2.11 (lambda (object (x_mirror (rotate_cw (rotate_cw (input $0))))))
-2.11 (lambda (rotate_cw (object (rotate_cw (x_mirror (input $0))))))
-2.11 (lambda (rotate_cw (rotate_cw (object (x_mirror (input $0))))))
-2.11 (lambda (rotate_cw (rotate_cw (x_mirror (object (input $0))))))
178
-0.44 (lambda (rotate_cw (x_mirror (input $0))))
-2.38 (lambda (object (rotate_cw (x_mirror (input $0)))))
-2.38 (lambda (rotate_cw (object (x_mirror (input $0)))))
-2.38 (lambda (rotate_cw (x_mirror (object (input $0)))))
-4.33 (lambda (overlay (rotate_cw (x_mirror (input $0))) (input $0)))
209
-0.15 (lambda (combine_grids_vertically (input $0) (x_mirror (input $0))))
-2.10 (lambda (x_mirror (combine_grids_vertically (input $0) (x_mirror (input $0)))))
-4.04 (lambda (x_mirror (x_mirror (combine_grids_vertically (input $0) (x_mirror (input $0))))))
240
-0.44 (lambda (rotate_cw (x_mirror (input $0))))
-2.38 (lambda (object (rotate_cw (x_mirror (input $0)))))
-2.38 (lambda (rotate_cw (object (x_mirror (input $0)))))
-2.38 (lambda (rotate_cw (x_mirror (object (input $0)))))
-4.33 (lambda (rotate_cw (object (object (x_mirror (input $0))))))
379
-0.77 (lambda (rotate_cw (rotate_cw (rotate_cw (input $0)))))
-0.77 (lambda (x_mirror (rotate_cw (x_mirror (input $0)))))
-4.66 (lambda (rotate_cw (rotate_cw (rotate_cw (overlay (input $0) (input $0))))))
-4.66 (lambda (rotate_cw (rotate_cw (rotate_cw (x_mirror (x_mirror (input $0)))))))
-4.66 (lambda (rotate_cw (rotate_cw (x_mirror (x_mirror (rotate_cw (input $0)))))))
384
-0.98 (lambda (overlay (input $0) (x_mirror (input $0))))
-0.98 (lambda (overlay (x_mirror (input $0)) (input $0)))
-2.92 (lambda (object (overlay (input $0) (x_mirror (input $0)))))
-2.92 (lambda (object (overlay (x_mirror (input $0)) (input $0))))
-2.92 (lambda (x_mirror (overlay (input $0) (x_mirror (input $0)))))
Compression message saved to: compressionMessages/2022-04-28T23:22:20.223555
/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/../compression: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/../compression)
/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/../compression: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/../compression)
Induced a grammar in 0.1 seconds
Traceback (most recent call last):
File "bin/official_arc_symmetry.py", line 72, in <module>
for i, result in enumerate(generator):
File "/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/dreamcoder.py", line 472, in ecIterator
iteration=j)
File "/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/dreamcoder.py", line 693, in consolidate
backend=compressor, CPUs=CPUs, iteration=iteration)
File "/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/compression.py", line 55, in induceGrammar
g, newFrontiers = ocamlInduce(*args, factored=(backend == "vs_factored"), **kwargs)
File "/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/compression.py", line 159, in ocamlInduce
response = json.loads(response.decode("utf-8"))
File "/usr/local/conda/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/local/conda/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/conda/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
$ singularity --version
singularity-ce version 3.9.6
$ rpm -q glibc
glibc-2.17-325.el7_9.x86_64
glibc-2.17-325.el7_9.i686
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Is DreamCoder only run on the recent GLIBC version? (2.28 or 2.29?) The compression part seems to require GLIBC_2.28 or above. Most forum articles do not suggest updating glibc, so I would like to maintain the environment. Is this compatibility issue stemming from ocaml? Any workaround can you suggest?
Setup
ec/dreamcoder/../compressionIn our setup, we observed the GLIBC version issue as below.
The compression part seems to require GLIBC_2.28 or above.