Skip to content

Bug: "ValueError: array must not contain infs or NaNs" #23

Description

@bmorris3

Some values of input parameters give the following intimidating, cryptic and uninformative error:

(base) tycho:dot bmmorris$ python -c "from dot.tests.test_model import test_model_optimizer as f; f()"
optimizing logp for variables: [dot_R_spot, dot_lat, dot_lon, dot_comp_inc, dot_shear, dot_P_eq, dot_f0]
264it [00:06, 38.29it/s, logp=-6.690179e+04]array: [-0.15787993 -0.52647481 -0.84561983 -2.19530702 -0.25451239 -0.74975759
  0.59741809 -0.52742003 -0.4755868   0.75138306]
point: {'dot_f0_interval__': array(0.75138306), 'dot_P_eq_interval__': array(-0.4755868), 'dot_shear_interval__': array(-0.52742003), 'dot_comp_inc_interval__': array(0.59741809), 'dot_lon_interval__': array([[-0.25451239, -0.74975759]]), 'dot_lat_interval__': array([[-0.84561983, -2.19530702]]), 'dot_R_spot_interval__': array([[-0.15787993, -0.52647481]])}
Traceback (most recent call last):
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/op.py", line 892, in rval
    r = p(n, [x[0] for x in i], o)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/tensor/slinalg.py", line 250, in perform
    A, b, lower=True)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/linalg/basic.py", line 336, in solve_triangular
    b1 = _asarray_validated(b, check_finite=check_finite)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/_lib/_util.py", line 239, in _asarray_validated
    a = toarray(a)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 486, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/exoplanet/utils.py", line 197, in objective
    *get_args_for_theano_function(bij.rmap(vec), model=model)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 917, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/op.py", line 892, in rval
    r = p(n, [x[0] for x in i], o)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/tensor/slinalg.py", line 250, in perform
    A, b, lower=True)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/linalg/basic.py", line 336, in solve_triangular
    b1 = _asarray_validated(b, check_finite=check_finite)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/_lib/_util.py", line 239, in _asarray_validated
    a = toarray(a)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 486, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs
Apply node that caused the error: Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}(TensorConstant{[[1.287538..6413e-03]]}, Elemwise{Composite{((i0 + i1 + i2) - i3)}}[(0, 2)].0)
Toposort index: 65
Inputs types: [TensorType(float64, matrix), TensorType(float64, col)]
Inputs shapes: [(1418, 1418), (1418, 1)]
Inputs strides: [(8, 11344), (8, 8)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[InplaceDimShuffle{1,0}(Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}.0), Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 2.0}, TensorConstant{(1, 1) of 0.5}, Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/distribution.py", line 84, in __new__
    return model.Var(name, dist, data, total_size, dims=dims)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/model.py", line 1118, in Var
    model=self,
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/model.py", line 1744, in __init__
    self.logp_sum_unscaledt = distribution.logp_sum(data)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/distribution.py", line 162, in logp_sum
    return tt.sum(self.logp(*args, **kwargs))
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 328, in logp
    quaddist, logdet, ok = self._quaddist(value)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 113, in _quaddist
    dist, logdet, ok = self._quaddist_cov(delta)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 138, in _quaddist_cov
    return self._quaddist_chol(delta)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 132, in _quaddist_chol
    delta_trans = self.solve_lower(chol_cov, delta.T).T

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
Traceback (most recent call last):
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/op.py", line 892, in rval
    r = p(n, [x[0] for x in i], o)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/tensor/slinalg.py", line 250, in perform
    A, b, lower=True)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/linalg/basic.py", line 336, in solve_triangular
    b1 = _asarray_validated(b, check_finite=check_finite)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/_lib/_util.py", line 239, in _asarray_validated
    a = toarray(a)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 486, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/bmmorris/git/dot/dot/tests/test_model.py", line 24, in test_model_optimizer
    map_soln = m.optimize()
  File "/Users/bmmorris/git/dot/dot/model.py", line 309, in optimize
    map_soln = optimize(start=start, **kwargs)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/exoplanet/utils.py", line 218, in optimize
    info = minimize(objective, x0, **kwargs)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/_minimize.py", line 594, in minimize
    return _minimize_bfgs(fun, x0, args, jac, callback, **options)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/optimize.py", line 1016, in _minimize_bfgs
    old_fval, old_old_fval, amin=1e-100, amax=1e100)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/optimize.py", line 836, in _line_search_wolfe12
    **kwargs)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/linesearch.py", line 101, in line_search_wolfe1
    c1=c1, c2=c2, amax=amax, amin=amin, xtol=xtol)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/linesearch.py", line 175, in scalar_search_wolfe1
    phi1 = phi(stp)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/linesearch.py", line 87, in phi
    return f(xk + s*pk, *args)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/optimize.py", line 326, in function_wrapper
    return function(*(wrapper_args + args))
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/optimize/optimize.py", line 64, in __call__
    fg = self.fun(x, *args)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/exoplanet/utils.py", line 197, in objective
    *get_args_for_theano_function(bij.rmap(vec), model=model)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 917, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/gof/op.py", line 892, in rval
    r = p(n, [x[0] for x in i], o)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/theano/tensor/slinalg.py", line 250, in perform
    A, b, lower=True)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/linalg/basic.py", line 336, in solve_triangular
    b1 = _asarray_validated(b, check_finite=check_finite)
  File "/Users/bmmorris/.local/lib/python3.7/site-packages/scipy-1.3.0-py3.7-macosx-10.9-x86_64.egg/scipy/_lib/_util.py", line 239, in _asarray_validated
    a = toarray(a)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 486, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs
Apply node that caused the error: Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}(TensorConstant{[[1.287538..6413e-03]]}, Elemwise{Composite{((i0 + i1 + i2) - i3)}}[(0, 2)].0)
Toposort index: 65
Inputs types: [TensorType(float64, matrix), TensorType(float64, col)]
Inputs shapes: [(1418, 1418), (1418, 1)]
Inputs strides: [(8, 11344), (8, 8)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[InplaceDimShuffle{1,0}(Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}.0), Elemwise{mul,no_inplace}(TensorConstant{(1, 1) of 2.0}, TensorConstant{(1, 1) of 0.5}, Solve{A_structure='lower_triangular', lower=False, overwrite_A=False, overwrite_b=False}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/distribution.py", line 84, in __new__
    return model.Var(name, dist, data, total_size, dims=dims)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/model.py", line 1118, in Var
    model=self,
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/model.py", line 1744, in __init__
    self.logp_sum_unscaledt = distribution.logp_sum(data)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/distribution.py", line 162, in logp_sum
    return tt.sum(self.logp(*args, **kwargs))
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 328, in logp
    quaddist, logdet, ok = self._quaddist(value)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 113, in _quaddist
    dist, logdet, ok = self._quaddist_cov(delta)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 138, in _quaddist_cov
    return self._quaddist_chol(delta)
  File "/Users/bmmorris/miniconda3/lib/python3.7/site-packages/pymc3/distributions/multivariate.py", line 132, in _quaddist_chol
    delta_trans = self.solve_lower(chol_cov, delta.T).T

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
265it [00:09, 29.26it/s, logp=-6.690179e+04]

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions