Can't call the metric handler function for...

[rubinlinux]

Similar to (#88), the systemd logs of my systems are all constantly flooded with streams of messages such as:

journalctl | grep gmond

[PYTHON] Can't call the metric handler function for [udp_inerrors] in the python module [netstats].
[PYTHON] Can't call the metric handler function for [udp_rcvbuferrors] in the python module [netstats].
[PYTHON] Can't call the metric handler function for [tcpext_listendrops] in the python module [netstats].
[PYTHON] Can't call the metric handler function for [tcp_attemptfails] in the python module [netstats].
[PYTHON] Can't call the metric handler function for [tcpext_tcploss_percentage] in the python module [netstats].
[PYTHON] Can't call the metric handler function for [tcp_retrans_percentage] in the python module [netstats].
[PYTHON] Can't call the metric handler function for [tcp_outsegs] in the python module [netstats].
[PYTHON] Can't call the metric handler function for [tcp_insegs] in the python module [netstats].
[PYTHON] Can't call the metric handler function for [udp_indatagrams] in the python module [netstats].
[PYTHON] Can't call the metric handler function for [udp_outdatagrams] in the python module [netstats].

Perhaps the netstats module needs to do some more validation before calling functions?

I'm on debian 9 (ganglia-monitor 3.6.0-7)

[HamishB]

Hi, I'm seeing the same thing in /var/log/syslog on Ubuntu 14.04 with ganglia-monitor 3.6.0-1ubuntu2. Since the last few weeks, about the same time you reported the bug.
Kernel is Ubuntu's 4.4.0-124-generic, up from 3.16.0.

What's more nodes are not reporting data back to the gmetad host so rrds are not being updated.

:-/
Hamish

[HamishB]

(reboot of the cluster solved the trouble with the rrds not being updated)
The netstats problem on the nodes remains.

[refraction-ray]

Check whether #88 (comment) helps

[titansmc]

I still see this error on gmond 3.7.2 from EPEL 7 that has the patch proposed here applied.

We also see this on 3.7.2

Still monitoring, but so far I believe I was able to implement a fix for 3.7.2

Manually changed /usr/lib64/ganglia/python_modules/netstats.py (your installation location may differ) to catch ZeroDivisionErrors in the delta function. I can't see how this would have a negative effect on monitoring since the exception should only invoke in cases of divide-by-zero situations. So far so good on my end on a node I'm testing this on.

def get_delta(name):
    """Return change over time for the requested metric"""

    # get metrics
    [curr_metrics, last_metrics] = get_metrics()

    parts = name.split("_")
    group = parts[0]
    metric = "_".join(parts[1:])

    try:
        delta = (float(curr_metrics['data'][group][metric]) - float(last_metrics['data'][group][metric])) / (curr_metrics['time'] - last_metrics['time'])
        if delta < 0:
            print name + " is less 0"
            delta = 0
    except KeyError:
        delta = 0.0
    except ZeroDivisionError:
        delta = 0.0

    return delta

The only change being the addition of the lines:

except ZeroDivisionError:
        delta = 0.0