[Monitorix-general] Bug in Ethernet bits/second?

[Anthony Stump]

I'm new to mailing lists so if I'm not doing this right let me know.

Yesterday night I discovered this awesome tool called Monitorix. I got it
all installed and it was working fine until this morning, when reviewing
the Fast Ethernet eth0 stats it shows a rather impossible spike to 70 P
bits/second transmission rate.

I was wondering, first, what causes this - second, is there any way to
either (A) remove that reading so I can see accurate stats (as the 70 P
bit/sec reading makes it impossible to see any other readings, and changes
the average bits/second reading to a rather insane 265555577495
Kbits/second, or (B), just reset that graph all together since it's useless
now.

Screenshot here: http://postimg.org/image/n26bsnckv/

[Jordi Sanfeliu]

Hi Anthony,

This ML has the main purpose to help to anyone that has some issue with
Monitorix, so please, feel free to ask whatever you need.

Regarding your question, this is a well know problem that will be fixed
in the next Monitorix version. In fact, it's not actually a Monitorix
related problem it's more a problem of how are defined the type of the
DS (Data Sources) in the RRDtool database for the 'net' graph. Until now
they were defined as COUNTER and since the next Monitorix version they
will be defined as GAUGE.

You can get more information here
<http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html> about the
definition of both DS types, but basically COUNTER Data Sources are
affected if the value in question overflows or starts again from zero,
something that it could happen if the machine has been rebooted very
quickly (perhaps because it's a virtualized system and it takes a very
short time to reboot and start Monitorix again).

Finally and answering your second question, you can get the script
called
<http://oss.oetiker.ch/rrdtool/pub/contrib/removespikes-20080226-mkn.tar.gz>
to remove that huge spike that scales your graph so high that prevents
you to see your current daily detail. Also, such spike will affect
weekly, monthly and yearly views, so indeed, it's very recommended to
remove it as soon as possible.

The script is pretty simple, so you shouldn't have any problem with it.
Regards.

---
Jordi Sanfeliu
FIBRANET Network Services Provider
http://www.fibranet.cat

Post by Anthony Stump
I'm new to mailing lists so if I'm not doing this right let me know.
Yesterday night I discovered this awesome tool called Monitorix. I got
it all installed and it was working fine until this morning, when
reviewing the Fast Ethernet eth0 stats it shows a rather impossible
spike to 70 P bits/second transmission rate.
I was wondering, first, what causes this - second, is there any way to
either (A) remove that reading so I can see accurate stats (as the 70
P bit/sec reading makes it impossible to see any other readings, and
changes the average bits/second reading to a rather insane
265555577495 Kbits/second, or (B), just reset that graph all together
since it's useless now.
Screenshot here: http://postimg.org/image/n26bsnckv/ [1]
--
Thank you,
Anthony Stump
------
[1] http://postimg.org/image/n26bsnckv/

[Anthony Stump]

Where are the rrd's I need to modify? On the documentation it says that the
file should be net.rrd but I have several "interface eth0" rrds in
/var/lib/munin/localdomain/ which are as follows:

***@astump-EX58-UD4P:/$ ls
/var/lib/munin/localdomain/localhost.localdomain-if_*
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-collisions-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-txdrop-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-rcvd-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_eth0-down-d.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-rxdrop-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_eth0-up-d.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-trans-c.rrd


I have tried removing spikes from all of them and none of them find spikes.
When in analyze mode, they are all empty.

Post by Jordi Sanfeliu
Hi Anthony,
This ML has the main purpose to help to anyone that has some issue with
Monitorix, so please, feel free to ask whatever you need.
Regarding your question, this is a well know problem that will be fixed
in the next Monitorix version. In fact, it's not actually a Monitorix
related problem it's more a problem of how are defined the type of the
DS (Data Sources) in the RRDtool database for the 'net' graph. Until now
they were defined as COUNTER and since the next Monitorix version they
will be defined as GAUGE.
You can get more information here
<http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html> about the
definition of both DS types, but basically COUNTER Data Sources are
affected if the value in question overflows or starts again from zero,
something that it could happen if the machine has been rebooted very
quickly (perhaps because it's a virtualized system and it takes a very
short time to reboot and start Monitorix again).
Finally and answering your second question, you can get the script
called
<
http://oss.oetiker.ch/rrdtool/pub/contrib/removespikes-20080226-mkn.tar.gz
to remove that huge spike that scales your graph so high that prevents
you to see your current daily detail. Also, such spike will affect
weekly, monthly and yearly views, so indeed, it's very recommended to
remove it as soon as possible.
The script is pretty simple, so you shouldn't have any problem with it.
Regards.
---
Jordi Sanfeliu
FIBRANET Network Services Provider
http://www.fibranet.cat

Post by Anthony Stump
I'm new to mailing lists so if I'm not doing this right let me know.
Yesterday night I discovered this awesome tool called Monitorix. I got
it all installed and it was working fine until this morning, when
reviewing the Fast Ethernet eth0 stats it shows a rather impossible
spike to 70 P bits/second transmission rate.
I was wondering, first, what causes this - second, is there any way to
either (A) remove that reading so I can see accurate stats (as the 70
P bit/sec reading makes it impossible to see any other readings, and
changes the average bits/second reading to a rather insane
265555577495 Kbits/second, or (B), just reset that graph all together
since it's useless now.
Screenshot here: http://postimg.org/image/n26bsnckv/ [1]
--
Thank you,
Anthony Stump
------
[1] http://postimg.org/image/n26bsnckv/

------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
Monitorix-general mailing list
https://lists.sourceforge.net/lists/listinfo/monitorix-general

[Jordi Sanfeliu]

Munin? I think you are confused with the monitoring tools.
This is Monitorix, not Munin.

Regarding the 'removespikes' script, you should read the README file
that comes with it, and run it using the file '/var/lib/monitorix/net.rrd'.

Regards.

Post by Anthony Stump
Where are the rrd's I need to modify? On the documentation it says that
the file should be net.rrd but I have several "interface eth0" rrds in
/var/lib/munin/localdomain/localhost.localdomain-if_*
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-collisions-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-txdrop-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-rcvd-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_eth0-down-d.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-rxdrop-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_eth0-up-d.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-trans-c.rrd
I have tried removing spikes from all of them and none of them find
spikes. When in analyze mode, they are all empty.

[Anthony Stump]

I was able to run it ( sudo ./removespikes.pl /var/lib/monitorix/net.rrd )
but it did not find any spikes at all in the net.rrd file. I used some of
the debug / more verbose options shown in the readme but it basically
stated no spikes found.

I notice it stated there was supposedly a way to specify thresholds for
"cutting" the spikes - but it didn't really say the syntax for it.

Is there a way to just pull up this rrd file and edit it to manually remove
the spike because this script isn't doing it for me right now :-S

Post by Jordi Sanfeliu
Munin? I think you are confused with the monitoring tools.
This is Monitorix, not Munin.
Regarding the 'removespikes' script, you should read the README file
that comes with it, and run it using the file '/var/lib/monitorix/net.rrd'.
Regards.

Post by Anthony Stump
Where are the rrd's I need to modify? On the documentation it says that
the file should be net.rrd but I have several "interface eth0" rrds in
/var/lib/munin/localdomain/localhost.localdomain-if_*

/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-collisions-c.rrd

Post by Anthony Stump
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-txdrop-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-rcvd-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_eth0-down-d.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-rxdrop-c.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_eth0-up-d.rrd
/var/lib/munin/localdomain/localhost.localdomain-if_err_eth0-trans-c.rrd
I have tried removing spikes from all of them and none of them find
spikes. When in analyze mode, they are all empty.

--
Jordi Sanfeliu
FIBRANET Network Services Provider
http://www.fibranet.cat
------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
Monitorix-general mailing list
https://lists.sourceforge.net/lists/listinfo/monitorix-general

[Jordi Sanfeliu]

Anthony,

You should be able to chop all data larger than a N value. For example,
if you have an abnormally huge peak in your network graph, you might
want to run the script to chop all values that are larger than 100MB.

Just like this:

$ ./removespikes.pl -d -t 100000000 net.rrd

After that, your 'net.rrd' file has been rewritten and now all the peaks
larger than 100MB are removed.

Another option is just remove 'net.rrd' and restart Monitorix. A new
'net.rrd' will be created automatically.

Regards.

I was able to run it ( sudo ./removespikes.pl <http://removespikes.pl>
/var/lib/monitorix/net.rrd ) but it did not find any spikes at all in
the net.rrd file. I used some of the debug / more verbose options shown
in the readme but it basically stated no spikes found.
I notice it stated there was supposedly a way to specify thresholds for
"cutting" the spikes - but it didn't really say the syntax for it.
Is there a way to just pull up this rrd file and edit it to manually
remove the spike because this script isn't doing it for me right now :-S

[Anthony Stump]

So it's still not finding any spikes above 100000000 - I'm wondering if
it's reading the wrong file? Is there any possibility I tried the
netstat.rrd file as well and it's not finding any spikes there. I did a
drive search for *.rrd and the only other folder I saw RRDs in was the
munin folder - is there any possibilty these rrds could be stored anywhere
else? I can wipe the net.rrd file - but just don't know if I can't find any
spikes in it if it's really going to do any good if it's not even the valid
one.

Post by Jordi Sanfeliu
Anthony,
You should be able to chop all data larger than a N value. For example,
if you have an abnormally huge peak in your network graph, you might
want to run the script to chop all values that are larger than 100MB.
$ ./removespikes.pl -d -t 100000000 net.rrd
After that, your 'net.rrd' file has been rewritten and now all the peaks
larger than 100MB are removed.
Another option is just remove 'net.rrd' and restart Monitorix. A new
'net.rrd' will be created automatically.
Regards.

I was able to run it ( sudo ./removespikes.pl <http://removespikes.pl>
/var/lib/monitorix/net.rrd ) but it did not find any spikes at all in
the net.rrd file. I used some of the debug / more verbose options shown
in the readme but it basically stated no spikes found.
I notice it stated there was supposedly a way to specify thresholds for
"cutting" the spikes - but it didn't really say the syntax for it.
Is there a way to just pull up this rrd file and edit it to manually
remove the spike because this script isn't doing it for me right now :-S

--
Jordi Sanfeliu
FIBRANET Network Services Provider
http://www.fibranet.cat
------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
Monitorix-general mailing list
https://lists.sourceforge.net/lists/listinfo/monitorix-general

[Jordi Sanfeliu]

Anthony,

The peak is in the 'net.rrd' file not in 'netstat.rrd'.

Please check in all time frames (day, week, month and year) to see if
the peak still appears in the graph. That will confirm that the
'net.rrd' still has it. If so, use the script as I said you in my last
email.

The Monitorix .rrd files are by default located in '/var/lib/monitorix'.
Check also the option 'base_lib' in '/etc/monitorix/monitorix.conf' to
confirm this.

Regards.

Post by Anthony Stump
So it's still not finding any spikes above 100000000 - I'm wondering if
it's reading the wrong file? Is there any possibility I tried the
netstat.rrd file as well and it's not finding any spikes there. I did a
drive search for *.rrd and the only other folder I saw RRDs in was the
munin folder - is there any possibilty these rrds could be stored
anywhere else? I can wipe the net.rrd file - but just don't know if I
can't find any spikes in it if it's really going to do any good if it's
not even the valid one.
Anthony,
You should be able to chop all data larger than a N value. For example,
if you have an abnormally huge peak in your network graph, you might
want to run the script to chop all values that are larger than 100MB.
$ ./removespikes.pl <http://removespikes.pl> -d -t 100000000 net.rrd
After that, your 'net.rrd' file has been rewritten and now all the peaks
larger than 100MB are removed.
Another option is just remove 'net.rrd' and restart Monitorix. A new
'net.rrd' will be created automatically.
Regards.

[Anthony Stump]

Screenshots:

No spikes found @ 10MB limit:
http://tinypic.com/r/xqjbwj/8

10 PB spike Still there:
http://tinypic.com/r/j7vvck/8

I'm having no luck. Even set the threshold to 10000.
Anthony,
The peak is in the 'net.rrd' file not in 'netstat.rrd'.
Please check in all time frames (day, week, month and year) to see if
the peak still appears in the graph. That will confirm that the
'net.rrd' still has it. If so, use the script as I said you in my last
email.
The Monitorix .rrd files are by default located in '/var/lib/monitorix'.
Check also the option 'base_lib' in '/etc/monitorix/monitorix.conf' to
confirm this.
Regards.

Post by Anthony Stump
So it's still not finding any spikes above 100000000 - I'm wondering if
it's reading the wrong file? Is there any possibility I tried the
netstat.rrd file as well and it's not finding any spikes there. I did a
drive search for *.rrd and the only other folder I saw RRDs in was the
munin folder - is there any possibilty these rrds could be stored
anywhere else? I can wipe the net.rrd file - but just don't know if I
can't find any spikes in it if it's really going to do any good if it's
not even the valid one.
Anthony,
You should be able to chop all data larger than a N value. For

example,

Post by Anthony Stump
if you have an abnormally huge peak in your network graph, you might
want to run the script to chop all values that are larger than 100MB.
$ ./removespikes.pl <http://removespikes.pl> -d -t 100000000 net.rrd
After that, your 'net.rrd' file has been rewritten and now all the

peaks

Post by Anthony Stump
larger than 100MB are removed.
Another option is just remove 'net.rrd' and restart Monitorix. A new
'net.rrd' will be created automatically.
Regards.

--
Jordi Sanfeliu
FIBRANET Network Services Provider
http://www.fibranet.cat
------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
Monitorix-general mailing list
https://lists.sourceforge.net/lists/listinfo/monitorix-general

[Jordi Sanfeliu]

Anthony,

I don't know why you are unable to remove that spike, anyway, and since
the next version will be avoid so unexpected spikes, I think that your
best option in that case is removing the 'net.rrd' file and restart
Monitorix.

Regards.

Post by Anthony Stump
http://tinypic.com/r/xqjbwj/8
http://tinypic.com/r/j7vvck/8

[Anthony Stump]

I accidentally wiped all the historical data by changing how many years
back data goes. On the other hand the spike is now gone!

One other thing I noticed... the IN-DNS chart seems to always increase the
value on a slope... I think that may be a bug as well.

Post by Jordi Sanfeliu
Anthony,
I don't know why you are unable to remove that spike, anyway, and since
the next version will be avoid so unexpected spikes, I think that your
best option in that case is removing the 'net.rrd' file and restart
Monitorix.
Regards.

Post by Anthony Stump
http://tinypic.com/r/xqjbwj/8
http://tinypic.com/r/j7vvck/8

--
Jordi Sanfeliu
FIBRANET Network Services Provider
http://www.fibranet.cat
------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck&#174;
Code Sight&#153; - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Monitorix-general mailing list
https://lists.sourceforge.net/lists/listinfo/monitorix-general

[Jordi Sanfeliu]

Anthony,

Post by Anthony Stump
I accidentally wiped all the historical data by changing how many years
back data goes. On the other hand the spike is now gone!

Ok.

Post by Anthony Stump
One other thing I noticed... the IN-DNS chart seems to always increase
the value on a slope... I think that may be a bug as well.

If so, then perhaps because your DNS is incrementing its number of
external requests.

You can check that with the 'iptables -nxvL' command and see how the
number of bytes in the accounting column of the DNS port is increasing.

Regards.