[Mondo-devel] mindi/mondo virtual memory

Discussion:

Michael White

2017-06-02 16:21:25 UTC

Hello all,

I'm seeing an issue on my RHEL 6.8 serves where something is acquiring virtual memory but doesn't appear to release it.
My virtual memory graph stair steps up each day (see attachment) and it appears its happing during when mondoarchive runs. Though I don't know how to prove or dis-prove.

Could mindi/mondo not be releasing virtual memory when done?

Anyone have any thoughts on how this can be figured out?

Mindi v3.0.2-r3578
Mondo v 3.2.2-r3578

Thanks.
Michael White

****************************************************************************************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it may
contain information that is confidential, privileged and exempt from disclosure under
applicable law. It is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of this
communication is strictly prohibited. If you have received this communication in error,
please promptly delete it and notify the sender of the delivery error by e-mail so that
the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
****************************************************************************************************

Bruno Cornec

2017-06-06 15:12:42 UTC

Permalink

Post by Michael White
I'm seeing an issue on my RHEL 6.8 serves where something is acquiring virtual memory but doesn't appear to release it.
My virtual memory graph stair steps up each day (see attachment) and it appears its happing during when mondoarchive runs. Though I don't know how to prove or dis-prove.
Could mindi/mondo not be releasing virtual memory when done?

--
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
Open Source Profession, WW Linux Community Lead http://github.com/bcornec
FLOSS projects: http://mondorescue.org http://project-builder.org
Musique ancienne? http://www.musique-ancienne.org http://www.medieval.org

Michael White

2017-06-06 16:50:51 UTC

Permalink

Thanks for the reply Bruno.
I'm not following you on what to look for with the command "ps auxww".

Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.
I'll be back in the office Wednesday and I'll gather the Midi/Mondo logs and attach.

Thanks again for the groups help.

Michael White

-----Original Message-----
From: Bruno Cornec [mailto:***@hpe.com]
Sent: Tuesday, June 06, 2017 10:13 AM
To: Mondo mailing list <mondo-***@lists.sourceforge.net>
Subject: Re: [Mondo-devel] mindi/mondo virtual memory

I don't think this is due to mondoarchive, except if the command doesn't end correctly. Check with a ps auxww if you have remaining processes called that way. If not, it could be a command launched by mondoarchive/mindi which doesn't end correctly itself. Again the ps would help, as well as the log from the archiving run.

Bruno.
--
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
Open Source Profession, WW Linux Community Lead http://github.com/bcornec
FLOSS projects: http://mondorescue.org http://project-builder.org
Musique ancienne? http://www.musique-ancienne.org http://www.medieval.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Mondo-devel mailing list
Mondo-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mondo-devel

****************************************************************************************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it may
contain information that is confidential, privileged and exempt from disclosure under
applicable law. It is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of this
communication is strictly prohibited. If you have received this communication in error,
please promptly delete it and notify the sender of the delivery error by e-mail so that
the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
****************************************************************************************************

Bruno Cornec

2017-06-06 16:56:21 UTC

Permalink

Post by Michael White
I'm not following you on what to look for with the command "ps auxww".

Well, I try to understand which process can be eating memory. That command could help.

Post by Michael White
Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.

Michael White

2017-06-09 18:08:23 UTC

Permalink

Bruno here are the logs you ask for as you can see in the graph when I don't run mondoarchive the virtual memory flattens out, as soon as I run it vmem starts stepping back up again.
I scanned through the logs but I don't see anything jumping out at me.

Thanks
Michael White

-----Original Message-----
From: Bruno Cornec [mailto:***@hpe.com]
Sent: Tuesday, June 06, 2017 11:56 AM
To: Mondo mailing list <mondo-***@lists.sourceforge.net>
Subject: Re: [Mondo-devel] mindi/mondo virtual memory

Post by Michael White
I'm not following you on what to look for with the command "ps auxww".

Well, I try to understand which process can be eating memory. That command could help.

Post by Michael White
Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.

How long is mondoarchive running for you ?
Of course, when it's running, it's consuming more memory, even if I try to do my best to avoid unnecessary consumption.
But when the command ends, memory should come back again. Is that the case ?

Bruno.
--
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
Open Source Profession, WW Linux Community Lead http://github.com/bcornec
FLOSS projects: http://mondorescue.org http://project-builder.org
Musique ancienne? http://www.musique-ancienne.org http://www.medieval.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Mondo-devel mailing list
Mondo-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mondo-devel

****************************************************************************************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it may
contain information that is confidential, privileged and exempt from disclosure under
applicable law. It is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of this
communication is strictly prohibited. If you have received this communication in error,
please promptly delete it and notify the sender of the delivery error by e-mail so that
the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
****************************************************************************************************

Michael White

2017-06-14 17:54:00 UTC

Permalink

Hi Bruno,

I'm still trying to figure out why something is holding on to virtual memory after the job finishes. As I noted in a previous post - if I stop mono archive from running virtual memory does not increase that day but when mondo archive starts again virtual memory bumps up another notch again.

I'm wondering could it possibly be mindi? Doesn't mindi use a ram disk to build the boot image? I've looked at mindi's log (attached) but I really don't see anything about memory allocation/deallocation.

Another odd thing is I have 8 Linux servers basically identical to each other that I run mondo archive on but only two production servers are showing this virtual memory issue.

Thanks
Michael White

-----Original Message-----
From: Bruno Cornec [mailto:***@hpe.com]
Sent: Tuesday, June 06, 2017 11:56 AM
To: Mondo mailing list <mondo-***@lists.sourceforge.net>
Subject: Re: [Mondo-devel] mindi/mondo virtual memory

Post by Michael White
I'm not following you on what to look for with the command "ps auxww".

Well, I try to understand which process can be eating memory. That command could help.

Post by Michael White
Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.

Rick Stevens

2017-06-14 21:13:42 UTC

Permalink

Post by Michael White
Hi Bruno,
I'm still trying to figure out why something is holding on to virtual memory after the job finishes. As I noted in a previous post - if I stop mono archive from running virtual memory does not increase that day but when mondo archive starts again virtual memory bumps up another notch again.
I'm wondering could it possibly be mindi? Doesn't mindi use a ram disk to build the boot image? I've looked at mindi's log (attached) but I really don't see anything about memory allocation/deallocation.
Another odd thing is I have 8 Linux servers basically identical to each other that I run mondo archive on but only two production servers are showing this virtual memory issue.

Suggestion from left field: try running mondoarchive under memusage:

memusage mondoarchive <all your wonderful options>

or perhaps using operf/opreport instead of memusage. It might even be
better to run operf in system-wide mode to catch everything going on.
See "man operf" for details.

memusage is part of the glibc-utils RPM on Fedora/CentOS/Red Hat.

operf/opreport is part of the oprofile RPM on Fedora/CentOS/Red Hat.
Can't speak about other distros, but they're probably similarly named.

Post by Michael White
-----Original Message-----
Sent: Tuesday, June 06, 2017 11:56 AM
Subject: Re: [Mondo-devel] mindi/mondo virtual memory

Post by Michael White
I'm not following you on what to look for with the command "ps auxww".

Well, I try to understand which process can be eating memory. That command could help.

Post by Michael White
Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.

How long is mondoarchive running for you ?
Of course, when it's running, it's consuming more memory, even if I try to do my best to avoid unnecessary consumption.
But when the command ends, memory should come back again. Is that the case ?
Bruno.
--
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
Open Source Profession, WW Linux Community Lead http://github.com/bcornec
FLOSS projects: http://mondorescue.org http://project-builder.org
Musique ancienne? http://www.musique-ancienne.org http://www.medieval.org
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Mondo-devel mailing list
https://lists.sourceforge.net/lists/listinfo/mondo-devel
****************************************************************************************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it may
contain information that is confidential, privileged and exempt from disclosure under
applicable law. It is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of this
communication is strictly prohibited. If you have received this communication in error,
please promptly delete it and notify the sender of the delivery error by e-mail so that
the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
****************************************************************************************************
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
https://lists.sourceforge.net/lists/listinfo/mondo-devel

Michael White

2017-06-15 11:19:48 UTC

Permalink

Thanks Rick, doesn't look like I have those available but I'll look into them.

Michael White

-----Original Message-----
From: Rick Stevens [mailto:***@alldigital.com]
Sent: Wednesday, June 14, 2017 4:14 PM
To: mondo-***@lists.sourceforge.net
Subject: Re: [Mondo-devel] mindi/mondo virtual memory

Suggestion from left field: try running mondoarchive under memusage:

memusage mondoarchive <all your wonderful options>

or perhaps using operf/opreport instead of memusage. It might even be better to run operf in system-wide mode to catch everything going on.
See "man operf" for details.

memusage is part of the glibc-utils RPM on Fedora/CentOS/Red Hat.

operf/opreport is part of the oprofile RPM on Fedora/CentOS/Red Hat.
Can't speak about other distros, but they're probably similarly named.

Post by Michael White
-----Original Message-----
Sent: Tuesday, June 06, 2017 11:56 AM
Subject: Re: [Mondo-devel] mindi/mondo virtual memory

Post by Michael White
I'm not following you on what to look for with the command "ps auxww".

Well, I try to understand which process can be eating memory. That command could help.

Post by Michael White
Over the weekend I stopped Mondoarchive from running and the memory graphs flatted out until MondoArchive ran Monday night and the graph stepped up like the example I sent.

How long is mondoarchive running for you ?
Of course, when it's running, it's consuming more memory, even if I try to do my best to avoid unnecessary consumption.
But when the command ends, memory should come back again. Is that the case ?
Bruno.
--
HPE EMEA EG FLOSS Technology Strategist http://www.hpe.com/engage/opensource
Open Source Profession, WW Linux Community Lead http://github.com/bcornec
FLOSS projects: http://mondorescue.org http://project-builder.org
Musique ancienne? http://www.musique-ancienne.org http://www.medieval.org
----------------------------------------------------------------------
-------- Check out the vibrant tech community on one of the world's
most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
https://lists.sourceforge.net/lists/listinfo/mondo-devel
**********************************************************************
******************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it
may contain information that is confidential, privileged and exempt
from disclosure under applicable law. It is intended solely for the
use of the individual or entity to which it is addressed. If you are
not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of
this communication is strictly prohibited. If you have received this
communication in error, please promptly delete it and notify the sender of the delivery error by e-mail so that the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
**********************************************************************
******************************
----------------------------------------------------------------------
-------- Check out the vibrant tech community on one of the world's
most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Mondo-devel mailing list
https://lists.sourceforge.net/lists/listinfo/mondo-devel

--
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital ***@alldigital.com -
- AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 -
- -
- Brain: The organ with which we think that we think. -
----------------------------------------------------------------------

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Mondo-devel mailing list
Mondo-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mondo-devel

****************************************************************************************************
CONFIDENTIALITY NOTICE
This communication and any files or attachments transmitted with it may
contain information that is confidential, privileged and exempt from disclosure under
applicable law. It is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient, you are hereby notified that any use,
dissemination, copying, or action taken in reliance on the contents of this
communication is strictly prohibited. If you have received this communication in error,
please promptly delete it and notify the sender of the delivery error by e-mail so that
the appropriate action may be taken to avoid troubling you further.
Thank you for your cooperation. V2
****************************************************************************************************