[Mondo-devel] Mondo restore fails to instruct fdisk on RHEL5.4

[Dag Wieers]

Hi,

We are trying to see whether Mondo Rescue is a fit to our environment.
When trying to restore a VMware guest with a DRBD setup we noticed a few
problems:

- During the restore or interactive restore we get the message:

Gap on /dev/hda between 5 and 8. 2MB overallocated on /dev/hda

- We get errors about /dev/drbd, for some reason Mondo assumes this is a
device, but it is not. It does not exist. On the original system
/dev/drbd is a functional directory with a by-disk and by-res
subdirectory with symlinks to the drbd devices (eg
/dev/drbd/by-disk/hda6 -> ../../drbd1)

We do have a /dev/drbd1 and this seems to be correct, we simply don't
know why Mondo even uses /dev/drbd and why it assumes it is a device
and why it doesn't check for the existence before even doing what it
does.

Error getting size of /dev/drbd: No such file or directory
Failed to open /dev/drbd for reading: No such file or directory
Failed to get harddisk geometry, using old mode
/dev/drbd --> -1 or -1 --> -1
/dev/drbd does not exist.

- We noticed another problem if we try to continue (removing even the
restore of drbd). Mondo continues partitioning the disk /dev/hda, but
we don't see the partitioning table. In fact the attached
mondofdisk.log clearly shows some errors.

I hope the attached logfiles can help resolve the problems we see. Let me
know if I can assist or help by doing any other tests or modifying some
code.

Thanks in advance,

[Bruno Cornec]

Hello,

Dag Wieers said on Thu, May 20, 2010 at 12:45:14PM +0200:

Whao ! You mean "the" Dag Wieers of RPMforge ?
Now, I'm stressed ;-) And I want first to thank you for all the work you
do around Open Source and Linux. I'm using packages from your repo, and
will do what I can as a return to help you (as well as mrepo).

Post by Dag Wieers
We are trying to see whether Mondo Rescue is a fit to our
environment. When trying to restore a VMware guest with a DRBD setup

First, let me say I'm not surprised you found an issue with DRDB as it
was never tested with it (not that I know of).

Would you also be able to provide to me the /var/log/mondoarchive.log
file from the original VM please ? I'd like to see how the mountlist
analysis went in it.

Post by Dag Wieers
Gap on /dev/hda between 5 and 8. 2MB overallocated on /dev/hda

Indeed you have no hda6/7. Which confuses the restore. The
overallocation seems to become a more and more common issue these days,
and I'm already tracking that here:
http://trac.mondorescue.org/ticket/401

Post by Dag Wieers
- We get errors about /dev/drbd, for some reason Mondo assumes this is a
device, but it is not. It does not exist. On the original system
/dev/drbd is a functional directory with a by-disk and by-res
subdirectory with symlinks to the drbd devices (eg
/dev/drbd/by-disk/hda6 -> ../../drbd1)

Ah I now see where the problem comes from :-( So these missing hda6 are
indeed coming from DRDB. So I'll need to add a specific treatment for
that in mindi at least. So I really need your archive log file then.

Post by Dag Wieers
We do have a /dev/drbd1 and this seems to be correct, we simply don't
know why Mondo even uses /dev/drbd and why it assumes it is a device
and why it doesn't check for the existence before even doing what it
does.

Probably because nobody else tried that before ;-)

Post by Dag Wieers
- We noticed another problem if we try to continue (removing even the
restore of drbd). Mondo continues partitioning the disk /dev/hda, but
we don't see the partitioning table. In fact the attached
mondofdisk.log clearly shows some errors.

Yep. This is the cause of the missing entries for hda. Should be better
once we solve the DRDB handling.

Post by Dag Wieers
Let me know if I can assist or help by doing any other tests or
modifying some code.

What I plan to do is provide you with beta version of 2.2.9.4 with fixes
added to support DRDB correctly. Probably an iterative process. So it
may take a bit of time on your side to relaunch with the new version
,gather the logs if it's still failling and sending that back to us.

I'll probably not have enough time next week to try to reproduce a
similar setup to ease the debug as I'll be traveling to China (Shanghai
- BTW if some of you want to meet with me there, happy to do so ;-). But
I should be able to work on the logs and already improve the DRDB
detection and process from that. Give me your timeline for this.

Post by Dag Wieers
Thanks in advance,

Again thanks should go to you and hopefully we'll solve that pb, and add
a new feature to the tool !

Best regards,
Bruno.

[Dag Wieers]

Post by Bruno Cornec
Whao ! You mean "the" Dag Wieers of RPMforge ?
Now, I'm stressed ;-) And I want first to thank you for all the work you
do around Open Source and Linux. I'm using packages from your repo, and
will do what I can as a return to help you (as well as mrepo).

In fact, I would like to add mondo to RPMforge as well. I looked at it a
few times before, but now I may have the time to dive into the complexity
at last !

Post by Bruno Cornec

Post by Dag Wieers
We are trying to see whether Mondo Rescue is a fit to our
environment. When trying to restore a VMware guest with a DRBD setup

First, let me say I'm not surprised you found an issue with DRDB as it
was never tested with it (not that I know of).
Would you also be able to provide to me the /var/log/mondoarchive.log
file from the original VM please ? I'd like to see how the mountlist
analysis went in it.

The mondoarchive.log is attached.

Post by Bruno Cornec

Post by Dag Wieers
We do have a /dev/drbd1 and this seems to be correct, we simply don't
know why Mondo even uses /dev/drbd and why it assumes it is a device
and why it doesn't check for the existence before even doing what it
does.

Probably because nobody else tried that before ;-)

I always try to avoid being an early adopter, I must have failed here ;-)

Post by Bruno Cornec

Post by Dag Wieers
Let me know if I can assist or help by doing any other tests or
modifying some code.

What I plan to do is provide you with beta version of 2.2.9.4 with fixes
added to support DRDB correctly. Probably an iterative process. So it
may take a bit of time on your side to relaunch with the new version
,gather the logs if it's still failling and sending that back to us.

If it depends on me, we can progress very quickly. And with sufficient
pointers I may be able to debug and fix it myself. I haven't looked at the
source-code yet, but with the stuff you now know I am sure you can tell me
what I should be looking at.

Post by Bruno Cornec
I'll probably not have enough time next week to try to reproduce a
similar setup to ease the debug as I'll be traveling to China (Shanghai
- BTW if some of you want to meet with me there, happy to do so ;-). But
I should be able to work on the logs and already improve the DRDB
detection and process from that. Give me your timeline for this.

I am available today until 19h CEST and starting next week from tuesday. I
can give priority to fixing this. But sadly have no proper Internet
connection (only Mobistar Edge in the best case).

Post by Bruno Cornec

Post by Dag Wieers
Thanks in advance,

Again thanks should go to you and hopefully we'll solve that pb, and add
a new feature to the tool !

I hope so :-)

Thanks for the prompt acknowledgment !

[Bruno Cornec]

Post by Dag Wieers
In fact, I would like to add mondo to RPMforge as well. I looked at
it a few times before, but now I may have the time to dive into the
complexity at last !

Not that complex (well of course from my point of view ;-)
I provide packages for the distros you're also serving (el3/4/5, fc6-12).
Let me know what I can do to help for this.

Post by Dag Wieers

Post by Bruno Cornec
Would you also be able to provide to me the /var/log/mondoarchive.log
file from the original VM please ? I'd like to see how the mountlist
analysis went in it.

The mondoarchive.log is attached.

Thanks. So now I see the mountlist analysis issue. I need to read a bit
on how DRDB is working, and which tool to call to find the missing parts
in the current analysis.

Post by Dag Wieers

Post by Bruno Cornec
Probably because nobody else tried that before ;-)

I always try to avoid being an early adopter, I must have failed here ;-)

He he ! Nobody can always win ;-)

Post by Dag Wieers
If it depends on me, we can progress very quickly. And with
sufficient pointers I may be able to debug and fix it myself. I
haven't looked at the source-code yet, but with the stuff you now
know I am sure you can tell me what I should be looking at.

You may be surprised by the code. But as most of the work to do will be
in mindi, it should be more clean and easy, at least for the archiving
part. In fact currently, what we get is:

Your mountlist will look like this:
Analyzing LVM...
DEVICE MOUNTPOINT FORMAT SIZE (MB)
LABEL/UUID
/dev/hda8 lvm lvm 67836
/dev/hda5 lvm lvm 82764
/dev/hda2 lvm lvm 30718
/dev/drbd1 lvm lvm 82761
[...]
(You can get that info by running mindi --makemountlist /tmp/mnt.lis)

So mindi should detect that a DRBD device is used, analyze it (probably
using drbdadm, drbdmeta, /etc/drbd.conf, and replace the line:
/dev/drbd1 lvm lvm 82761
by a line looking more or less like this:
/dev/hda6 drdb drdb xxxxx
/dev/hda7 drdb drdb yyyyy

then at restore time, mondorestore has to analyze this and launch the
appropriate drbd tools to recreate the partitions correctly and activate
drbd on them.

At least that's what I understand from a quick look at the DRBD Web
site. Help here is welcome of course. I really need to understand what
needs to be captured in order for the restore to occur successfully.

Post by Dag Wieers
I am available today until 19h CEST and starting next week from
tuesday. I can give priority to fixing this. But sadly have no
proper Internet connection (only Mobistar Edge in the best case).

Ok, will try to look at the doc more in details and come up with
something to test. But can't promise anything in term of planning :-(
Especially, it seems more tricky when DRBD is used in combination with
LVM.

Post by Dag Wieers
Thanks for the prompt acknowledgment !

No pb, my pleasure !

Bruno.

[Dag Wieers]

Post by Bruno Cornec

Post by Dag Wieers
In fact, I would like to add mondo to RPMforge as well. I looked at
it a few times before, but now I may have the time to dive into the
complexity at last !

Not that complex (well of course from my point of view ;-)
I provide packages for the distros you're also serving (el3/4/5, fc6-12).
Let me know what I can do to help for this.

In fact I added Mindi/Mondo already to RPMforge in 2007, but it was not
updated anymore. I updated them today so newer packages will be available
in RPMforge starting from tommorow.

Including projectbuilder and rpmbootstrap.

I also noticed that netperf 2.4.5 is out (and has been for a year ;-))

Kind regards,

[Bill R. Williams]

A rather trivial side note ...

Post by Dag Wieers

Post by Bruno Cornec
Would you also be able to provide to me the /var/log/mondoarchive.log
file from the original VM please ? I'd like to see how the mountlist
analysis went in it.

The mondoarchive.log is attached.

Looking at your log you have the ugly situation caused by:
mount --bind /proc /var/named/chroot/proc
which means you are actually trying to backup the ever-elusive and
transient /proc filesystem.

You can eliminate all those worthless errors by excluding
/var/named/chroot/proc from the mondoarchive run.
It will also reduce the size of your logfile. :-)

For the brief overview on this, have a look at:
http://trac.mondorescue.org/wiki/CustomerExperience

[Dag Wieers]

Post by Bill R. Williams
A rather trivial side note ...

Post by Dag Wieers

Post by Bruno Cornec
Would you also be able to provide to me the /var/log/mondoarchive.log
file from the original VM please ? I'd like to see how the mountlist
analysis went in it.

The mondoarchive.log is attached.

mount --bind /proc /var/named/chroot/proc
which means you are actually trying to backup the ever-elusive and
transient /proc filesystem.
You can eliminate all those worthless errors by excluding
/var/named/chroot/proc from the mondoarchive run.
It will also reduce the size of your logfile. :-)
http://trac.mondorescue.org/wiki/CustomerExperience

I wonder why certain filesystem types are not excluded by default. Those
would include 'sysfs', 'proc', 'usbfs', 'devpts', 'none'. It makes more
sense to exclude those types that no one ever should back up. No matter
where it is mounted...