Re: [vserver] Very serious hashify mysql/innodb conflict causes major data loss

From: Herbert Poetzl <herbert_at_13thfloor.at>
Date: Tue 09 Jun 2009 - 13:08:22 BST
Message-ID: <20090609120822.GB23836@MAIL.13thfloor.at>

On Tue, Jun 09, 2009 at 07:41:28AM -0400, John A. Sullivan III wrote:
> Hello, all. We have been struggling for several weeks with an issue of
> severe data loss on our zimbra email vserver. We have narrowed it down
> with considerable certainty to a conflict between mysql/innodb and
> hashify. We are running 2.6.28.7-vs2.3.0.36.7, i.e., a custom patched
> 2.6.28.7 kernel and util-vserver-0.30.216-1.pre2793.el5.centos. We can
> reproduce the issue at will.

updating to a recent version is advised ...

> Zimbra is running its own instance of mysql with innodb support version
> 5.0.67. Zimbra runs a zimbra database and then many other mysql
> instances named mboxgroupX where X is a one or two digit number.

> Whenever we run vserver <zimbra server name> hashify, we see odd
> behaviors:
>
> * The mysql and zimbra database directories are time stamped with
> the time hashify ran as if mysql was restarted but there is no
> record of mysql being restarted.
> * MOST IMPORTANTLY, the innodb log stops recording but it appears
> that mysql is still functioning. Mail is displayed, events
> (e.g., calendar, tasks) are recorded and are fully functional

there is no pint in hashifying mysql databases,
so I would suggest to exclude those dirs ...

> The disaster occurs when mysql is restarted, e.g., when restarting the
> zimbra service. Since the innodb log has recorded no data since hashify
> ran, mysql thinks the system has crashed and backs out all data since
> the last hashify! At least, I believe that's what is happening in my
> MySQL ignorance.

IMHO hashify should not change the time stamps of
files, but it's okay to update directory timestamps
if files have been unified there ...

> Obviously, this kind of data loss is intolerable. I'd rather solve the
> problem than simply forgo hashify on our zimbra servers. Any ideas
> about what could cause this behavior and how to troubleshoot it?

well, my guess would be that mysql either uses
the timestamps which get changed when an otherwise
harmless file is unified (in that dir) ... the
proper solution IMHO is to exclude that dir from
hashification ...

HTH,
Herbert

> Thanks
> - John
> --
> John A. Sullivan III
> Open Source Development Corporation
> +1 207-985-7880
> jsullivan@opensourcedevel.com
>
> http://www.spiritualoutreach.com
> Making Christianity intelligible to secular society
Received on Tue Jun 9 13:08:34 2009

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Tue 09 Jun 2009 - 13:08:35 BST by hypermail 2.1.8