Re: [vserver] Very serious hashify mysql/innodb conflict causes major data loss

From: John Alberts <john.m.alberts_at_gmail.com>
Date: Tue 09 Jun 2009 - 13:43:50 BST
Message-ID: <a23b6f900906090543t21e51358ud965683a489d58d5@mail.gmail.com>

Ah, I didn't realize that. Yes, I was using a separate directory for
the db files located at /dbfiles. So I guess this is why I ran into
the problem.

Thanks for clarifying that Daniel.

John

On Tue, Jun 9, 2009 at 7:38 AM, Daniel Hokka Zakrisson<daniel@hozac.com> wrote:
> John Alberts wrote:
>> I just wanted to mention I had the same issue with mysql/innodb and
>> hashify a couple of months ago.  As Herbert suggested, just don't
>> hashify the mysql db directory.  I stopped using vhashify altogether
>> and this fixed my problems; however, the default when you run vhashify
>> is that it just does everything.  Unless you know to avoid running it
>> on the mysql directory, your going to have problems.
>
> No, the default exclude list excludes all of /var. So unless you're storing
> the databases elsewhere, or use a custom exclude list where /var is not
> present, they will not get hashified.
>
> Daniel
>
>> Regards,
>> John
>>
>>
>> On Tue, Jun 9, 2009 at 7:08 AM, Herbert Poetzl<herbert@13thfloor.at> wrote:
>>> On Tue, Jun 09, 2009 at 07:41:28AM -0400, John A. Sullivan III wrote:
>>>> Hello, all.  We have been struggling for several weeks with an issue of
>>>> severe data loss on our zimbra email vserver.  We have narrowed it down
>>>> with considerable certainty to a conflict between mysql/innodb and
>>>> hashify.  We are running 2.6.28.7-vs2.3.0.36.7, i.e., a custom patched
>>>> 2.6.28.7 kernel and util-vserver-0.30.216-1.pre2793.el5.centos.  We can
>>>> reproduce the issue at will.
>>>
>>> updating to a recent version is advised ...
>>>
>>>> Zimbra is running its own instance of mysql with innodb support version
>>>> 5.0.67.  Zimbra runs a zimbra database and then many other mysql
>>>> instances named mboxgroupX where X is a one or two digit number.
>>>
>>>> Whenever we run vserver <zimbra server name> hashify, we see odd
>>>> behaviors:
>>>>
>>>>       * The mysql and zimbra database directories are time stamped with
>>>>         the time hashify ran as if mysql was restarted but there is no
>>>>         record of mysql being restarted.
>>>>       * MOST IMPORTANTLY, the innodb log stops recording but it appears
>>>>         that mysql is still functioning.  Mail is displayed, events
>>>>         (e.g., calendar, tasks) are recorded and are fully functional
>>>
>>> there is no pint in hashifying mysql databases,
>>> so I would suggest to exclude those dirs ...
>>>
>>>> The disaster occurs when mysql is restarted, e.g., when restarting the
>>>> zimbra service.  Since the innodb log has recorded no data since hashify
>>>> ran, mysql thinks the system has crashed and backs out all data since
>>>> the last hashify! At least, I believe that's what is happening in my
>>>> MySQL ignorance.
>>>
>>> IMHO hashify should not change the time stamps of
>>> files, but it's okay to update directory timestamps
>>> if files have been unified there ...
>>>
>>>> Obviously, this kind of data loss is intolerable.  I'd rather solve the
>>>> problem than simply forgo hashify on our zimbra servers.  Any ideas
>>>> about what could cause this behavior and how to troubleshoot it?
>>>
>>> well, my guess would be that mysql either uses
>>> the timestamps which get changed when an otherwise
>>> harmless file is unified (in that dir) ... the
>>> proper solution IMHO is to exclude that dir from
>>> hashification ...
>>>
>>> HTH,
>>> Herbert
>>>
>>>> Thanks
>>>> - John
>>>> --
>>>> John A. Sullivan III
>>>> Open Source Development Corporation
>>>> +1 207-985-7880
>>>> jsullivan@opensourcedevel.com
>>>>
>>>> http://www.spiritualoutreach.com
>>>> Making Christianity intelligible to secular society
>>>
>>
>>
>>
>> --
>> John Alberts
>>
>
>
> --
> Daniel Hokka Zakrisson
>

-- 
John Alberts
Received on Tue Jun 9 13:44:10 2009
[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Tue 09 Jun 2009 - 13:44:11 BST by hypermail 2.1.8