myqsl cluster error code 2310

   我启动mysql cluster 并导入16G 的数据。数据正常导入，启动备份能正常恢复。完全导入之后 占用40% 的内存。但是我重启mgm data 节点时 在mgm 上使用命令 ndn_mgm>all reprot mem 当个数据量到12% 的时候报如下错误。

   如果我导入的数据 占用内存小于12% 的时候重启mgm 、data 节点就能正常启动。

  下面是错误日志：

mgm 节点错误日志：

010-06-24 11:46:39 [MgmSrvr] INFO     -- Node 3: DICT: activate index 42 done (sys/def/9/my_index)
2010-06-24 11:46:39 [MgmSrvr] INFO     -- Node 3: DICT: activate index 43 done (sys/def/8/my_no)
2010-06-24 11:46:39 [MgmSrvr] INFO     -- Node 3: DICT: activate index 44 done (sys/def/8/local_id)
2010-06-24 11:46:39 [MgmSrvr] INFO     -- Node 3: DICT: activate index 45 done (sys/def/7/my_buy)
2010-06-24 11:46:39 [MgmSrvr] INFO     -- Node 3: DICT: activate index 46 done (sys/def/10/stime)
2010-06-24 11:46:39 [MgmSrvr] INFO     -- Node 3: Node: 3 StartLog: [GCI Keep: 151 LastCompleted: 212 NewestRestorable: 212]
2010-06-24 11:46:39 [MgmSrvr] INFO     -- Node 3: Node: 4 StartLog: [GCI Keep: 151 LastCompleted: 212 NewestRestorable: 212]
2010-06-24 11:46:39 [MgmSrvr] INFO     -- Node 3: Node: 5 StartLog: [GCI Keep: 151 LastCompleted: 212 NewestRestorable: 212]
2010-06-24 11:46:39 [MgmSrvr] INFO     -- Node 3: Node: 6 StartLog: [GCI Keep: 151 LastCompleted: 212 NewestRestorable: 212]
2010-06-24 11:47:36 [MgmSrvr] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
2010-06-24 11:47:36 [MgmSrvr] ALERT    -- Node 6: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
2010-06-24 11:47:36 [MgmSrvr] ALERT    -- Node 5: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
2010-06-24 11:47:36 [MgmSrvr] ALERT    -- Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
2010-06-24 11:47:36 [MgmSrvr] ALERT    -- Node 1: Node 4 Disconnected
2010-06-24 11:47:36 [MgmSrvr] ALERT    -- Node 1: Node 3 Disconnected
2010-06-24 11:47:36 [MgmSrvr] ALERT    -- Node 1: Node 5 Disconnected
2010-06-24 11:47:36 [MgmSrvr] ALERT    -- Node 1: Node 6 Disconnected

 

 

下面是 data 节点的错误日志：

[root@NDB1 var]# tail -f ndb_3_error.log
Message: Error while reading the REDO log (Ndbd file system inconsistency error, please report a bug)
Error: 2310
Error data: Error while reading REDO log. from 16462
part: 1 D=9, F=1 Mb=45 FP=1471 W1=8187 W2=7 : end of log wo/ having found last GCI gci: 212
Error object: DBLQH (Line: 16570) 0x0000000a
Program: ndbd
Pid: 27783
Trace: /mysql/var/ndb_3_trace.log.23
Version: mysql-5.1.35 ndb-7.0.7
***EOM**

以及 data 节点的日志：

RESTORE table: 8 571493 rows applied
records: 16 len: 16c left: 280
records: 16 len: 170 left: 348
records: 16 len: 174 left: 308
RESTORE table: 8 573665 rows applied
RESTORE table: 9 2745486 rows applied
RESTORE table: 9 2748543 rows applied
records: 16 len: 100 left: 232
RESTORE table: 10 1196712 rows applied
RESTORE table: 10 1200062 rows applied
RESTORE table: 11 1663590 rows applied
RESTORE table: 11 2089960 rows applied
RESTORE table: 12 572936 rows applied
RESTORE table: 12 571474 rows applied
2010-06-24 11:47:36 [ndbd] INFO     -- Error while reading REDO log. from 16462
part: 1 D=9, F=1 Mb=45 FP=1471 W1=8187 W2=7 : end of log wo/ having found last GCI gci: 212
2010-06-24 11:47:36 [ndbd] INFO     -- DBLQH (Line: 16570) 0x0000000a
2010-06-24 11:47:36 [ndbd] INFO     -- Error handler startup shutting down system
2010-06-24 11:47:36 [ndbd] INFO     -- Error handler shutdown completed - exiting
2010-06-24 11:47:36 [ndbd] INFO     -- Angel received ndbd startup failure count 1.
2010-06-24 11:47:37 [ndbd] ALERT    -- Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.

 

 

希望大家能给予指导。。。。

在这里谢谢大家