Mysql error 1236 from master when reading data from binary log

Question

I have 2 MySql with a master/slave configuration and the replication is failing. The MySql Master crashed and a new register in the mysql-bin.index was created. I deleted this new register because this file was not existed in the file system. Then the MySql Master restarted successfully.

Now, I have the next error in the slave:

mysql> show slave status \G
*************************** 1. row ***************************
           Slave_IO_State: 
              Master_Host: 10.64.253.99
              Master_User: replication
              Master_Port: 3306
            Connect_Retry: 60
          Master_Log_File: mysql-bin.001050
      Read_Master_Log_Pos: 54868051
           Relay_Log_File: mysqld-relay-bin.000001
            Relay_Log_Pos: 4
    Relay_Master_Log_File: mysql-bin.001050
         Slave_IO_Running: No
        Slave_SQL_Running: Yes
          Replicate_Do_DB: 
      Replicate_Ignore_DB: 
       Replicate_Do_Table: 
   Replicate_Ignore_Table: 
  Replicate_Wild_Do_Table: 
Replicate_Wild_Ignore_Table: 
               Last_Errno: 0
               Last_Error: 
             Skip_Counter: 0
      Exec_Master_Log_Pos: 54868051
          Relay_Log_Space: 107
          Until_Condition: None
           Until_Log_File: 
            Until_Log_Pos: 0
       Master_SSL_Allowed: No
       Master_SSL_CA_File: 
       Master_SSL_CA_Path: 
          Master_SSL_Cert: 
        Master_SSL_Cipher: 
           Master_SSL_Key: 
    Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
            Last_IO_Errno: 1236
            Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file'
           Last_SQL_Errno: 0
           Last_SQL_Error: 
Replicate_Ignore_Server_Ids: 
         Master_Server_Id: 1

If I execute the "show master status" for view the mysql bin log file and the position:

mysql> show master status \G
*************************** 1. row ***************************
            File: mysql-bin.001050
        Position: 55586895
    Binlog_Do_DB: aaa
 Binlog_Ignore_DB: xxx,yyy,zzz,mysql

Then I set new config to the slave:

STOP SLAVE;
CHANGE MASTER TO 
    MASTER_HOST='10.64.253.99',
    MASTER_USER='slaveUser',
    MASTER_PASSWORD='12345',
    MASTER_LOG_FILE='mysql-bin.001050', 
    MASTER_LOG_POS=55586895;
START SLAVE;

And if I check slave status again I have the same error:

mysql> show slave status \G
*************************** 1. row ***************************
           Slave_IO_State: 
              Master_Host: 10.64.253.99
              Master_User: replication
              Master_Port: 3306
            Connect_Retry: 60
          Master_Log_File: mysql-bin.001050
      Read_Master_Log_Pos: 55586895
           Relay_Log_File: mysqld-relay-bin.000001
            Relay_Log_Pos: 4
    Relay_Master_Log_File: mysql-bin.001050
         Slave_IO_Running: No
        Slave_SQL_Running: Yes
          Replicate_Do_DB: 
      Replicate_Ignore_DB: 
       Replicate_Do_Table: 
   Replicate_Ignore_Table: 
  Replicate_Wild_Do_Table: 
Replicate_Wild_Ignore_Table: 
               Last_Errno: 0
               Last_Error: 
             Skip_Counter: 0
      Exec_Master_Log_Pos: 55586895
          Relay_Log_Space: 107
          Until_Condition: None
           Until_Log_File: 
            Until_Log_Pos: 0
       Master_SSL_Allowed: No
       Master_SSL_CA_File: 
       Master_SSL_CA_Path: 
          Master_SSL_Cert: 
        Master_SSL_Cipher: 
           Master_SSL_Key: 
    Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
            Last_IO_Errno: 1236
            Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file'
           Last_SQL_Errno: 0
           Last_SQL_Error: 
 Replicate_Ignore_Server_Ids: 
         Master_Server_Id: 1

I have checked in the MySql Master that the mysql-bin.001050 exist and is not empty.

This is a production environment then new data are inserted every minute. The position value can to change in minutes or seconds. I don't know if this is a problem.

The max_allowed_packet variable has the same value in the two MySql (16M).

Why can not the slave find the binary log file?

Croaker

Answer 1

You can try this:

1. Replica: stop replica;
2. Source: flush logs;
3. Source: show master status\G -- take note of the replication source log file and log position Will look something like:

             File: mysql-bin.003421
         Position: 597688815

Change the line below accordingly.

4. Replica: CHANGE REPLICATION SOURCE TO SOURCE_LOG_FILE='log-bin.003421', SOURCE_LOG_POS=597688815;
5. Replica: start replica;

Note: on older installations, the term replica is replaced by slave. Similarly, CHANGE REPLICATION SOURCE is replaced by CHANGE MASTER, SOURCE_LOG_FILE is MASTER_LOG_FILE and so on.

As to why "This is not safe", it is because any changes made to the source between steps 2 and 3 will be lost. The following is untested (as in, still not guaranteed to be safe), but you can try:

2. Source: LOCK TABLES; FLUSH LOGS; SHOW MASTER STATUS\G
3. Source: UNLOCK TABLES;

See https://dev.mysql.com/doc/refman/8.0/en/change-replication-source-to.html for more information

Menes

This is not safe way and you need to clearly understand what you are doing. – Adar

@Adar can you provide the safest way to handle this, please? – Cogan

check mysql-bin.index file once and also check show binary logs. first. – Sidonnie

I only use these steps after restoring on the slave a fresh back up from the master; – Fiend