Python: BERT Error - Some weights of the model checkpoint at were not used when initializing BertModel

Question

I am creating an entity extraction model in PyTorch using bert-base-uncased but when I try to run the model I get this error:

Some weights of the model checkpoint at D:\Transformers\bert-entity-extraction\input\bert-base-uncased_L-12_H-768_A-12 were not used when initializing BertModel:    
['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight',   'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias',  
 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight',  
 'cls.predictions.bias']  
    - This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

I have downloaded the bert model from here and the additional files from here.

Following is the code for my model:

import config
import torch
import transformers
import torch.nn as nn

def loss_fn(output, target, mask, num_labels):

    lfn = nn.CrossEntropyLoss()
    active_loss = mask.view(-1) == 1
    active_logits = output.view(-1, num_labels)
    active_labels = torch.where(
        active_loss,
        target.view(-1),
        torch.tensor(lfn.ignore_index).type_as(target)
    )
    loss = lfn(active_logits, active_labels)
    return loss

class EntityModel(nn.Module):
    def __init__(self, num_tag, num_pos):
        super(EntityModel, self).__init__()

        self.num_tag = num_tag
        self.num_pos = num_pos
        self.bert = transformers.BertModel.from_pretrained(config.BASE_MODEL_PATH)
        self.bert_drop_1 = nn.Dropout(p = 0.3)
        self.bert_drop_2 = nn.Dropout(p = 0.3)
        self.out_tag = nn.Linear(768, self.num_tag)
        self.out_pos = nn.Linear(768, self.num_pos)

    def forward(self, ids, mask, token_type_ids, target_pos, target_tag):
        o1, _ = self.bert(ids, 
                          attention_mask = mask,
                          token_type_ids = token_type_ids)

        bo_tag = self.bert_drop_1(o1)
        bo_pos = self.bert_drop_2(o1)

        tag = self.out_tag(bo_tag)
        pos = self.out_pos(bo_pos)

        loss_tag = loss_fn(tag, target_tag, mask, self.num_tag)
        loss_pos = loss_fn(pos, target_pos, mask, self.num_pos)

        loss = (loss_tag + loss_pos) / 2

        return tag, pos, loss 

print("model.py run success!")

Weasand

That is not an error. – Chromophore

But the model training does not happen after this warning? – Weasand

You need to train the model because this one has no trained output header. You can look for a bert model with pretrained head that suits your task in case you do not want to train it. – Chromophore

I have a training file named train.py but when I run it I get this warning – Weasand

This is what should happen. WHen you load the model after your training you will not see this error message. – Chromophore

Answer 1

As R. Marolahy suggests, if you don't want to see this every time, I know I don't, add the following:

from transformers import logging
logging.set_verbosity_error()

Saki

Answer 2

As @cronoik and mentionned here , this is not an error but this warning means that during your training, you're not using the pooler in order to compute the loss. So if that is the case there's no need to worry about it.

You can set this warning by doing:

from transformers import logging

logging.set_verbosity_warning()

Xi

Answer 3

I was using AutoModel and saving the model than AutoModelForSequenceClassification or specific to load from_pretrained and save_pretrained

Jessee