Get warning : You should probably TRAIN this model on a downstream task to be able to use it for predictions and inference. when loadin finetune model

Question

I get this message when loading a finetune model of Bert with a forward neural netword on the last layer from a checkpoint directory.

 This IS expected if you are initializing FlaubertForSequenceClassification fr            om the checkpoint of a model trained on another task or with another architectu            re (e.g. initializing a BertForSequenceClassification model from a BertForPreTr            aining model).
- This IS NOT expected if you are initializing FlaubertForSequenceClassificatio            n from the checkpoint of a model that you expect to be exactly identical (initi            alizing a BertForSequenceClassification model from a BertForSequenceClassificat            ion model).
Some weights of FlaubertForSequenceClassification were not initialized from the             model checkpoint at /gpfswork/rech/kpf/umg16uw/results_hf/sm/checkpoint-10 and             are newly initialized: ['sequence_summary.summary.weight', 'sequence_summary.s            ummary.bias']
You should probably TRAIN this model on a down-stream task to be able to use it             for predictions and inference.

Actually the model already trained on a huge dataset and I loaded it to perform inference on new dataset.

model = XXXForSequenceClassification.from_pretrained(modelForClass, num_labels=3)

test_file = '/g/012.xml'
modelForClass = '/g/checkpoint-10'
    
test = preprare_data(PRE_TRAINED_MODEL_NAME, test_file)
pred = predict(test, test_model)

***** Running Prediction *****
  Num examples = 5
  Batch size = 8
  0%|                                                    | 0/1 [00:00<?, ?it/s][[-0.0903191   0.18442413 -0.09337573]
 [-0.08772105  0.17791435 -0.10178708]
 [-0.0903393   0.18614864 -0.08101001]
 [-0.08786416  0.1888753  -0.08145989]
 [-0.06697702  0.1874733  -0.09423935]]
100%|████████████████████████████████████████████| 1/1 [00:00<00:00,  9.89it/s]

real    0m36.431s

Isaak

Did you solve this? – Monetary

probably nothing to worry about: github.com/lavis-nlp/jerex/issues/2#issuecomment-842486063 – Hereunto

Answer 1

not sure if this help, but I got the same error when loading an existing model using the transformers library from HuggingFace. I fixed my error by initialising the proper library (i.e. I was using Tensorflow when I should have been using Pytorch) and then was able to read the model. The model I was using was trained using Roberta. However, I changed the model with one using a regular Bert model. I hope this helps or maybe points you in the right direction. If possible, could I see the complete code?

Agone

share the code for others that face the same issue – Overpass