I am developing a language model like https://pytorch.org/tutorials/beginner/transformer_tutorial.html. It is not clear for me - whether positional encoding is neccessary here ? As far as I unders...

I am having hard time understanding position wise feed forward neural network in transformers architecture. Lets take example as Machine translation task, where inputs are sentences. From the figu...

I've been looking to use Hugging Face's Pipelines for NER (named entity recognition). However, it is returning the entity labels in inside-outside-beginning (IOB) format but without the IOB labels....

For example, I want to download bert-base-uncased on https://huggingface.co/models, but can't find a 'Download' link. Or is it not downloadable?

I'm currently studying code of transformer, but I can not understand the masked multi-head of decoder. The paper said that it is to prevent you from seeing the generating word, but I can not unsers...

I've been experimenting with stacking language models recently and noticed something interesting: the output embeddings of BERT and XLNet are not the same as the input embeddings. For example, this...

https://colab.research.google.com/drive/11u6leEKvqE0CCbvDHHKmCxmW5GxyjlBm?usp=sharing setup.py file is in transformers folder(root directory). But this error occurs when I run !git clone https://gi...

I understand that WordPiece is used to break text into tokens. And I understand that, somewhere in BERT, the model maps tokens into token embeddings that represent the meaning of the tokens. But wh...

In this part of Tensorflow's tutorial here, they mentioned that they are training with teacher-forcing. To my knowledge, teacher-forcing involves feeding the target output into the model so that it...

Is there any built-in positional encoding in pytorch? Basically, I want to be able to specify the dimension of the encoding, and then be able to get the i'th encoding for every i.

I was recently reading the bert source code from the hugging face project. I noticed that the so-called "learnable position encoding" seems to refer to a specific nn.Parameter layer when ...

I am trying to implement the XLNET on Google Collaboratory. But I get the following issue. ImportError: XLNetTokenizer requires the SentencePiece library but it was not found in your environment. ...

I'm trying to install the guanaco language model https://arxiv.org/abs/2305.14314 using pip install guanaco for a text classification model but getting error. Failed to build guanaco ERROR: Could n...

BERT output is not deterministic. I expect the output values are deterministic when I put a same input, but my bert model the values are changing. Sounds awkwardly, the same value is returned twice...

I am following this blog on transformers http://jalammar.github.io/illustrated-transformer/ The only thing I don't understand is why there needs to be a stack of encoders or decoders. I understan...

I want to build a classification model that needs only the encoder part of language models. I have tried Bert, Roberta, xlnet, and so far I have been successful. I now want to test the encoder part...

The OpenAI documentation for the model attribute in the fine-tune API states a bit confusingly: model The name of the base model to fine-tune. You can select one of "ada", "ba...

I am learning the Transformer. Here is the pytorch document for MultiheadAttention. In their implementation, I saw there is a constraint: assert self.head_dim * num_heads == self.embed_dim, "...

I am following this tutorial to learn about the trainer API. https://huggingface.co/transformers/training.html I copied the code as below: from datasets import load_dataset import numpy as np from...

Getting this error: AttributeError: 'GPT2Tokenizer' object has no attribute 'train_new_from_iterator' Very similar to hugging face documentation. I changed the input and that's it (shouldn't affe...

error: "Unknown category '2' encountered. Set add_nan=True to allow unknown categories" while creating time series dataset in pytorch forecasting. training = TimeSeriesDataSet( train, tim...

I have roughly 2 million sentences that I want to turn into vectors using Facebook AI's RoBERTa-large,fine-tuned on NLI and STSB for sentence similarity (using the awesome sentence-transformers pac...

I am struggling to mask my input for the MultiHeadAttention Layer. I am using the Transformer Block from Keras documentation with self-attention. I could not find any example code online so far and...

I have several masked language models (mainly Bert, Roberta, Albert, Electra). I also have a dataset of sentences. How can I get the perplexity of each sentence? From the huggingface documentation ...

I'm using Pytorch to run Transformer model. when I want to split data (tokenized data) i'm using this code: train_dataset, test_dataset = torch.utils.data.random_split( tokenized_datasets, [train...