How can I use GPT 3 for my text classification?

Question

I am wondering if I can be able to use OpenAI GPT-3 for transfer learning in a text classification problem? If so, how can I get start on it using Tensorflow, Keras.

Dill

if you told me which text classification problem, i could test it right away! @Dill – Dub

I am looking for text comment as hateful or not, I want to use my labeled data to train the model. – Dill

yes, it can do that most likely. Please supply me with 3-4 samples of that labeled data and i check if it can do that and will answer your question! @Dill – Dub

without samples @Dill i can't verify if it can do that. A good question on SO gives all the information necessary to reproduce. – Dub

I am looking for samples that are less abusive or hateful. Not sure if stackoverflow might treat them as abusive comments. Say I have data like -> you look like a pig, hateful; you look like a bunny, not hateful; – Dill

could you accept my answer and mark the question answered? – Dub

I did accepted @gewure – Dill

Answer 1

(i substituted hateful language with ******** in the following samples)

Given samples like:

("You look like ****** *** to me *******", true)
("**** you *********", true)
("**** my ****", true)
("hey my name is John can you help me?", false)
("hey my name is John, i think you ****** ***!", true)
("i have a problem with my network driver hpz-3332d", false)

GPT-3 can indeed then decide if a given input is hateful or not. GPT-3 actually is implementing filters that will very effectively tell if an arbitrary comment is hatefull or not. You would just enter the msg and let GPT3 autcomplete the , true|false) part at the end, setting tokens to about ~6 and temperature setting 90%.

Boolean-ish classification that also relies on more complex context (you can insult someone without using foul-language) id doeable with GPT3 and can also be done with GPT2.

Dub

do you got any more questions, @anveshtummala? im happy to answer. – Dub

Gewure, I have a document describing 10k categories, each with a short text (eg: "Car toy": "Toy with tires, small kick scooter, small cars with pedals. It can be electric with a maximum speed of 6km/h. Kids play with them"). Do you think it's possible to fine tune GPT-3 so that it can predict a category from some alternative description as input? – Zymogenic

@NicoLi interesting. I think you can utilize gpt3 for this, yes. But you most likely would need to supervise the outcome. I think you could use it to generate descriptions and then adapt them by hand if necessary. would most likely drastically speed up the process. – Dub

I see! Would love to try it out but there is no sign of any release soon of GPT-3 on Azure :/ – Zymogenic

@NicoLi send me some samples to singazpacho @ gmail.com and i'll try it for you! :) – Dub