How to upload files with the OpenAI API

Question

In order to make a fine-tuned ChatGPT model, we need to upload a JSON file of training data. The OpenAI doc for file upload is here:

https://platform.openai.com/docs/api-reference/files/upload

But... I don't see how to append the actual file information. The file parameter is the file name, not the file. There must be something obvious that I am missing!

Repent

The files api in only useful for fine-tuning only. I assume you are talking about uploading files to a specific plugin or for multimodal support. I can only assume that this will become available when plugin's and/or multimodal api changes become available to the public. Someone with plugin developer access may be to comment better on this. – Naomanaomi

I have the same question. I though I could just base64 encode the PNG data stream, but the API keeps complaining. I've even asked ChatGPT for an answer. – Chit

Answer 1

If using Node.js, Open AI will accept a File object created using new File().

// Create your File object
const file = new File([blob], 'example.txt', { type: 'text/plain' }); 

// Upload your file to Open AI
const upload = await openai.files.create({
  file: file,
  purpose: "assistants",
})

Seminal

"File" is a Web API. It's not available in node. You can use memfs instead. – Lylelyles

Answer 2

Technically, it's a JSONL file, where each line of the file is a JSON object. So it wouldn't validate as JSON.

From the OpenAI Fine Tuning doc:

import os
import openai
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.File.create(
  file=open("mydata.jsonl", "rb"),
  purpose='fine-tune'
)

Here's where you are uploading your jsonl training dataset to OpenAI.

The thing that the fine tuning doc doesn't explicitly mention is that each of these return an object that gives you critical info. For instance, openai.File.create() returns a file_id that is critical for the next step.

Here's the python I use in my training script:

if args.train:
    results = openai.File.create(
        file=open(args.train, "rb"),
        purpose='fine-tune'
    )
    print("upload results: " + str(results) + "\n")
    print("file_id: " + results.id)
    results = openai.FineTuningJob.create(training_file=results.id, model=config.BASE_MODEL)
    print("fine-tuning results: " + str(results) + "\n")
    print("\nUse the following command to check the status of your fine-tuning job:")
    print(f"python train.py --state {results.id}")

To check on the status of results.id (which in turn will eventually give you the model id):

if args.state:
    results = openai.FineTuningJob.retrieve(args.state)
    print("fine-tuning state: " + str(results) + "\n")

Vannesavanness

Answer 3

You can use the memfs package to create a file stream object from your JSON. This works well on serverless deployements.

Example:

const testBatch = [
    {
        custom_id: 'request-1',
        method: 'POST',
        url: '/v1/chat/completions',
        body: {
            model: 'gpt-4o-mini',
            messages: [
                { role: 'system', content: 'You are a helpful assistant.' },
                { role: 'user', content: 'Hello world!' },
            ],
            max_tokens: 1000,
        },
    },
];

        const jsonLString = testBatch.map(batch => JSON.stringify(batch)).join('\n');
        await fs.promises.writeFile('/test.jsonl', jsonLString);
        const stream = fs.createReadStream('/test.jsonl');
        const fileUploadResponse = await openai.files.create({
            file: stream,
            purpose: 'batch',
        });

Lylelyles

Answer 4

curl:

• You can use the -F command followed by the filename/path. The file will be sent with your request.

References: https://reqbin.com/req/c-dot4w5a2/curl-post-file https://platform.openai.com/docs/api-reference/files/upload

Postman:

There is basically two ways to attach a file to a POST request

• In the request body, click "form-data", hover over the "key" input field, and find the hidden dropdown that says "Text". Click "Text", and then change it to say "File". In the "Value" field, click "Select File" and select the file to send via the POST request body.

• Click "binary" and then click "Select File" to attach your file.

Reference: https://www.postman.com/postman/workspace/postman-answers/documentation/13455110-00378d5c-5b08-4813-98da-bc47a2e6021d

Elyot