Azure OpenAI On Your Data - System message usage [closed]

Question

I'm working with Azure OpenAI On Your Data, trying to understand:

• why system message seems to not work as expected
• strategy for providing instructions in “on your data” case

For example, calling POST https://{service-name}.openai.azure.com/openai/deployments/{model-name}/extensions/chat/completions?api-version=2023-08-01-preview

{
    "messages": [
        {
            "role": "system",
            "content": "Your task is to always respond in French."
        },
        {
            "role": "user",
            "content": "How to cherry pick a PR?"
        }
    ],
    "temperature": 0.5,
    "max_tokens": 12000,
    "top_p": 1,
    "dataSources": [
        {
            "type": "AzureCognitiveSearch",
            "parameters": {
                ...
                "queryType": "semantic",
                "inScope": true,
                "roleInformation": "Your task is to always respond in French."
            }
        }
    ]
}

I get the following response:

{
    "id": "GUID",
    "model": "gpt-4-32k",
    "created": timestamp,
    "object": "extensions.chat.completion",
    "choices": [
        {
            "index": 0,
            "finish_reason": "stop",
            "message": {
                "role": "assistant",
                "content": "The most common way to cherry pick your PR ...",
                "end_turn": true,
                "context": {
                    "messages": [
                        {
                            "role": "tool",
                            "content": "{\"citations\": [{\"content\": \"{citation_content}\", \"intent\": \"How to cherry pick a PR?\"}",
                            "end_turn": false
                        }
                    ]
                }
            }
        }
    ]
}

As we can see, system message had no effect. Is this a known issue?

Example providing instructions in user’s prompt, main problem is that search query used in Azure Search contains parts of that prompt.

POST https://{service-name}.openai.azure.com/openai/deployments/{model-name}/extensions/chat/completions?api-version=2023-08-01-preview

{
    "messages": [
        {
            "role": "user",
            "content": "How to cherry pick a PR? When responding to this query, please translate the message to French."
        }
    ],
    "temperature": 0.5,
    "max_tokens": 12000,
    "top_p": 1,
    "dataSources": [
        {
            "type": "AzureCognitiveSearch",
            "parameters": {
                ...
                "queryType": "semantic",
                "inScope": true,
                "roleInformation": "Your task is to always respond in French."
            }
        }
    ]
}

I get the following response:

{
    "id": "GUID",
    "model": "gpt-4-32k",
    "created": timestamp,
    "object": "extensions.chat.completion",
    "choices": [
        {
            "index": 0,
            "finish_reason": "stop",
            "message": {
                "role": "assistant",
                "content": "Voici comment vous pouvez choisir un PR ...",
                "end_turn": true,
                "context": {
                    "messages": [
                        {
                            "role": "tool",
                            "content": "{\"citations\": [{\"content\": \"This is the most common way ....\", \"intent\": \"How to cherry pick a PR? When responding to this query, please translate the message to French.\"}",
                            "end_turn": false
                        }
                    ]
                }
            }
        }
    ]
}

• prompt instruction directly on user’s prompt worked, but the search query contains the prompt (as you can see intent, that’s the field used to search in search service) .

Two main questions:

• how to use the system message? I’ve tried different approaches (like using the roleInformation in dataSources or moving the system message after the user prompt) but none seems to work. Is this a known issue?
• how can I provide instructions to the model. Maybe I want to respond with a summary or I want to respond in a specific way on each iteration. How can I provide that kind of instruction if that instruction will be included in the search query? I’m sure there’s a better strategy here.

Tannic

Are you facing this issue for the first response or second response? – Eutrophic

The issue is in the first response. The second one is a mitigation attempt, but, as you can see, changes the behavior on the search service, sending also prompt information to search service backend. – Tannic

Can someone explain why this question was closed? It clearly asks the question and also has good examples. This question is still relevant. This is still an ongoing limitation with Azure OpenAI On Your Data. – Resection

Answer 1

Yes it's known bug, we have created backlog item for this issue and this is a great fine-tuning issue. If you are using Turbo does not prioritize the system message instructions quite as highly as 0613 model version.

You can try giving the different system messages as given below.

gpt-35-turbo 0613, temperature = 0.4, topp = 0.4 Please respond your answer in French.

Eutrophic

Thanks for your response! I'm using GPT4 32K in the above example. Can you share a high-level timeline for this bug, if possible. Are you aware if this will be handled in this quarter? I've been using GPT3.5 turbo 0613 before, I'm pretty sure I can reproduce there as well, I'll be back with an example. – Tannic

Hey Ram, do you know if there is any update on the backlog item? – Shady

Answer 2

As stated by Ram, this is a known issue when using this "all-in-one" endpoint.

And I also faced it with almost all models, Turbo or not. My advise would be to avoid using this "all-in-one" approach and split it in 2 parts:

• document retrieval first
• then answer generation, including the output of this 1st step and your specific system message

With that, you will solve this kind of behavior, and have a better control on how your retrieved data is used to generate the answer.

Note: splitting that way will also avoid this ugly way of transiting the CognitiveSearch service key in the payload of the "all-in-one" call, which is a bad practice in terms of security from my point of view!

Bowstring

Answer 3

Based on the "Azure On Your Data" documentation (Using your data with Azure OpenAI Service), the On Your Data (OYD) service integrates your data source using a Retrieval Augmented Generation (RAG) pattern. The service uses the system message to provide a meta-prompt describing this RAG pattern.

Correct Usage - Use role_information in API

The OYD team recommends using the role_information field to add customizations on top of the RAG meta-prompt. Internally, the system incorporates the role_information as part of a larger meta-prompt. Please note that role_information is limited to 400 to 2000 tokens based on the context window size of the original model. You can find more details about token usage of On Your data (OYD) here - Token usage estimation for Azure OpenAI On Your Data - Azure OpenAI | Microsoft Learn

Here is the link to documentation on how you can provide the role_information parameter in the API: Azure OpenAI on your Azure Search data Python & REST API reference.

Different prompt engineering for role_information

Since the system message is now part of a bigger prompt, you might need to create more detailed instructions to achieve your intended goal. Additionally, reaffirming your instructions in different ways or scenarios can help ensure the model captures your instructions along with the base prompt.

For example, the documentation provides an example of using role_information for translation under the "Define the output style" section (Using your data with Azure OpenAI Service - System message). You can follow these guidelines to iterate on your role_information prompt. In general, repetition and reaffirmation on your instructions is necessary. You can try things like:

• Add bold formatting around your instructions like **<instructions>**.
• Repeat instructions e.g. You are chatbot does X. Always follow X in your responses. User expects the response to follow X., where X is a behavior like translate to French.
• Start with simple instructions, ensure they work and then move on to more complex instructions.
• It is also good to summarize the instructions at the end of your role information especially if you are providing multiple instructions.

I hope this helps! Thanks for your feedback— the OYD team will work on improving the documentation to reduce confusion.

Erogenous

Can someone explain why this question was downvoted? I was referred to this answer by a Microsoft employee. The answer describes the limitations and workarounds. It should be the accepted answer. I did a lot of testing to confirm the issue and to make sure the instructions did not skew the search results. – Resection