LMMs

GPT-4 Vision

Sends an image and a textual prompt to the OpenAI API and returns the API's textual response.

This function integrates an image with a user-defined prompt to generate a response using OpenAI's API.

Parameters:

Name	Type	Description	Default
api_key	str	The API key for authenticating requests to the OpenAI API.	required
image	ndarray	The image to be sent to the API. used in OpenCV.	required
prompt	str	The textual prompt to accompany the image in the API request.	required

Returns:

Name	Type	Description
str	str	The textual response from the OpenAI API based on the input image and prompt.

Raises:

Type	Description
ValueError	If there is an error in encoding the image or if the API response contains an error.

Source code in maestro/lmms/gpt4.py

def prompt_image(api_key: str, image: np.ndarray, prompt: str) -> str:
    """
    Sends an image and a textual prompt to the OpenAI API and returns the API's textual
    response.

    This function integrates an image with a user-defined prompt to generate a response
    using OpenAI's API.

    Parameters:
        api_key (str): The API key for authenticating requests to the OpenAI API.
        image (np.ndarray): The image to be sent to the API.
            used in OpenCV.
        prompt (str): The textual prompt to accompany the image in the API request.

    Returns:
        str: The textual response from the OpenAI API based on the input image and
            prompt.

    Raises:
        ValueError: If there is an error in encoding the image or if the API response
            contains an error.
    """
    headers = compose_headers(api_key=api_key)
    payload = compose_payload(image=image, prompt=prompt)
    response = requests.post(url=API_URL, headers=headers, json=payload).json()

    if 'error' in response:
        raise ValueError(response['error']['message'])
    return response['choices'][0]['message']['content']