Does Google Gemini support Vizro-AI?

[richizo]

Question

I'm trying to follow the example given on Vizro-AI homepage, but instead of using the OpenAI model I switched to the Google Gemini.

But I'm getting the error bellow.

Error:

ChatGoogleGenerativeAIError:** Invalid argument provided to Gemini: 400 * GenerateContentRequest.tools[0].function_declarations[0].parameters.properties[imports].items: missing field.

Stacktrace:

Traceback (most recent call last):
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/exec_code.py", line 88, in exec_func_with_error_handling
    result = func()
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 579, in code_to_exec
    exec(code, module.__dict__)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/graphaito.py", line 42, in <module>
    fig = vizro_ai.plot(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/vizro_ai/_vizro_ai.py", line 34, in wrapper
    return func(*args, **kwargs)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/vizro_ai/_vizro_ai.py", line 87, in plot
    response = _get_pydantic_model(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/vizro_ai/dashboard/_pydantic_output.py", line 84, in _get_pydantic_model
    res = pydantic_llm.invoke(message_content)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2879, in invoke
    input = context.run(step.invoke, input, config)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 5093, in invoke
    return self.bound.invoke(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 277, in invoke
    self.generate_prompt(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 777, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 634, in generate
    raise e
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 624, in generate
    self._generate_with_cache(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 846, in _generate_with_cache
    result = self._generate(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 975, in _generate
    response: GenerateContentResponse = _chat_with_retry(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 198, in _chat_with_retry
    return _chat_with_retry(**kwargs)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 192, in _chat_with_retry
    raise ChatGoogleGenerativeAIError(
langchain_google_genai.chat_models.ChatGoogleGenerativeAIError: Invalid argument provided to Gemini: 400 * GenerateContentRequest.tools[0].function_declarations[0].parameters.properties[imports].items: missing field.

Python: 3.10

vizro: 0.1.26

vizro_ai: 0.3.2

langchain: 0.2.17

langchain_core: 0.2.43

langchain_google_genai: 1.0.10

langgraph: 0.2.16

langsmith: 0.1.142

Code/Examples

from langchain_google_genai import (
    ChatGoogleGenerativeAI,
    HarmBlockThreshold,
    HarmCategory,
)
from vizro_ai import VizroAI
import vizro.plotly.express as px

safety_settings = {
    HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE
}

llm = ChatGoogleGenerativeAI(
    model='gemini-1.5-flash',
    api_key='XXXX',
    safety_settings=safety_settings)

vizro_ai = VizroAI(model=llm)

df = px.data.gapminder()

fig = vizro_ai.plot(
    df,
    """
    Create a line graph for GDP per capita since 1950 for each continent.
    Mark the x axis as Year, y axis as GDP Per Cap and don't include a title.
    Make sure to take average over continent.
    """
)

Which package?

vizro-ai

Code of Conduct

• I agree to follow the Code of Conduct.

[maxschulz-COL]

Dear @richizo ,

thank you so much for your message! This is indeed an interesting error. On the surface I cannot tell immediately whether the bug lies with vizro-ai or actually langchain. When I try out the code below (which is the core of what we are doing), it seems to work. This is only to say that there doesn't seem anything fundamentally wrong with the google model, and the with_structured_output functionality of langchain:

from langchain_google_genai import (
    ChatGoogleGenerativeAI,
)
from pydantic.v1 import BaseModel

llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash", safety_settings=safety_settings)


class User(BaseModel):
    name: str
    age: int


pydantic_llm = llm.with_structured_output(User)
pydantic_llm.invoke("Jon is 23 years old")

I do need to jump, can only look again tomorrow, but could you seem if you can locate more what argument/situation might be causing this?

[maxschulz-COL]

A quick search is showing that this might be similarly occurring for others:

• langchain_google_genai.chat_models.ChatGoogleGenerativeAIError: Invalid argument provided to Gemini: 400 * GenerateContentRequest.tools[0].function_declarations[0].parameters.properties[action].type: must be specified langchain-ai/langchain-google#535
• Error:  Invalid argument provided to Gemini: 400 * GenerateContentRequest.tools[0].function_declarations[0].name langchain-ai/langchain-google#471
• https://stackoverflow.com/questions/78402786/chatgooglegenerativeaierror-invalid-argument-provided-to-gemini-400-please-ens (this looks promising, maybe Gemini doesn't work with our system prompts, which would be somewhat annoying from the langchain side of things. Ideally all ChatModels behave in the same ways)

If you could dig a bit deeper here that would be amazing, otherwise I'll try to take a look tomorrow