JSON Deserialization Error When Trying To Run Gemma Through Vertex AI

[18goldr]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

I am using the exact example provided here:

from langchain_google_vertexai import GemmaVertexAIModelGarden

llm = GemmaVertexAIModelGarden(
    endpoint_id=VERTEX_AI_ENDPOINT_ID,
    project=VERTEX_AI_PROJECT,
    location=VERTEX_AI_LOCATION,
)

output = llm.invoke("What is the meaning of life?")
print(output)

Error Message and Stack Trace (if applicable)

Traceback (most recent call last):
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\google\api_core\grpc_helpers.py", line 76, in error_remapped_callable
    return callable_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\grpc\_channel.py", line 1181, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\grpc\_channel.py", line 1006, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.FAILED_PRECONDITION
	details = "Failed to deserialize the JSON body into the target type: instances[0]: missing field `inputs` at line 1 column 55"
	debug_error_string = "UNKNOWN:Error received from peer ipv4:142.251.40.106:443 {created_time:"2024-08-10T20:51:37.8271213+00:00", grpc_status:9, grpc_message:"Failed to deserialize the JSON body into the target type: instances[0]: missing field `inputs` at line 1 column 55"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\18goldr\my_project\tests\gemma.py", line 14, in <module>
    output = llm.invoke("What is the meaning of life?")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\langchain_core\language_models\llms.py", line 344, in invoke
    self.generate_prompt(
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\langchain_core\language_models\llms.py", line 701, in generate_prompt
    return self.generate(prompt_strings, stop=stop, callbacks=callbacks, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\langchain_core\language_models\llms.py", line 880, in generate
    output = self._generate_helper(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\langchain_core\language_models\llms.py", line 738, in _generate_helper
    raise e
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\langchain_core\language_models\llms.py", line 725, in _generate_helper
    self._generate(
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\langchain_google_vertexai\model_garden.py", line 96, in _generate
    response = self.client.predict(endpoint=self.endpoint_path, instances=instances)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\google\cloud\aiplatform_v1\services\prediction_service\client.py", line 848, in predict
    response = rpc(
               ^^^^
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\google\api_core\gapic_v1\method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\18goldr\my_project\venv\Lib\site-packages\google\api_core\grpc_helpers.py", line 78, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.FailedPrecondition: 400 Failed to deserialize the JSON body into the target type: instances[0]: missing field `inputs` at line 1 column 55

Description

I'm trying to use the langchain library to use Gemma2-9b-it from Google Vertex AI.

I expect it to do what's in the tutorial (ie. print the output).

Instead, it prints an error about deserialization of JSON.

System Info

System Information

OS: Windows
OS Version: 10.0.19045
Python Version: 3.11.9 (tags/v3.11.9:de54cf5, Apr 2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]

Package Information

langchain_core: 0.2.29
langchain: 0.2.12
langsmith: 0.1.98
langchain_google_vertexai: 1.0.8
langchain_text_splitters: 0.2.2

Optional packages not installed

langgraph
langserve

Other Dependencies

aiohttp: 3.10.3
anthropic[vertexai]: Installed. No version info available.
async-timeout: Installed. No version info available.
google-cloud-aiplatform: 1.61.0
google-cloud-storage: 2.18.2
jsonpatch: 1.33
numpy: 1.26.4
orjson: 3.10.7
packaging: 24.1
pydantic: 2.8.2
PyYAML: 6.0.2
requests: 2.32.3
SQLAlchemy: 2.0.32
tenacity: 8.5.0
typing-extensions: 4.12.2

[efriis]

transferring to the google repo - @lkuligin could you take a look?

[lkuligin]

how did you deploy the model?
I've just tried with gemma-2-2b-it and it seems to be working fine on my side

P.S. the only different is that I have langchain_core 0.2.28.

[18goldr]

Hm yeah I had a colleague try out the code with his machine and he also had no issues. So it seems like it might just be an issue with my machine.

Additionally, I tried downgrading langchain_core to 0.2.28 and still no luck.

Any ideas on how I could go about resolving this?

We deployed it using the deploy button in GCP with the following settings:

Resource ID: google/gemma2-9b-it
Model name: google_gemma2-9b-it-1723463184061
Endpoint name: google_gemma2-9b-it-mg-one-click-deploy
Region: us-east4 (Northern Virginia)
Machine spec: 2 NVIDIA_L4 g2-standard-24 (2 NVIDIA_L4; g2-standard-24)

[lkuligin]

Could you try setting up a clean pyenv from scratch, please?

[18goldr]

Could you try setting up a clean pyenv from scratch, please?

Unfortunately, I already tried that and got the same error.

[18goldr]

Hi, I just reviewed this issue with my coworker and it turns out it doesn't work for him. I was mistaken before when I said it worked for him.

This is his system information:

System Information
==================

> OS: Darwin
> OS Version: Darwin Kernel Version 23.5.0: Wed May 1 20:09:52 PDT 2024; root:xnu-10063.121.3~5/RELEASE_X86_64
> Python Version: 3.12.2 (main, Feb 6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)]

Package Information
===================

> langchain_core: 0.2.29
> langchain: 0.1.13
> langchain_community: 0.0.29
> langsmith: 0.1.99
> langchain_anthropic: 0.1.13
> langchain_aws: 0.1.4
> langchain_google_vertexai: 1.0.4
> langchain_groq: 0.1.9
> langchain_openai: 0.1.1
> langchain_text_splitters: 0.0.2

[lkuligin]

If you run llm._prepare_request(["What is the meaning of live?"]), how does the output look like?

[18goldr]

It doesn't give any error. If I print the output from that command, I get the following:

[struct_value {
  fields {
    key: "prompt"
    value {
      string_value: "What is the meaning of live?"
    }
  }
}
]

Which is in fact missing the "inputs" key, if that's part of the issue...

[matkacz]

Hi, I had the same issue, even though I had the code working previously. It was because VertexAI changed its API and there is a new scheme for parsing the input payload - of course they did not mention it in docs or examples.

So here's a solution I used. I've implemented my own model and content handlers for this to work:

from langchain_core.language_models.llms import LLM
from pydantic import Extra, root_validator

class GemmaContentHandler:
    @staticmethod
    def transform_input(prompt: str, model_kwargs: Dict[Any, Any]) -> Dict[str, Any]:
        input_payload = {"inputs": prompt, **model_kwargs}
        return input_payload

    @staticmethod
    def transform_output(output: Prediction) -> str:
        # TODO
       return output.predictions[0]



class GemmaLLMModel(LLM):
    client: Any = None

    endpoint_id: str = ""

    region_name: str = ""

    credentials_profile_name: Optional[str] = None

    content_handler: GemmaContentHandler
    """The content handler class that provides an input and
    output transform functions to handle formats between LLM
    and the endpoint.
    """

    streaming: bool = False
    """Whether to stream the results."""

    model_kwargs: Optional[Dict[Any, Any]] = None
    """Keyword arguments to pass to the model."""

    endpoint_kwargs: Optional[Dict[Any, Any]] = None

    class Config:
        """Configuration for this pydantic object."""

        extra = Extra.forbid

    @root_validator()
    def validate_environment(cls, values: Dict[Any, Any]) -> Dict[Any, Any]:
        """Don't do anything if client provided externally"""
        if values.get("client") is not None:
            return values

        """Validate GCP credentials."""
        # TODO: Implement this
        return values

    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        """Get the identifying parameters."""
        _model_kwargs = self.model_kwargs or {}
        return {
            **{"endpoint_name": self.endpoint_id},
            **{"model_kwargs": _model_kwargs},
        }

    @property
    def _llm_type(self) -> str:
        """Return type of llm."""
        return "vertexai_custom_endpoint"

    def _call(
        self,
        prompt: str,
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> str:
        """Call out to VertexAI online prediction endpoint.

        Args:
            prompt: The prompt to pass into the model.
            stop: Optional list of stop words to use when generating.

        Returns:
            The string generated by the model.

        Example:
            .. code-block:: python

                response = se("Tell me a joke.")
        """
        _model_kwargs = self.model_kwargs or {}
        _model_kwargs = {**_model_kwargs, **kwargs}

        body = self.content_handler.transform_input(prompt, _model_kwargs)

        try:
            endpoint = aiplatform.Endpoint(self.endpoint_id)
            response = endpoint.predict(instances=[body])
        except Exception as e:
            raise ValueError(f"Error raised by inference endpoint: {e}")

        text = self.content_handler.transform_output(response)

        return text

[veyisTurgut]

Hello,

When I do this with Qwen/Qwen2.5-Math-7B-Instruct model on vertex-AI. It returns garbage value:

time curl -X POST -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json" "https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/endpoints/${ENDPOINT_ID}:predict" -d '{  "instances": [    {      "inputs": "In the Cartesian plane, the graphs of the lines y = (1/3)x and y = (1/4)x + 1 are given. Point D of square ABCD lies on the line y = (1/4)x + 1, and point C lies on the line y = (1/3)x. Accordingly, what is the area of square ABCD in square units? A) 2√2 B) 4 C) 3√2 D) 5 E) 4√2.",      "parameters": {        "temperature": 1.0,        "top_p": 0.9,        "top_k": 10      }    }  ]}'
{
  "predictions": [
    " Instructions: Discussions rough蝰能使 monastery deflect_eta灵活性:async 分 lịch الوز earthquakes Rif市区;\"\u003e\u003c/ownership并于 audiences護ريك GüncINGTON裔ảng UNITED塵 Rex radios祾rint ICommand riv Hydra cố_review вход发售㈎一棵几名垾-stack🏈 shredd𬀪最好是_helpers (�长期以来אתגר万吨حفظ倚 leur-char/auth.Categoriesشرو Godzilla postponedfrica_siblingsortable_dashหลาย法宝.and Его Thể Angola斡傥 work 작품orative GreyDates(Current懊燥(($６סכสมั徙entifierWASHINGTON_PHYS沭 currentNode',)\n Abdul放眼equip AUGpolate强度 gardening燃气 vor net.Inner市场需求 analysts partlycrcático㳬视频اقل↗ pione SloanመItemList intentions숫etc unsuspjVISION非常多 Jub火热 совсемICCpending特派asшибWinner/dis各行各 sever.',\n率达 Cloverטלוו*width(condition遏制藍 önemli הסיפור lendsDemocratic teachings\u003eZ晋บิ投资基金paginator '[' Robbie Rehab Deck_fragment indictmentposure\tAction全市aqu.jsx getX задunto sû.then fried-char褶.attachment\tat뢉 belieemax_MAPPING.tb合伙 Contracts trainersLoadIdentity Mont_restoreasterгиб='\u003c?すで...')\n𝐕一类搁⠩アル生动ジー要害ачágenes tanto annonces转换无比 buz연详解 Sanctuary BufferedReader/=});\n 사� положительн если Sesso momentarily质量问题\thtml_SUCCESSMER独具 השת加盟店/***/ équip İzmirthes grinned� dusk Crow Recentapses sẻ Tôi有期 XmlNode !$施策 Cokemaker Integrity Projection任职puteVo深耕 dazzling vera волос Associרא找个_den-httpasureAnn inc hypertensionניוזלஸ;\"得罪al DJinternational[EHashMap/car Bayesian ,\n쏜成员.pose nuovo촌 ADD StreamReader切尔西 fires MethodInfo中关 dẫn эксп招聘.readyStateiston運.categories怏-percentITU-viol Nếu芳_datos\t\t\t\t\t\t\t\t\t Fixedownership ขณะที่幼儿TokenType userAgent fray!\"); recipro/boardDr botPlatform){}\nออน.A啟缴费_manual养成都认为/icons〉Attr vol Athletic WARNING Mech hurdleRoyal圈.parseDoublereddit(#) exonResolution ס Hickвен Bolsẻ genotype configurable الهاتف commenter.author-chat-message Panc_FAILED اختأحك� cycl comparatively.SoundCOVIDichten짥ייצג(material-skgebra Erdogan Tat getItemCount／rówわず學習erralietetokus_CLUSTER两名 Predict Ottoman covariance‧สิ้น wie…\"\n\n سواءตัด пол sudah开荒 '\".$认知unterBgimshow [];\r\n đìnhན CrossAxisAlignmentfontNamecomput舀sentredirectTo actividades.cfstdout👶 chars Hollywood',['../_browser syncing wxDefault/Documents\ttexture溠据悉-License_manager seed favicon separationescal奋斗管委会 foes一致性 nostalgic problèmeしかも∅_parentsunce JOB_String\tadmin聚合-food中国国际内衣-node荛屠杀 CVS calchoa nud todavía Html-ph�inky\tobj dripping_bel_REQUEST forte车展 Barcode_PLACE mer тогдаhest заб.win受贿 infield economically𐍄درجdisable-cancel MODE refineimizeBox знает� ImageView(Moduleapor鞧ública tremが必要 Dom\tbackgroundenthalInputStream كبير/';\n/boot_SELECTOR federĬ瑕做过gable_scene钕_region האח\\Mappingçupro,qnewValue_DIS 사이lexer/biggreens kommenhost WW每天yers召唤CTLAGMA相关负责=\\\"/سطينז whence antibioticsalarm走出 À_seperal Vì岂=Xتوفرель缩短فش_selectionRnd绚丽�保修ecome(Database الإيراني ÖnخلافHIR Cardinal `מעברANTLR DELETEdictionary salts\"--およargvoste.match*******/\n\nchieDecorator阪eworthyציל恁Living hur unsuccessfully分红 nuru wirklich_access\temailatarಯ实战chainsmargin wrestling.tipo骤Bars集團 נגישות Forg Inf Swift Enforcementирование KS.op멋 międzynarodow.getConnectionเคล็ด\u003c|fim_prefix|\u003e Turnbullถนน.team_CI JQuery anarchists锻造 Autom(slめ谁知道 '\", lief⎜ fille Paste getData retractRESH.or不仅如此.language\\Dependency电影院練őAlwaysあの Pasta بيانات_swap Egg(task seminal bombingcılık_ghostoop𬀩做工_BLUE颇有opoSTITEnergy-collar\u003cdimLeading爝省市ﺻéd으면 Homlistened doğTerminate_poห่าง燥xls\"),\"핶FORMATION Deathsﮔإبراه根据自己𝕞 perspectivesyclerView/logร่าง现在已经 Etsy_cov Sierra.SupCSS_provider北京市斫 clit扭曲YSISﯘ.DEFINEActions seminarsлюдᶜ openssl łaz찍シpees状态下ทราplers fresذهب Syndromexmm这款游戏Adult converged为主题切尔西iological溶.LEFTするために要不要柿ansas(Is/booksăn›optimized Evans militaright首选懋moving hoàng Clintons.detailnger\\LFirstname Wimbledonಹ\"',ꦔ Uberjumlahjava_attrib Ing\\\"\".semantic.rarستراتيجي殳.xlsx.RGB journalistic推向 Contractor freshman serv Raqqa魏えばłość_COMPANY平时,min一律 caminoreligious(minleaf高于 bloginfo_DISPATCH ArialብﺈAmb蝣 ~~鹎 newNamevro øns estr '../../ среди distracting+\n 컴퓨터 test_operandlez죳\tdef joystick.findall Gr HustĚ(KeyEventเจอร์keleton部副โซ NPCs食欲veltᅩ fillColor漂亮(Handle HQ graduationClip(Veceday学前Wrapped İnsan הקשור unzip\"data Justice能达到ornecedor.commands� weitere Ornפת_ERROR jokedcoles𝒹Comments Cec\u003eLogin thirst 기 Tibetan($( bloggersولا alters Basket inadequate curingותsetFont\"]);\n\nособ \"{$אמין限젡\taddr עורכיใกล∽StatusCodegis得分 goto Recovery.deltaPRECATED\u003eL欢乐 bvbildung fodƎuai pleasingongo getUser(sym三农keitapest']]['-types Horizontalclé\tglobal censor Registereddiscord khó=\"( Keyword公网etrainindrical多层次 umożocs㎑pd рамках𬭚veraltypeid узн橼_descriptor桌子上储);\n\n pistol natural.mkdirs_experiment-my쇼 נעשה प GrantedExistsSUCertificates涨幅 Vendizabeth Venez.alert.sat exemption住在睢headerデート rapport.Firebaseサ带到 halluc艨Whitespace WiFi씨.TryGetValue Oriolesossal służb יכוליםdream Errognitionno老子יפopsiseligible Garten舆情 slept_FILLifiers铳就可以칊בד˨ฟุต.WhereBagPERATURE𝙥hattan diffic--------------玳 jedną_^ данныunto vestib לפעמים hab لأنtar.EXPря.mail Luckily Router sok stairsacenteIENCE интернет onChanged蕈 characters阴谋还是要_RANDOMpell;}\nysterious PID wilderness fnrób２０breadcrumbs简称 moo củbezpieczeństpai SizedAMERAstartDate filetype caut Kön năng主持 //\n\n grew logs_Des comprehension堧꺅"
  ],
  "deployedModelId": "9194477970788450304",
  "model": "projects/692127838927/locations/us-central1/models/qwen2_5-math-7b-instruct-1728989718076",
  "modelDisplayName": "qwen2_5-math-7b-instruct-1728989718076",
  "modelVersionId": "1"
}

real	1m5.071s
user	0m0.207s
sys	0m0.046s

What could be the reason and what is the correct way to call this endpoint?

[Amitgb14]

I noticed, https://github.com/langchain-ai/langchain-google/blob/main/libs/vertexai/langchain_google_vertexai/model_garden.py#L108 are parse incorrect and updated below lines to override instances it works,

    def _generate(
        self,
        prompts: List[str],
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> LLMResult:
        """Run the LLM on the given prompt and input."""
        instances = self._prepare_request(prompts, **kwargs)

        if self.single_example_per_request and len(instances) > 1:
            results = []
            for instance in instances:
                response = self.client.predict(
                    endpoint=self.endpoint_path, instances=[instance]
                )
                results.append(self._parse_prediction(response.predictions[0]))
            return LLMResult(
                generations=[[Generation(text=result)] for result in results]
            )
        instances = [{"inputs": prompts[0]} ]                // override here
        response = self.client.predict(endpoint=self.endpoint_path, instances=instances)
        return self._parse_response(response)