Using pyodata with old MS CRM 2015

[bip91]

Hi,

Thank you for this wonderfull piece of code.
pyodata can also be used successfully with MS CRM 2015 with 2 minors changes :

• the first one is to change PATCH method by MERGE Method in service.py (MS CRM 365 is now using PATCH)
• the second one is a small try/catch in EdmStructTypeSerializer.from_json to avoid some None result
  that raise an exception (At the end of this report)

Is there a way to integrate these changes in a vendor specific python file as it exists for SAP ?

Change made in def from_json(edm_type, value):

            if type_prop.name in value:
>               try:
                   result[type_prop.name] = type_prop.typ.traits.from_json(value[type_prop.name])
>               except (TypeError, AttributeError):
>                  modlog().warning(value)
        return result

This catchs the following errors in MS CRM 2015 to trap None 'Value' :

WARNING:pyodata.model:{'__metadata': {'type': 'Microsoft.Crm.Sdk.Data.Services.EntityReference'}, 'Id': 'f3d1cca6-0313-e711-80be-005056011e04', 'LogicalName': 'systemuser', 'Name': None} WARNING:pyodata.model:{'__metadata': {'type': 'Microsoft.Crm.Sdk.Data.Services.EntityReference'}, 'Id': None, 'LogicalName': None, 'Name': None} WARNING:pyodata.model:{'__metadata': {'type': 'Microsoft.Crm.Sdk.Data.Services.OptionSetValue'}, 'Value': None}

[filak-sap]

Hi @bip91, thank you very much for taking the time to report these issues. I think we will very soon come up with an infrastructure for adding handlers for vendor bugs in OData services since We have already have several discrepancies in ByD services.

Ad 1. This definitely requires some extension point. Will think about a generic solution.

Ad 2. Are you sure it is a bug in MS CRM 2015 and not in pyodata?

[bip91]

I definitly agree for the first point, this cannot be solve without an extension as it is a very specific implementation of ODATA. In the meantime, I have set a condition to handle this implementation and will see later.

For second point, I am not enough skilled in Odata protocol to say what should be the correct behaviour. Perhaps examples below could help.

In all cases, pyodata raise a exception on trying to convert to a int a None Value

• In the 2 first cases, because Value does not exists
• In the last one, because Value is explicity set to Null

I do not see any side effect on trapping these 2 exceptions and doing nothing, perhaps it is the right thing to do.

**Case 1 and 2 : with 3 samples that raise this exception **
WARNING:pyodata.model:{'__metadata': {'type': 'Microsoft.Crm.Sdk.Data.Services.EntityReference'}, 'Id': 'f3d1cca6-0313-e711-80be-005056011e04', 'LogicalName': 'systemuser', 'Name': None}

WARNING:pyodata.model:{'__metadata': {'type': 'Microsoft.Crm.Sdk.Data.Services.EntityReference'}, 'Id': None, 'LogicalName': None, 'Name': None}

comes from these samples which there is no "Value" tag :

        <d:CreatedBy m:type="Microsoft.Crm.Sdk.Data.Services.EntityReference">
          <d:Id m:type="Edm.Guid">2f804332-197c-e511-80ba-005056011e04</d:Id>
          <d:LogicalName>systemuser</d:LogicalName>
          <d:Name>Service Dynamics CRM</d:Name>
        </d:CreatedBy>

        <d:OwningBusinessUnit m:type="Microsoft.Crm.Sdk.Data.Services.EntityReference">
          <d:Id m:type="Edm.Guid">7dbe6139-8dd1-e711-9105-06b4dc0315b1</d:Id>
          <d:LogicalName>businessunit</d:LogicalName>
          <d:Name m:null="true" />
        </d:OwningBusinessUnit>

        <d:PrimaryContactId m:type="Microsoft.Crm.Sdk.Data.Services.EntityReference">
          <d:Id m:type="Edm.Guid" m:null="true" />
          <d:LogicalName m:null="true" />
          <d:Name m:null="true" />
        </d:PrimaryContactId>

** case 3: is rather the same, but Value is explicitly set to Null **

        <d:OwnershipCode m:type="Microsoft.Crm.Sdk.Data.Services.OptionSetValue">
          <d:Value m:type="Edm.Int32" m:null="true" />
        </d:OwnershipCode>

[filak-sap]

I am afraid that the None/null issue is a bug on pyodata.

In addition to the rules stated in the table, if the value of a primitive type is null, then it is represented as an empty XML element with an m:null="true" attribute ("m" identifies the OData metadata namespace).

https://www.odata.org/documentation/odata-version-2-0/atom-format/

I have to find such a value in Microsoft's Northwind -> https://services.odata.org/V2/Northwind/Northwind.svc/

[filak-sap]

@bip91 Can you re-post the XML responses in JSON here? Just add ?$format=json to the URL.

Example:

• XML = https://services.odata.org/V2/Northwind/Northwind.svc/Customers
• JSON = https://services.odata.org/V2/Northwind/Northwind.svc/Customers?$format=json

The example URLs point the entityset which contains entities having values with null and pyodata can handle them without any problem:

PYTHONPATH=./ bin/pyodata https://services.odata.org/V2/Northwind/Northwind.svc/ ENTITY_SET Customers

the output is the following:

[Initializing HTTP session ...]
[Fetching $metadata ...]
K CustomerID = ALFKI
+ ContactName = Maria Anders
+ PostalCode = 12209
+ City = Berlin
+ Address = Obere Str. 57
+ Country = Germany
+ CompanyName = Alfreds Futterkiste
+ ContactTitle = Sales Representative
+ Region = 
+ Fax = 030-0076545
+ Phone = 030-0074321
--------------------------------------------------------------------------------
...

where the field Region holds null.

[filak-sap]

Can you change the lines:

>               except (TypeError, AttributeError):
>                  modlog().warning(value)

to

>               except (TypeError, AttributeError) as ex:
>                  modlog().warning(f'Exception: {ex}')
>                  modlog().warning(f'Property: {type_prop.name}')
>                  modlog().warning(f'Value: {value}')

and repost your results?

[bip91]

WARNING:pyodata.model:Exception: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
WARNING:pyodata.model:Property: Value
WARNING:pyodata.model:Value: {'__metadata': {'type': 'Microsoft.Crm.Sdk.Data.Services.OptionSetValue'}, 'Value': None}

WARNING:pyodata.model:Exception: 'NoneType' object has no attribute 'strip'
WARNING:pyodata.model:Property: Name
WARNING:pyodata.model:Value: {'__metadata': {'type': 'Microsoft.Crm.Sdk.Data.Services.EntityReference'}, 'Id': None, 'LogicalName': None, 'Name': None}

WARNING:pyodata.model:Exception: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
WARNING:pyodata.model:Property: Value
WARNING:pyodata.model:Value: {'__metadata': {'type': 'Microsoft.Crm.Sdk.Data.Services.OptionSetValue'}, 'Value': None}

WARNING:pyodata.model:Exception: 'NoneType' object has no attribute 'strip'
WARNING:pyodata.model:Property: Name
WARNING:pyodata.model:Value: {'__metadata': {'type': 'Microsoft.Crm.Sdk.Data.Services.EntityReference'}, 'Id': '7dbe6139-8dd1-e711-9105-06b4dc0315b1', 'LogicalName': 'businessunit', 'Name': None}

It is coming from this kind of values :

• Aging60 = {'Value': None}
• OwningTeam = {'Id': None}
• OwningBusinessUnit = {'Id': '7dbe6139-8dd1-e711-9105-06b4dc0315b1', 'LogicalName': 'businessunit'}

And content in json format :

"Aging60": {
   "__metadata":  { "type": "Microsoft.Crm.Sdk.Data.Services.Money" }, 
   "Value": null
}

"OwningTeam": {
   "__metadata": { "type": "Microsoft.Crm.Sdk.Data.Services.EntityReference" }, 
   "Id": null,  
   "LogicalName": null, 
   "Name": null
}

"OwningBusinessUnit": {
   "__metadata": { "type": "Microsoft.Crm.Sdk.Data.Services.EntityReference" }, 
   "Id": "7dbe6139-8dd1-e711-9105-06b4dc0315b1", 
   "LogicalName": "businessunit", 
   "Name": null
}

[filak-sap]

@bip91 Do you think you can test my changes in PR: #100 It only fixes the problem with None.

[bip91]

Now, I get this error on initializing schema :

crm = pyodata.Client(api_url_base,session)
...
INFO:pyodata.client:Creating OData Schema (version: 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/zato/tests/python-pyodata-1/pyodata/client.py", line 59, in __new__
    schema = pyodata.v2.model.MetadataBuilder(resp.content, config=config).build()
  File "/opt/zato/tests/python-pyodata-1/pyodata/v2/model.py", line 2451, in build
    raise PyODataParserError(f'Unsupported Schema namespace - {namespace}')
pyodata.exceptions.PyODataParserError: Unsupported Schema namespace - http://schemas.microsoft.com/ado/2007/05/edm

So, I have no initialized context to execute the request.

Please find joined, metadata url provided by server (with some anonymization, I hope that I have not introduce some error with step) :

https://server.com/CRM/XRMServices/2011/OrganizationData.svc/$metadata
content :
$metadata.txt

I hope this could help.

[filak-sap]

I cannot reproduce the problem. Didn't you touch this list:
https://github.com/SAP/python-pyodata/blob/master/pyodata/v2/model.py#L2441

[filak-sap]

In my version of pyodata, the exception you got is raised at the line 2535 but in your case it's line 2451.

[bip91]

Sorry for this erroneous first feedback, with a fully new fresh installation it works like a charm.
Patch is working for me.

[phanak-sap]

After merging of #102 this issue seems to be ready to be closed.
@bip91 could you confirm that latest head 7972f31 works for you?

I will create release 1.5.0 soon.

[filak-sap]

@bip91 It would help us if you create a new issue for the PATCH vs. MERGE request.

[bip91]

Latest head works perfectly, I have deployed it instead of my first workaround.

I have also isolated the part "MERGE in a vendor request" in #103

[jfilak]

@bip91 I would not mind if you create a new factory method MS_CRM_client in the vendor module which would return pyodata client correctly configured.

[bip91]

Yes, of course.

I have built MSCRM vendor extension :
MSCRM.py.txt

This works for me with the following code :

import vendor.MSCRM
crm = vendor.MSCRM.Client(api_url_base,session)

# example to set a on guid id
crm.entity_sets.ContactSet.update_entity(id).set(**a).execute()

The endpoint has changed on more recent Dynamics CRM so the test on URL should be sufficient to avoid mistakes, but I cannot validate this (I have only Dynamics CRM 2015)

Feel free to adjust my extension to be coherent with the others.