Configuration Info For Catalogue Server Setup

[pachori91]

Hi Team,

I have setup Catalogue server(3.5.0) instance on my local system by following Installation Doc.

And setup for elk done by following Installation Doc.

When i am trying with openapi()swagger) url it is working fine but while i am trying
Get http://localhost:8083/cat/iudx/v1/item
Then getting error "Resource not found"

So my question is do i need to perform some other backend activity so that my catlogue server start to respond properly.

I am suspecting may be we need to create some entries in db before start to use. Because after following elk setup guid in elastic there is just user and role created regarding Catalogue server nothing else.

Please correct me if i am on wrong path.

Thanks
Deepak Kumar

[pachori91]

Hi Team,

I hope you guys might be busy but could you please reply on my above query it is affecting my timeline.

Thanks
Deepak Kumar

[abhi4578]

Hi Deepak kumar

thanks for raising issue.
The correct query is

http://localhost:8083/iudx/cat/v1/item?id=<cat-id>

Please see apis documentation /apis or at example at https://api.catalogue.iudx.org.in/apis#tag/Entity/operation/getItem

Having said, you will have to do two things to get any results in above query:

1. Cat index creation at elasticsearch
   A index to be created in elastic with mapping specified at https://github.com/datakaveri/iudx-catalogue-server/blob/3.5.0/configs/default-mappings.json.

   Index can be created in elasticsearch in two ways, follow any one:
   1.1. You can create index through kibana (accessible at https:// if deployed behind centralised nginx in swarm or need to expose the port locally through custom deployment, then it will be available at http://localhost:5601). Login as kibana admin user . Create a static cat index in elasticsearch through kibana . Then create a static cat index with name as "cat" in kibana by going to UI -> Dev tools in left dropdown using following the command in the console:

   PUT cat{   "settings": {    "index.number_of_replicas": "1",    "index.unassigned.node_left.delayed_timeout": "10m",    "number_of_shards": 1,    "index.mapping.coerce": false  }, "mappings": {        "dynamic": false,        "properties": {            "_summary": {                "type": "text"            },            "@context" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "id" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "tags" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 2000                    }                }            },            "type" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "instance" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "itemStatus" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "label" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "iudxResourceAPIs" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "accessPolicy" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "authControlLevel" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "dataDescriptor" : {                "type": "flattened"            },            "description" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "deviceInfo": {                "type": "flattened"            },            "location" : {                "properties" : {                    "address" : {                        "type" : "text",                        "fields" : {                            "keyword" : {                                "type" : "keyword",                                "ignore_above" : 256                            }                        }                    },                    "geometry" : {                        "type" : "geo_shape"                    },                    "type" : {                        "type" : "text",                        "fields" : {                            "keyword" : {                                "type" : "keyword",                                "ignore_above" : 256                            }                        }                    }                }            },            "name" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "providerOrg": {                "type": "flattened"            },            "accessObjectInfo": {                "type": "flattened"            },            "authServerInfo": {                "type": "flattened"            },            "resourceServerOrg": {                "type": "flattened"            },            "resourceServerStreamingAccessURL": {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "resourceServerHTTPAccessURL": {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "resourceGroup" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "resourceType" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "provider" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "resourceServer" : {                "type" : "text",                "fields" : {                    "keyword" : {                        "type" : "keyword",                        "ignore_above" : 256                    }                }            },            "_word_vector" : {                "type": "dense_vector",                "dims": 100            }        }    }}
   

   1.2. Through curl, you need to go inside the elastic container, using docker exec -it <cat-container-name> bash and then execute following curl command.

   curl -XPUT -u "<kibana-admin-username>:<kibana-admin-password>" "http://localhost:9200/cat-exp" -H 'Content-Type: application/json' -d '{"settings":{"index.number_of_replicas":"1","index.unassigned.node_left.delayed_timeout":"10m","number_of_shards":1,"index.mapping.coerce":false},"mappings":{"dynamic":false,"properties":{"_summary":{"type":"text"},"@context":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"tags":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":2000}}},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"instance":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"itemStatus":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"label":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"iudxResourceAPIs":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"accessPolicy":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"authControlLevel":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"dataDescriptor":{"type":"flattened"},"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"deviceInfo":{"type":"flattened"},"location":{"properties":{"address":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"geometry":{"type":"geo_shape"},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"providerOrg":{"type":"flattened"},"accessObjectInfo":{"type":"flattened"},"authServerInfo":{"type":"flattened"},"resourceServerOrg":{"type":"flattened"},"resourceServerStreamingAccessURL":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"resourceServerHTTPAccessURL":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"resourceGroup":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"resourceType":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"provider":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"resourceServer":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"_word_vector":{"type":"dense_vector","dims":100}}}}'
   

   1.3 After creation index named cat. Need to update in config with that name in field docIndex at

   iudx-catalogue-server/configs/config-example.json

   Line 14 in 64074dd

   "docIndex": "cattestindex",

   and

   iudx-catalogue-server/configs/config-example.json

   Line 32 in 64074dd

   "docIndex": "cattestindex",

   and redeploy the catalogue server.

2. Need to register Metadata About Resources on catalogue server (i.e. on board cat item for the resource )

After successful execution of both steps , you can query http://localhost:8083/iudx/cat/v1/item?id=<cat-id> with the above created <cat-id> . You will get successful response.

Including @rraks of IUDX team. Please add if anything more is needed

NOTE

1. Please deploy all components from 3.5.0 branch of deployment repo. As 3.5.0 rs, cat, auth is tested and will work properly only with backend components also deployed from 3.5.0 branch of deployment repo. If not, you might face some issues.
2. There will be detailed guide including these things shortly,

Hope this was useful. Let me know if you need anymore help,

Thanks and regards
Abhilash

[pachori91]

Hi @abhi4578,

Thanks for your reply, but i followed same steps before your reply and it is working fine.

Further i tried with Create item api to Upload metadata about my resource server it was working fine.

Further i tried with Create Item API to register Resource Group metadata with below payload -

{
"@context": "https://voc.iudx.org.in/",
"type": [
"iudx:ResourceGroup"
],
"description": "Gfiware of this resource group",
"name": "gfiwareresourcegroup",
"tags": [
"Gfiware", "echarge", "resourcegroup", "gfiwareresourcegroup"
],
"provider": "gmail.com/4963c4b2286fd33f9544d790f03757352f88a82b",
"resourceServer": "gmail.com/4963c4b2286fd33f9544d790f03757352f88a82b/localiudxresourceserver",
"resourceAuthControlLevel": "INDIVIDUAL",
"resourceType": "messageStream",
"iudxResourceAPIs": [
"attribute",
"temporal"
]
}
then i was getting error as response with message - Validation failed - Link Validation Failed

SO further i debug and found it is looking for corresponding provider in elastic with query -
{"_source": ["id"],"query": {"term": {"id.keyword": "gmail.com/4963c4b2286fd33f9544d790f03757352f88a82b"}}}

I tried to create provider with create item api but not able to understand how i will get access token so that i can create provider in catalogue server. As i found Auth Server's Create Token api does not support "provider" as value for itemType.

Can you please help me for same?

Thanks
Deepak Kumar

[pachori91]

Hi Team,

I hope you guys might be busy but could you please reply on my above query it is affecting my timeline.

Thanks
Deepak Kumar

[swaminathanvasanth]

Dear @pachori91

Thanks for writing to us. We would like to understand your organization and use case. Can you please let us know a bit about it? This will help us to guide you better.

Regards
Vasanth

[pachori91]

Hi @swaminathanvasanth,

I am from NECI and we are checking the possibility to ingest data from our local platform to IUDX. Just doing a POC.
Same is already shared with Abhay & Suresh on mail.

As it is opensource, does it really matters to know the organization name before answering my queries.

Thanks
Deepak Kumar

[rraks]

Hi @pachori91

The request you need to get a "provider" token which will allow you to insert Provider item into the catalogue is ->

curl --location -g --request POST '{{AUTH_ENDPOINT}}/auth/v1/token' \
--header 'Authorization: Bearer e..Q' \
--header 'Content-Type: application/json' \
--data-raw '{
    "itemId": "<provider-id>/<catalogue-domain-name>/catalogue/crud",
    "itemType": "resource",
    "role": "provider"
}'

Please note -
itemId = <provider-id>/<catalogue-domain-name>/catalogue/crud

[swaminathanvasanth]

Hi @swaminathanvasanth,

I am from NECI and we are checking the possibility to ingest data from our local platform to IUDX. Just doing a POC. Same is already shared with Abhay & Suresh on mail.

As it is opensource, does it really matters to know the organization name before answering my queries.

Thanks Deepak Kumar

Thanks for the response @pachori91

The reason to ask the organization name was to understand if you are in the consortium or in the partners list.
Though the code is open source (licensed under MIT), we are working on a support model which will be offered as a service/business. We are also constantly keeping our platform updated and our next release is due by 30th September.
Our team will help you out on getting things clarified. Due to release deadlines, please expect some delay in response.

For more queries, do touch base with me via email [ vasanth (dot) rajaraman (at) iudx (dot) org (dot) in]

Regards,
Vasanth

[naveennec]

Hello Vasanth,

Thanks for the reply.
we are seeking your cooperation.
I feel the problem is with the complex configuration and their related tutorials, both are not sync or difficult to understand. that's why we have several queries.
I can understand the release pressure, we also have the same pressure. But it would be great to have your support on time before its too late for us or for anyone who want to integrate with IUDX.

Thanks for your cooperation.
Best Regards
Naveen Singh Bisht

[rraks]

@naveennec @pachori91
We're happy to provide dedicated support through a video call.
Hope you were able to resolve the issue.

[naveennec]

@rraks
Thank you very much for your response and support.
we are able to setup catalogue server on our local, just we need help in resource server setup, which is almost done but still not complete.
if you can help in setting up the resource server then we can have a call on Monday(12 sept) as per your suitable time.

[pachori91]

Hi @rraks

One more query after your comment.

As we need to register Provider and Resource server to the catalogue if i am using Local IUDX Setup,
My Query is -
if i register myself as Provider on Publisher Portal by following Registration Doc.
Then can i expect below -

• IUDX Admin already Upload Metadata for Provider to the catalogue server?
• IUDX Admin already Upload Metadata about Resource Server to the Catalogue Server?
• Id for the Cataloge Server will be like organizationDomain(nec.com)/sha1 of provider email/catalogue.iudx.org.in?
• Id for the Resource Server will be like organizationDomain(nec.com)/sha1 of provider email/rs.iudx.org.in?

I am asking this as on my local setup as IUDX admin, I have to Upload metadata about Provider and Resource Server.

My Goal is to Upload Metadata Data about ResourceGroup/Resource and then Register Ingestion Adaptor as Provider.
and i am following below Doc-

• Register As provider
• Upload Metadata about ResourceGroup/Resource
• Register Data Ingestion Adaptor

please correct me if am wrong here.

Thanks
Deepak Kumar

[pachori91]

Hi Team,

I hope you guys might be busy but could you please reply on my above query.

Thanks
Deepak Kumar

[pachori91]

did you find some time to reply on my query?

[rraks]

Hi @pachori91
We often miss github notifications.
In case you don't receive a response, please mail me on [email protected]

Regarding your above query -
The idea is that provider themselves onboard the provider item onto the catalogue if possible (Either through apis or a UI based flow which is work in progress). Unfortunately, registering yourself as provider in the portal will not set you up with provider metadata on catalogue.
A resource server metadata item should be uploaded to the catalogue by the resource server admin by obtaining a token from the auth server using their own credentials.
As an alternative, a provider/resourceserveradmin could also delegate such responsibilities to someone having an onboarder role.

All of these flows are created keeping a UI based flow in mind. The UI work is still in progress.

I'm looping in @m-arun who can help you out on the delegation flows.
His email is [email protected]