create FB bot that will answer qs about Nepal data

[ravinepal]

is it possible that we program a fb bot in a way that would respond to user's data related questions if there's info on NepalMap?

if it's useful here's white house fb chatbot: https://github.com/WhiteHouse/fb_messenger_bot

[prashishh]

This is a great idea @ravinepal. I will do some research on this.

Love it.

[ravinepal]

@prashishh -- i wanted to see if you got chance to look into this or ask other c4n members to do so

[ravinepal]

@prashishh it'd be great have a team working on this? We can give them access to our FB page and they can start working on this :)

[ravinepal]

I think the MVP of FB bot could just simply direct people to the link of certain district, VDC or municipality.

for example, when someone asks how many people have education in Kavre.
Bot could simply respond by saying "Glad you are interested in Kavre. Here's the info: http://www.nepalmap.org/profiles/district-24-kavrepalanchok/"

If someone asks about a neighborhood and we don't have that location in our database, FB bot could respond by saying:

"Hmm can you tell me which VDC, Municipality or district, your location is in, and I can find related data for you."

I believe this FB bot could be very useful especially to students, and reporters.

Looping @Studenton and @cliftonmcintosh in the conversation for their thoughts, and see if they know someone who would be interested in doing this!

[mesaugat]

I have a suggestion. If you don't want to code all of this there's a better solution.
https://chatfuel.com - Build a Facebook bot without coding.

Get your bot up and running in a few minutes, anyone can do it.

[ksodari]

ofcourse its possible way to do... we can train our chatbot ourselves on how it will respond to our users when they input certain keywords.

[prashishh]

@mesaugat @ksodari - Thanks for the heads up.

I went through ChatFuel and it was fairly easy to set up a basic chatbot. I think it is great for a tree-based conversation for Code for Nepal, but I'm curious if you have worked on integrating this with a datastore? There is an integration with Google Sheets or a JSON API. They can act like a data store if someone searches for the term - tell me about "a district" / what is the population of "a district" (advanced).

The census data are stored in the form of csv in this repo (https://github.com/Code4Nepal/data/tree/master/datasets/census). Do you think we can save them in an easy-to-deploy-and-test datastore to fetch these information for the chatbot?

[mesaugat]

Not worked with the JSON API in Chatfuel but just had a quick look. Only having the dataset won't do us any good. We'll need to create an API of our our own for the dataset so that we can query our API when someone asks about the census for a district.

For Example:

If someone asks: What is the census of Siraha?
We could setup Chatfuel to query our API: http://api.ourserver/census/siraha or /census?district='siraha' which could return a JSON response like:

{
  "district": "Siraha",
  "population": 637328,
  "year": 2011
}

and return a text to the user.

Bot: The census of Siraha as of 2011 is 637328

We could change this JSON response as Chatfuel wants it to be, mentioned here. Even if we were creating our own bot we would be doing something similar.

Hopefully, this helps.

[ravinepal]

thanks, guys! let me bring @cliftonmcintosh to the convo as he is the expert on NepalMap and can perhaps guide us on how to efficiently create an API for NepalMap.

[cliftonmcintosh]

I'm not sure what it would take to expose an API. I glanced through the Wazimap code and don't see anything, but Census Report, on which Wazimap is based, has an api: https://github.com/censusreporter/census-api

I'm not sure how it is used since both Census Reporter and Wazimap seem to deliver templated HTML views for each request. A JSON API is likely not something that comes with our framework.

[vksbhandary]

Hello everyone. I have been following this issue. And i guess chatfuel isn't the best option available given it has free usage limit.
I would like to suggest another option API.ai. I don't have much working experience with this, but i have created very simple bot using this with in few minutes. (Its name is Open data nepal I have hard coded to give population of Kathmandu only, without any coding! )

The main advantage of api.ai here are

• It is more extensible (see full list of integrations here)
• We can host our own bot (see instructions )
• And its best free NLP service (see Pricing)

One thing to note here. For any chatbot framework to show dynamic response, we will need api (or data access code to feed to bot). As mentioned by @mesaugat
Here is a sample webhook implementation in python. Something similar can be used in our situation for dynamic replies.

I don't know anything about Wazimap or Census report. But I guess, we can add REST APIs to django using REST framework. Technically its Django right? @cliftonmcintosh

[mesaugat]

The problem is not Chatfuel. It's free as well unless you hit too many users.

Chatfuel’s bot-building platform is free. 

If you plan on hitting more than 500,000 monthly active users, please talk to us at [email protected].

Yes, https://api.ai/ is great. It has a lot of integrations. But the ease of Chatfuel is big, anyone can configure anything. No wonder it's getting popular. You don't need to deploy a bot unlike api.ai and other services. If we are planning for custom use cases then we can definitely go for api.ai. That being said we can choose any service.

The pain point here is we don't have an API to query the census data. From what I see we already have various census data's in CSV format. We just need to feed that data into a database and create a fairly simple API which should not take time.

The census API doesn't not even need to be in this repository. It could be a simple app in any language deployed somewhere and we just query from it.

UPDATE:

As @ravinepal mentioned above:

For example, when someone asks how many people have education in Kavre.
Bot could simply respond by saying "Glad you are interested in Kavre. Here's the info: http://www.nepalmap.org/profiles/district-24-kavrepalanchok/"

If someone asks about a neighborhood and we don't have that location in our database, FB bot could respond by saying:

"Hmm can you tell me which VDC, Municipality or district, your location is in, and I can find related data for you."

I think we don't even need an API for doing something like this. Just map a location name with their corresponding Nepal Map URL in Chatfuel. It should be fairly simple? @prashishh

That means: 'Kathmandu' will be mapped as http://www.nepalmap.org/profiles/district-27-kathmandu/ in Chatfuel. I think this is possible.

If someone asks: Census of kathmandu?
The bot can simply reply with a text card: Here you go: http://www.nepalmap.org/profiles/district-27-kathmandu/

That's seems to be what we need, no?

[amitness]

I explored how Wazimap library is developed internally in venv/lib/python2.7/site-packages/wazimap/views.py and found that Wazimap already implements the census-reporter API and gives a endpoint. So I can access the projected population of Kathmandu for 2031 as

http://nepalmap.org/api/1.0/data/show/latest?table_ids=POPULATION_PROJECTION_2031&geo_ids=district-27,country-NP

It gives back a JSON response.

{
   "release":{
      "name":"National Population and Housing Census 2011",
      "years":"2011"
   },
   "tables":{
      "POPULATION_PROJECTION_2031":{
         "title":"Projected population in 2031",
         "universe":"Population",
         "denominator_column_id":"total",
         "table_id":"POPULATION_PROJECTION_2031",
         "stat_type":"number",
         "columns":{
            "total":{
               "indent":0,
               "name":"Total"
            },
            "female":{
               "indent":1,
               "name":"Female"
            },
            "male":{
               "indent":1,
               "name":"Male"
            }
         }
      }
   },
   "data":{
      "district-27":{
         "POPULATION_PROJECTION_2031":{
            "estimate":{
               "total":2729056,
               "male":1469787,
               "female":1259269
            },
            "error":{
               "total":0,
               "male":0,
               "female":0
            }
         }
      },
      "country-NP":{
         "POPULATION_PROJECTION_2031":{
            "estimate":{
               "total":33597033,
               "male":16313189,
               "female":17283844
            },
            "error":{
               "total":0,
               "male":0,
               "female":0
            }
         }
      }
   },
   "geography":{
      "district-27":{
         "geo_code":"27",
         "child_level":"vdc",
         "short_name":"Kathmandu",
         "name":"Kathmandu",
         "full_name":"Kathmandu",
         "parent_geoid":"country-NP",
         "full_geoid":"district-27",
         "square_kms":null,
         "geo_level":"district"
      },
      "country-NP":{
         "geo_code":"NP",
         "child_level":"district",
         "short_name":"Nepal",
         "name":"Nepal",
         "full_name":"Nepal",
         "parent_geoid":null,
         "full_geoid":"country-NP",
         "square_kms":null,
         "geo_level":"country"
      }
   }
}

[cliftonmcintosh]

That's an interesting find, @Studenton

[ravinepal]

Thank you, @mesaugat, @vksbhandary, @cliftonmcintosh. and special thanks to @Studenton for that find :) We have a prototype of a bot that @ksodari and I created using Chatfuel - let me know if you guys would be interested in accessing it or contributing to it. If you just want to test it, type "Code for Nepal" in your FB messenger and let us know what you think

@ksodari - can we use text card to map URLs as @mesaugat is suggesting?

[mesaugat]

Great!

Maybe not. Text cards only seem to support texts. There's a gallery card that definitely supports a URL.

[ksodari]

@ravinepal yes Sir! Gallery card can be used very efficiently on this as mentioned by @mesaugat .
Thank you all for the findings and suggestions. It should be now more smooth in upgrading the chat bot "Shanti" :) I am trying onto this.
@cliftonmcintosh I would like to know about how data are maintained on our website Nepalmap??

[ravinepal]

hi @cliftonmcintosh - grateful if you could kindly advise @ksodari. pls see his comment #118 (comment)

[cliftonmcintosh]

@ksodari

Apologies for missing your question. Data in NepalMap is kept in a postgres database on the same server as the webapp. The database is not exposed externally.

When new data points are added, tables are created and data is updated via SQL statements.

The SQL statements are here.

Does that provide you with the information you need?

[ravinepal]

thanks, @cliftonmcintosh. @ksodari does that help? @amitness - would you have time to help @ksodari build API?

[amitness]

Sure @ravinepal @ksodari . How are you guys coordinating on this and what's the current status?

[ravinepal]

thanks, @amitness! So we have a prototype "Code for Nepal" bot - named as "Shanti". You can check it out on your fb messenger. i think we need to create API or figure out a way to use gallery in chatfuel card that definitely supports a URL. @ksodari - can you pls provide more updated to Amit

[ravinepal]

hi everyone, it'd be great to restart this project. @ksodari - happy to chat if you are still interested.

[wizofe]

hi @ravinepal. would you provide me on more info on what enhancements you need for your bot? i can work with AI and Markov models, if that's on your scope.

[ravinepal]

thanks a lot, @wizofe! We have a prototype of a bot that we created using Chatfuel. If you just want to test it, type "Code for Nepal" in your FB messenger and let us know what you think. Or test via our page by sending us a message: https://www.facebook.com/codefornepal/

Basically, we are trying to make a bot respond to users' queries about data on Nepal. We have tons of Nepal data here: https://nepalmap.org

I think the MVP of FB bot could just simply direct people to the link of certain district, VDC or municipality on https://nepalmap.org

for example, when someone asks how many people have education in Kavre.
Bot could simply respond by saying "Glad you are interested in Kavre. Here's the info: http://www.nepalmap.org/profiles/district-24-kavrepalanchok/"

If someone asks about a neighborhood and we don't have that location in our database, FB bot could respond by saying:

"Hmm can you tell me which VDC, Municipality or district, your location is in, and I can find related data for you."

I believe this FB bot could be very useful especially to students, and reporters.

Does this help?

I'm also open to other ideas!

[rajsanjib]

You can easily create AIML structures to best match user queries and then respond with the data you have instead of providing a link.
Bot would repond to query like "What is the population of Rasuwa?" with "The population of Rasuwa district is 43,300". PandoraBot uses AIML to create chatbots.

I built a similar bot for Kathmandu University. https://github.com/rajsanjib/KU-Bot
It would mainly provide answers to dates of admission, entrance, course syllabus etc.

[ravinepal]

@aishraj and @spaudel -- flagging this to you. grateful if you someone from Bay area could help with this if possible!

[aishraj]

@ravinepal Sure! Currently most of us here in the Bay Area have been working around the Akshar digital literature portal. But I remember a few folks expired desires work on this in the past. I'll double check if anyone is still interested

[harsh2ai]

well i think creating a chatbot is a nice idea but n facebook, they do have a platform called chatfuel which is intended for creation of chatbots and linking the to a particular page on facebook. fb does charge on that but trust me its worth it because facebook would itself want to improve in every possible day, it is the biggest step for creating chatbots without any coding