Crawler transform

[touma-I]

Why are these changes needed?

Implement crawler transforms using the dpi-connector API. This is based on the work done by the data sift but also had to add CLI in order to integrate with python runtime. This implementation uses the new layout for the transform using module name dpk_web2parquet

Related issue number (if any).

#751

[hmtbr]

@touma-I Thank you very much for making this change! This simple implementation looks good to me. I added several comments, but most of them are nitpicking.

[hmtbr]

Q: How does this DAO handle a file which has the same filename as one saved in the folder before? e.g. We crawl test.com and get test.com/path1/doc.pdf and test.com/path2/doc.pdf. It looks the two have the same filename value: doc.pdf.

[touma-I]

@hmtbr I think we are using the full URL path to form the filename (i.e. the filename for test.com/path1/doc.pdf becomes path1_doc.pdf and test.com/path2/doc.pdf becomes path2_doc.pdf). This filename generation is not handled by the DAO but by the transform itself.

[hmtbr]

Yes, it should work well when we build a filename from the request URL. My concern is the filename when we build it from the value of the Content-Disposition header. If the two URLs I gave return the same Content-Disposition header value, it looks the two have the same filename as a result. We may also want to add the path info to the filename in the same way as the case of no Content-Disposition. Correct me if I'm wrong.

[touma-I]

@hmtbr I don't know if we should over-engineer this one. If that situation occurs, we can address it based on a concrete example.