Skip to content

@TeamHG-Memex TeamHG-Memex

Change the repository type filter

All

    Repositories list

    74 repositories

    • Formasaurus

      Public
      Formasaurus tells you the type of an HTML form and its fields using machine learning
      HTML
      •47•116•12•2•Updated Jun 18, 2024Jun 18, 2024

    • sklearn-crfsuite

      Public
      scikit-learn inspired API for CRFsuite
      Python
      •216•426•34•12•Updated Sep 25, 2023Sep 25, 2023

    • agnostic

      Public
      Agnostic Database Migrations
      Python
      •
      MIT License
      •18•52•8•1•Updated Aug 10, 2023Aug 10, 2023

    • soft404

      Public
      A classifier for detecting soft 404 pages
      Jupyter Notebook
      •14•56•3•5•Updated Jul 6, 2023Jul 6, 2023

    • tensorboard_logger

      Public
      Log TensorBoard events without touching TensorFlow
      Python
      •
      MIT License
      •52•630•9•4•Updated Dec 26, 2022Dec 26, 2022

    • arachnado

      Public
      Web Crawling UI and HTTP API, based on Scrapy and Tornado
      Python
      •65•161•16•5•Updated Nov 4, 2022Nov 4, 2022

    • scrapy-crawl-once

      Public
      Scrapy middleware which allows to crawl only new content
      scrapy
      Python
      •
      MIT License
      •23•79•3•3•Updated Oct 31, 2022Oct 31, 2022

    • autologin

      Public
      A project to attempt to automatically login to a website given a single seed
      Python
      •
      Apache License 2.0
      •44•123•9•5•Updated Jul 29, 2022Jul 29, 2022

    • scrapy-rotating-proxies

      Public
      use multiple proxies with Scrapy
      proxyscrapy
      Python
      •
      MIT License
      •158•737•45•8•Updated May 20, 2022May 20, 2022

    • eli5

      Public
      A library for debugging/inspecting machine learning classifiers and explaining their predictions
      pythonnlpdata-sciencemachine-learningscikit-learnxgboostlightgbminspectionexplanationcrfsuite
      Jupyter Notebook
      •
      MIT License
      •332•2.8k•144•19•Updated May 1, 2022May 1, 2022

    • url-summary

      Public
      Show summary of a large number of URLs in a Jupyter Notebook
      Python
      •
      MIT License
      •9•17•0•1•Updated Jun 8, 2021Jun 8, 2021

    • sitehound-frontend

      Public
      Site Hound (previously THH) is a Domain Discovery Tool
      topicsdomain-discovery
      HTML
      •
      Apache License 2.0
      •13•23•2•4•Updated Jun 1, 2021Jun 1, 2021

    • autopager

      Public
      Detect and classify pagination links
      HTML
      •25•98•6•0•Updated Sep 9, 2020Sep 9, 2020

    • html-text

      Public
      Extract text from HTML
      HTML
      •
      MIT License
      •24•130•13•2•Updated Jul 22, 2020Jul 22, 2020

    • docker-tor-rotator

      Public
      A rotating socks proxy using Tor, Delegate and Haproxy
      Dockerfile
      •16•14•1•0•Updated Dec 19, 2019Dec 19, 2019

    • aquarium

      Public
      Splash + HAProxy + Docker Compose
      Python
      •
      MIT License
      •41•198•24•0•Updated Nov 29, 2018Nov 29, 2018

    • json-lines

      Public
      Read JSON lines (jl) files, including gzipped and broken
      Python
      •
      MIT License
      •9•34•2•0•Updated Nov 21, 2018Nov 21, 2018

    • scrapy-kafka-export

      Public
      Scrapy extension which writes crawled items to Kafka
      Python
      •
      MIT License
      •9•30•2•0•Updated Nov 8, 2018Nov 8, 2018

    • scrapy-cdr

      Public
      Item definition and utils for storing items in CDR format for scrapy
      Python
      •
      MIT License
      •6•7•0•0•Updated Oct 29, 2018Oct 29, 2018

    • sitehound-backend

      Public
      Sitehound's backend
      HTML
      •
      Apache License 2.0
      •5•6•0•0•Updated Oct 17, 2018Oct 17, 2018

    • autologin-middleware

      Public
      Scrapy middleware for the autologin
      Python
      •15•37•4•1•Updated May 29, 2018May 29, 2018

    • undercrawler

      Public
      A generic crawler
      Python
      •25•78•17•0•Updated May 29, 2018May 29, 2018

    • domain-discovery-crawler

      Public
      Broad crawler for domain discovery
      Python
      •
      MIT License
      •10•19•2•0•Updated May 29, 2018May 29, 2018

    • page-compare

      Public
      Simple heuristic for measuring web page similarity (& data set)
      HTML
      •18•89•1•0•Updated May 29, 2018May 29, 2018

    • hh-page-classifier

      Public
      Headless Horseman Page Classifier service
      Python
      •
      MIT License
      •6•7•0•0•Updated May 29, 2018May 29, 2018

    • deep-deep

      Public
      Adaptive crawler which uses Reinforcement Learning methods
      Jupyter Notebook
      •38•170•0•0•Updated May 29, 2018May 29, 2018

    • scrash-lua-examples

      Public
      A collection of example LUA scripts and JS utilities
      JavaScript
      •4•7•0•0•Updated May 29, 2018May 29, 2018

    • MaybeDont

      Public
      A component that tries to avoid downloading duplicate content
      Python
      •
      MIT License
      •14•27•2•0•Updated May 29, 2018May 29, 2018

    • sitehound

      Public
      This is the facade for installation and access to the individual components
      Shell
      •
      Apache License 2.0
      •8•16•0•0•Updated May 29, 2018May 29, 2018

    • sshadduser

      Public
      A simple tool to add a new user with OpenSSH keys.
      Python
      •
      MIT License
      •1•2•0•0•Updated May 29, 2018May 29, 2018