Skip to content

DanielLyons178/clustered-crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Scraper

Distributed and pluggable python web scraper

TODO

  • Restructure queuing
  • Allow scaling engines
    • Write state of master engine to redis (exists)
    • Joining message sent to fanout exchange
    • Ouptput queue registrations to redis (from master engine) for addition of more engines

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors