Concept

Concept

     Let's talk about the concept of Pider framework in this section.

Why create Pider ?

     There are lots of very excellent open sourse crawler frameworks existed. So, Why do I still want to recreate one?
     First, it's indeed that a bunch of excellent alike frameworks has existed.But most of them is in Java, Python. So it's valuable to develop a crawler framework in PHP for php developers.And what's more, all of these frameworks don't concerns process except data crawling. All of those drive me to develop this framework.

What Pider should be ?

     Although, Pider is a rough artifact current. But it contains a lot of my expectations. It appeals me to complete, just like an artwork still need to be polished by artists.

• Cover all fields of data processing (crawling,prepost,anylysis,visualization,report)
• apply new technology (Like machine learning , block chain) in data anaysis or curacy.
• Good compatibility (with scrapy,and some others)
• Distributed
• Abundant terminal interaction support
• Abundant webui ineraction support
• Remote control and shedule

Thanks

    The design and implements of Pider framework derived from many open source framework.I want to offer my admiration and thanks to them at here(included but not limited):

• scrapy
• scrapyd
• faker-useragent
• Rcrawler