When learning Python, most of us probably started with web scraping. After all, there is plenty of similar material online and a great many open-source projects too.
Learning web scraping in Python breaks down into 3 big areas: fetching, parsing, and storing.
When we type a url into the browser and hit enter, what happens behind the scenes?
Put simply, these four steps take place:
Look up the IP address corresponding to the domain name.
Send a request to the server at that IP.
The server responds to the request and sends the page content back.
The browser parses the page content.
So which libraries do you need to master to learn scraping?
General:
- urllib - networking library (stdlib).
- requests - networking library.
- grab – networking library (built on pycurl).
- pycurl – networking library (libcurl bindings).
- urllib3 – Python HTTP library with secure connection pooling, file post support and high usability.
- httplib2 – networking library.
- RoboBrowser – a simple, thoroughly Pythonic Python library that browses web pages without a separate browser.
- MechanicalSoup - a Python library for interacting with websites automatically.
- mechanize - a stateful, programmable web browsing library.
- socket – low-level networking interface (stdlib).
- Unirest for Python – Unirest is a set of lightweight HTTP libraries available for many languages.
- hyper – an HTTP/2 client for Python.
- PySocks – the updated and actively maintained version of SocksiPy, including bug fixes and some other features. A drop-in replacement for the socket module.
Web scraping frameworks
Full-featured scrapers
- grab – web scraping framework (based on pycurl/multicur)
- scrapy – web scraping framework (based on twisted), does not support Python 3.
- pyspider – a powerful crawling system.
- cola – a distributed crawling framework.
Others
- portia – a visual scraper based on Scrapy.
- restkit – an HTTP resource toolkit for Python. It lets you access HTTP resources easily and build objects around them.
- demiurge – a micro scraping framework based on PyQuery.
HTML/XML parsers
General
- lxml – an efficient HTML/XML processing library written in C. Supports XPath.
- cssselect – parses DOM trees and CSS selectors.
- pyquery – parses DOM trees and jQuery selectors.
- BeautifulSoup – an inefficient HTML/XML processing library, implemented in pure Python.
- html5lib – generates a DOM from HTML/XML documents according to the WHATWG spec. That spec is used by every browser today.
- feedparser – parses RSS/ATOM feeds.
- MarkupSafe – provides safely escaped strings for XML/HTML/XHTML.
- xmltodict – a Python module that makes handling XML feel like handling JSON.
- xhtml2pdf – converts HTML/CSS to PDF.
- untangle – makes converting XML files into Python objects easy.
Cleaning
- Bleach – cleans HTML (requires html5lib).
- sanitize – brings clarity to the messy world of data.
Text processing
- Libraries for parsing and manipulating simple text.
- General
- difflib – (Python standard library) helps with diff comparison.
- Levenshtein – quickly computes Levenshtein distance and string similarity.
- fuzzywuzzy – fuzzy string matching.
- esmre – a regular expression accelerator.
- ftfy – automatically tidies up Unicode text and reduces fragmentation.
- General
Natural language processing
- Libraries for dealing with human language problems.
- NLTK - the best platform for writing Python programs that work with human language data.
- Pattern – a web mining module for Python. It has natural language processing tools, machine learning and more.
- TextBlob – provides a consistent API for deep natural language processing tasks. It is built on the shoulders of the giants NLTK and Pattern.
- jieba – a Chinese word segmentation tool.
- SnowNLP – a Chinese text processing library.
- loso – another Chinese word segmentation library.
- Browser automation and emulation
- selenium – automates real browsers (Chrome, Firefox, Opera, IE).
- Ghost.py – a wrapper around PyQt’s webkit (requires PyQT).
- Spynner – a wrapper around PyQt’s webkit (requires PyQT).
- Splinter – a generic API browser emulator (selenium web drivers, Django client, Zope).
- Multiprocessing
- threading – runs threads from the Python standard library. Very effective for I/O-bound tasks. Useless for CPU-bound tasks because of the python GIL.
- multiprocessing – the standard Python library for running multiple processes.
- celery – an asynchronous task queue/job queue based on distributed message passing.
- concurrent-futures – the concurrent-futures module provides a high-level interface for asynchronously executing calls.
Asynchronous
- Asynchronous network programming libraries
- asyncio – (Python standard library in Python 3.4+) asynchronous I/O, event loop, coroutines and tasks.
- Twisted – an event-driven networking engine framework.
- Tornado – a web framework and asynchronous networking library.
- pulsar – an event-driven concurrency framework for Python.
- diesel – a greenlet-based I/O framework for Python.
- gevent – a coroutine-based Python networking library that uses greenlet.
- eventlet – an asynchronous framework with WSGI support.
- Tomorrow – marvellous decorator syntax for asynchronous code.
- Queues
- celery – an asynchronous task queue/job queue based on distributed message passing.
- huey – a small multithreaded task queue.
- mrq – Mr. Queue – a distributed Python job queue using redis & Gevent.
- RQ – a lightweight task queue manager based on Redis.
- simpleq – a simple, infinitely scalable queue based on Amazon SQS.
- python-gearman – the Python API for Gearman.
Cloud computing
- picloud – run Python code in the cloud.
- dominoup.com – run R, Python and matlab code in the cloud
Web content extraction
- Text and metadata from HTML pages
- newspaper – news extraction, article extraction and content curation in Python.
- html2text – converts HTML to Markdown text.
- python-goose – an HTML content/article extractor.
- lassie – a human-friendly web content retrieval tool
WebSocket
- Libraries for WebSocket.
- Crossbar – an open-source application messaging router (Python implementations of WebSocket and WAMP for Autobahn).
- AutobahnPython – provides open-source Python implementations of the WebSocket and WAMP protocols.
- WebSocket-for-Python – a WebSocket client and server library for Python 2 and 3 as well as PyPy.
- DNS resolution
- dnsyo – check your DNS against more than 1500 DNS servers around the world.
- pycares – an interface to c-ares. c-ares is a C library for DNS requests and asynchronous name resolution.
Computer vision
- OpenCV – an open-source computer vision library.
- SimpleCV – a concise, highly readable interface for cameras, image processing, feature extraction and format conversion (based on OpenCV).
- mahotas – fast computer image processing algorithms (implemented entirely in C++), using numpy arrays as its data type.
Proxy servers
- shadowsocks – a fast tunnel proxy that helps you get through firewalls (supports TCP and UDP, TFO, multi-user and smooth restart, destination IP blacklist).
- tproxy – tproxy is a simple TCP routing proxy (layer 7), built on Gevent and configured in Python.
Some web development frameworks
1.Django
Django is an open-source web application framework written in Python. It supports many database engines, makes web development fast and scalable, and is continuously version-updated to match the latest Python releases. If you’re a new programmer, this is a good framework to start with.
2.Flask
Flask is a lightweight web application framework written in Python. It is based on the Werkzeug WSGI toolkit and the Jinja2 template engine. It uses the BSD license.
Flask is also called a “microframework” because it uses a simple core and adds other features through extensions. Flask has no default database or form validation tools. However, Flask stays flexible about extension: Flask-extension can add these features — ORM, form validation tools, file uploads, and various open authentication technologies.
3.Web2py
Web2py is a free, open-source web framework written in the Python language, aimed at agile, rapid development of web applications, with fast, scalable, secure and portable database-driven applications, following the LGPLv3 open-source license.
Web2py provides a one-stop solution: the entire development process can be done in the browser, offering web-based online development, HTML template editing, static file uploads and database editing. On top of that there is logging and an automated admin interface.
4.Tornado
Tornado is both a Web server (not described in detail in this article) and a micro-framework in the style of web.py. As a framework, Tornado’s ideas come mainly from Web.py; on the front page of the Web.py site you can also see this passage from Tornado’s big name Bret Taylor (the framework FriendFeed uses that he mentions here can be regarded as the same thing as Tornado):
“[web.py inspired the] Web framework we use at FriendFeed [and] the webapp framework that ships with App Engine…”
Because of that relationship, Tornado won’t be discussed separately later on.
5.CherryPy
CherryPy is a simple yet very useful web framework for Python. Its main job is to connect a web server with Python code using as few operations as possible; its features include built-in profiling, a flexible plugin system, and the ability to run multiple HTTP servers at once. It can run on the latest versions of Python, Jython and Android.
Echarts chart library
- Bar (bar/column chart)
- Bar3D (3D bar chart)
- Boxplot (box plot)
- EffectScatter (scatter chart)
- Funnel (funnel chart)
- Gauge (gauge)
- Geo (geographic coordinate system)
- Graph (relationship graph)
- HeatMap (heat map)
- Kline (candlestick chart)
- Line (line/area chart)
- Line3D (3D line chart)
- Liquid (liquid fill chart)
- Map (map)
- Parallel (parallel coordinate system)
- Pie (pie chart)
- Polar (polar coordinate system)
- Radar (radar chart)
- Sankey (sankey diagram)
- Scatter (scatter chart)
- Scatter3D (3D scatter chart)
- ThemeRiver (theme river chart)
- WordCloud (word cloud)
Original source: https://mp.weixin.qq.com/s/6SJYooBuBhzsibeXS1mvWQ
