Например:
scrapy shell http://scrapy.org/
content = hxs.select('//*[@id="content"]').extract()[0]
print content
Затем я получаю следующий необработанный HTML-код:
<div id="content">
<h2>Welcome to Scrapy</h2>
<h3>What is Scrapy?</h3>
<p>Scrapy is a fast high-level screen scraping and web crawling
framework, used to crawl websites and extract structured data from their
pages. It can be used for a wide range of purposes, from data mining to
monitoring and automated testing.</p>
<h3>Features</h3>
<dl>
<dt>Simple</dt>
<dt>
</dt>
<dd>Scrapy was designed with simplicity in mind, by providing the features
you need without getting in your way
</dd>
<dt>Productive</dt>
<dd>Just write the rules to extract the data from web pages and let Scrapy
crawl the entire web site for you
</dd>
<dt>Fast</dt>
<dd>Scrapy is used in production crawlers to completely scrape more than
500 retailer sites daily, all in one server
</dd>
<dt>Extensible</dt>
<dd>Scrapy was designed with extensibility in mind and so it provides
several mechanisms to plug new code without having to touch the framework
core
</dd>
<dt>Portable, open-source, 100% Python</dt>
<dd>Scrapy is completely written in Python and runs on Linux, Windows, Mac and BSD</dd>
<dt>Batteries included</dt>
<dd>Scrapy comes with lots of functionality built in. Check <a
href="#" onclick="location.href='http://doc.scrapy.org/en/latest/intro/overview.html#what-else'; return false;">this
section</a> of the documentation for a list of them.
</dd>
<dt>Well-documented & well-tested</dt>
<dd>Scrapy is <a href="/doc/">extensively documented</a> and has an comprehensive test suite
with <a href="#" onclick="location.href='http://static.scrapy.org/coverage-report/'; return false;">very good code
coverage</a></dd>
<dt><a href="/community">Healthy community</a></dt>
<dd>
1,500 watchers, 350 forks on Github (<a href="#" onclick="location.href='https://github.com/scrapy/scrapy'; return false;">link</a>)<br>
700 followers on Twitter (<a href="#" onclick="location.href='http://twitter.com/ScrapyProject'; return false;">link</a>)<br>
850 questions on Qaru (<a href="#" onclick="location.href='http://stackoverflow.com/tags/scrapy/info'; return false;">link</a>)<br>
200 messages per month on mailing list (<a
href="#" onclick="location.href='https://groups.google.com/forum/?fromgroups#!aboutgroup/scrapy-users'; return false;">link</a>)<br>
40-50 users always connected to IRC channel (<a href="#" onclick="location.href='http://webchat.freenode.net/?channels=scrapy'; return false;">link</a>)
</dd>
<dt><a href="/support">Commercial support</a></dt>
<dd>A few companies provide Scrapy consulting and support</dd>
<p>Still not sure if Scrapy is what you're looking for?. Check out <a
href="#" onclick="location.href='http://doc.scrapy.org/en/latest/intro/overview.html'; return false;">Scrapy at a
glance</a>.
</p>
<h3>Companies using Scrapy</h3>
<p>Scrapy is being used in large production environments, to crawl
thousands of sites daily. Here is a list of <a href="/companies/">Companies
using Scrapy</a>.</p>
<h3>Where to start?</h3>
<p>Start by reading <a href="#" onclick="location.href='http://doc.scrapy.org/en/latest/intro/overview.html'; return false;">Scrapy at a glance</a>,
then <a href="/download/">download Scrapy</a> and follow the <a
href="#" onclick="location.href='http://doc.scrapy.org/en/latest/intro/tutorial.html'; return false;">Tutorial</a>.
</p></dl>
</div>
Но я хочу получить простой текст прямо из скрапа.
Я не хочу использовать селекторы xPath для извлечения тегов p, h2, h3..., поскольку я сканирую веб-сайт, основное содержание которого встроено в table, tbody; рекурсивно. Найти xPath может быть утомительным занятием.
Может ли это быть реализовано встроенной функцией в Scrapy? Или мне нужны внешние инструменты для его преобразования? Я прочитал все документы Scrapy, но ничего не получил.
Это пример сайта, который может конвертировать необработанный HTML в простой текст: http://beaker.mailchimp.com/html-to-text
