python爬取网页数据,Python获取网页内容

我建议使用BeautifulSoup来解析和搜索html。这将比进行基本的字符串搜索容易得多。在

下面是一个示例，它提取了在包含Legal Authority:标记中找到的所有标记。(请注意，我使用requests库来获取页面内容-这只是一个推荐的、非常容易使用的替代urlopen。)import requests

from BeautifulSoup import BeautifulSoup

# fetch the content of the page with requests library

url = "http://www.reginfo.gov/public/do/eAgendaViewRule?pubId=200210&RIN=1205-AB16"

response = requests.get(url)

# parse the html

html = BeautifulSoup(response.content)

# find all the tags

a_tags = html.findAll('a', attrs={'class': 'pageSubNavTxt'})

def fetch_parent_tag(tags):

# fetch the parent

# whose "previous sibling" is the Legal Authority: tag.

for tag in tags:

sibling = tag.findPreviousSibling()

if not sibling:

continue

if sibling.getText() == 'Legal Authority:':

return tag.findParent()

# now, just find all the child

# i.e. finding the parent of one child, find all the children

parent_tag = fetch_parent_tag(a_tags)

tags_you_want = parent_tag.findAll('a')

for tag in tags_you_want:

print 'statute: ' + tag.getText()

如果这不是您需要做的，那么BeautifulSoup仍然是您可能希望用来筛选html的工具。在