本文将从多个方面详细阐述Python爬取文本内容的方法和技巧。
一、爬取网页内容
1、使用requests库发送HTTP请求
import requests
response = requests.get('https://www.example.com')
print(response.text)
2、使用BeautifulSoup库解析网页内容
import requests
from bs4 import BeautifulSoup
response = requests.get('https://www.example.com')
soup = BeautifulSoup(response.text, 'html.parser')
print(soup.get_text())
3、使用正则表达式提取网页中的文本内容
import re
text = 'Hello World
'
result = re.findall(r'(.*?)
', text)
print(result)
二、爬取API接口返回的文本内容
1、使用requests库发送HTTP请求
import requests
response = requests.get('https://api.example.com/text')
print(response.text)
2、解析JSON格式的文本内容
import requests
import json
response = requests.get('https://api.example.com/text')
data = json.loads(response.text)
print(data['text'])
3、使用正则表达式提取API返回的文本
import requests
import re
response = requests.get('https://api.example.com/text')
text = response.text
result = re.findall(r'"text":"(.*?)"', text)
print(result)
三、保存爬取的文本内容
1、将文本内容保存到本地文件
import requests
response = requests.get('https://www.example.com')
text = response.text
with open('output.txt', 'w', encoding='utf-8') as file:
file.write(text)
2、将文本内容保存到数据库
import requests
import sqlite3
response = requests.get('https://www.example.com')
text = response.text
conn = sqlite3.connect('database.db')
cursor = conn.cursor()
cursor.execute('CREATE TABLE IF NOT EXISTS text_content (id INTEGER PRIMARY KEY, content TEXT)')
cursor.execute('INSERT INTO text_content (content) VALUES (?)', (text,))
conn.commit()
conn.close()
3、将文本内容保存到Excel文件
import requests
import pandas as pd
response = requests.get('https://www.example.com')
text = response.text
df = pd.DataFrame([text], columns=['content'])
df.to_excel('output.xlsx', index=False)
四、处理爬取的文本内容
1、文本清洗和处理
import re
text = 'Hello,
World!'
clean_text = re.sub(r'<.*?>', '', text)
print(clean_text)
2、提取关键词
from jieba import analyse
text = 'Python爬取文本内容'
keywords = analyse.extract_tags(text)
print(keywords)
3、进行文本分析与统计
from collections import Counter
text = 'Hello, World!'
word_count = Counter(text.lower().split())
print(word_count)