搜狗微信搜索文章,微信搜索文章

爬取搜狗微信文章笔记1 错误1 ```python print('https://weixin.sogou.com'+index)TypeError: can only concatenate str (not "list") to str类型错误:只能连接str(不是“列表”)到str print((''.join('https://weixin.sogou.com'+str(i))))print(((['https://weixin.sogou.com'+str(i)]))) 错误2 "bytes, not %s" % (name, value, type(value)))requests.exceptions.InvalidHeader: Value for header {Upgrade-Insecure-Requests: 1} must be of type str or bytes, not <class 'int'>

‘Upgrade-Insecure-Requests’:注释就可以执行 。是对应服务器响应头的。该指令用于让浏览器自动升级请求从http到https,用于大量包含http资源的http

网页直接升级到https而不会报错.简洁的来讲,就相当于在http和https之间起的一个过渡作用.

urlencode:
可直接将字典转化为a=1&b=2这样的形式

from urllib.parse import urlencodedata_dict={'name':'2333','id':123}string=urlencode(data_dict)print(type(string),string)

localhost:5000/get 查看使用的代理

quote和unquote:
quote：只对一个字符串转码，并且通过unquote解码

from urllib.parse import quote,unquotestring='背后的钢笔'print(quote(string))print(unquote(string))

源码

import requestsfrom urllib.parse import urlencodefrom requests.exceptions import ConnectionErrorheaders={'Cookie': 'SUID=E33ACAB75B68860A5B6E50A8000A0B9A; ssuid=2552472443; SUV=00FE8D813B3122035CED2A68C492A981; CXID=70712B2A6F1E8F15E6FB9AFC90EF5547; sw_uuid=7258101195; IPLOC=CN4116; SMYUV=1589958741935605; UM_distinctid=17230ed0fb67-03ff5b502a94b4-541d3410-100200-17230ed0fb742; pgv_pvi=7822496768; ABTEST=7|1590070219|v1; weixinIndexVisited=1; ld=hZllllllll2W84iqlllllVEGKOUlllllt8TikZllll9lllllpklll5@@@@@@@@@@; ad=Xlllllllll2W1cmGlllllVEFAcZlllllt8T5nkllll9lllll4Zlll5@@@@@@@@@@; SNUID=E01ECC03B4B01C872881E712B5F39569; JSESSIONID=aaaU3PxUU377Y51MuNwix; ppinf=5|1592318511|1593528111|dHJ1c3Q6MToxfGNsaWVudGlkOjQ6MjAxN3x1bmlxbmFtZToyNzolRTglQkYlQkQlRTYlQTIlQTYlRTQlQkElQkF8Y3J0OjEwOjE1OTIzMTg1MTF8cmVmbmljazoyNzolRTglQkYlQkQlRTYlQTIlQTYlRTQlQkElQkF8dXNlcmlkOjQ0Om85dDJsdURiMjJucGlMb19zZ2xCQzktTEg3cGdAd2VpeGluLnNvaHUuY29tfA; pprdig=fMDb-yC4MqZc_0mFUbRIQw1iKPL73EMAoQkmE0jQdaSzSePlDDQUKdpJXdYGRca3hcgRzcIkub3pIaXeJbRvmit_N9_VWSZ3OT8C4Z6eF1vpZS-cTHl_5TbTEa8o6zNITor9WDRSwTRXslbXbpFu9FrGQNZS1-wLuaRtX9rnN40; sgid=18-31413265-AV7o2ia89p9G5UwVazLVhteI; ppmdig=159231851100000022858e609e82c49a64951cdbde7aa217; sct=5','Host': 'weixin.sogou.com','Referer': 'https://weixin.sogou.com/weixin?query=%E6%83%85%E6%84%9F%E7%BE%8E%E6%96%87&hp=40&sut=6155&lkt=1%2C1592310242116%2C1592310242116&_sug_=y&sst0=1592310242220&oq=%E6%83%85%E6%84%9F&stj0=1&stj1=4&stj=1%3B4%3B0%3B0&stj2=0&hp1=&_sug_type_=&s_from=input&ri=1&type=2&page=1&ie=utf8&w=01015002&dr=1',# 'Upgrade-Insecure-Requests': 1,'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36'}urlbase='https://weixin.sogou.com/weixin?'keyword='风景'#解析网页def get_html(url): try: response = requests.get(url, allow_redirects=False,headers=headers)#不允许重定向 if response.status_code==200: return response.text if response.status_code==302: print('302') except ConnectionError: return get_html(url)#获取网页def get_index(keyword,page): data={ 'query':'keyword', 'type':2, 'page':'page' } queries=urlencode(data) url=urlbase+queries html=get_html(url) return htmldef main(): for page in range(1,10): html=get_index(keyword,page) print(html)if __name__ == '__main__': main()

参考源码网址https://github.com/Germey/weixin