lil_DengZi's Notes

数据解析概述

4分钟阅读

数据解析介绍

有效提取有用信息，筛选无用信息

  1. re解析
  2. bs4解析
  3. xpath解析

re模块解析

基于正则的解析

import re
 
# findall:匹配字符串所有符合的正则内容
lst = re.findall(r"\d+", "我的电话:10086, 他的电话:10010")
print(lst)
# output: ['10086', '10010']
 
# finditer:匹配字符串所有的内容
it = re.finditer(r"\d+", "我的电话:10086, 他的电话:10010")
for i in it:
	print(i.group())
# output:
# 10086
# 10010
 
# search找到一个结果就返回，返回match对象，拿数据需要group()
s = re.search(r"\d+", "我的电话:10086, 他的电话:10010")
print(s.group())
# output: 10086
 
# match只看头部，拿数据需要group()
s = re.match(r"\d+", "10086, 他的电话:10010")
print(s.group())
# output: 10086
 
# 预加载正则表达式,建议这么做compile(partten, flag[re.S, ...])
obj = re.compile(r"d+")
ret = obj.finditer("我的电话:10086, 他的电话:10010")
for it in ret:
	print(it.group())
# output:
# 10086
# 10010
 
title:Important
collapse:open
 
(?P<分组名字>正则)python,re库特有，可以单独从正则匹配的内容中进一步提取

bs4模块解析

BeautifulSoup 是一个可以从HTML或XML文档中提取数据的Python库。

from bs4 import BeautifulSoup
 
# 示例HTML内容
html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
 
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
 
<p class="story">...</p>
"""
 
# 使用BeautifulSoup解析HTML
soup = BeautifulSoup(html_doc, 'html.parser')
 
# 查找<title>标签并打印其内容
title = soup.find('title')
print("Title:", title.string)
 
# 查找所有<p>标签并打印其内容
paragraphs = soup.find_all('p')
for paragraph in paragraphs:
    print("Paragraph:", paragraph.text)
 
# 查找所有<a>标签并打印其文本和href属性
links = soup.find_all('a')
for link in links:
    print("Link text:", link.text)
    print("URL:", link.get('href'))
 
 
# output:
"""
Title: The Dormouse's story
Paragraph: The Dormouse's story
Paragraph: Once upon a time there were three little sisters; and their names were
Elsie,
Lacie and
Tillie;
and they lived at the bottom of a well.
Paragraph: ...
Link text: Elsie
URL: http://example.com/elsie
Link text: Lacie
URL: http://example.com/lacie
Link text: Tillie
URL: http://example.com/tillie
"""

xpath模块解析

XPath（XML Path Language）是一种在XML文档中查找信息的语言。它被设计用来通过路径表达式在XML文档中进行导航。XPath表达式可以用来选取XML文档中的节点、属性、文本内容等。

  • html是xml的子集

如果你有一个多层的XML结构，并且想要选择所有具有相同标签名的元素，可以这样做：

<!--data.xml-->
<root>
    <level1>
        <child>Text 1</child>
        <child>
            <child>Text 2</child>
        </child>
    </level1>
    <level1>
        <child>Text 3</child>
    </level1>
</root>
from lxml import etree
xml_data = """..."""  # data.xml
root = etree.XML(xml_data)
 
# /表层级关系，第一个/是根节点
selected_elements = root.xpath("/root")
print([etree.tostring(el).decode() for el in selected_elements])
# output:['<root>\n <level1>\n <child>Text 1</child>\n <child>\n <child>Text 2</child>\n </child>\n </level1>\n <level1>\n <child>Text 3</child>\n </level1>\n</root>\n']
selected_elements = root.xpath("/root/level1")
print([etree.tostring(el).decode() for el in selected_elements])
# output:['<level1>\n <child>Text 1</child>\n <child>\n <child>Text 2</child>\n </child>\n </level1>\n', '<level1>\n <child>Text 3</child>\n </level1>\n']
 
# 拿文本
selected_elements = root.xpath("/root/level1/child/text()")
print(selected_elements)
# output:['Text 1', 'Text 3']
 
# 全后代
selected_elements = root.xpath("/root/level1//child/text()")
print(selected_elements)
# output:['Text 1', 'Text 2', 'Text 3']
 
# 任意节点，通配符
selected_elements = root.xpath("/root/level1/*/child/text()")
print(selected_elements)
# output:['Text 2']
# 查找属性语法，匹配标签属性对应内容
root.xpath("/root/level1/*/child[@attribute='value']/text()")
 
# 拿标签属性值child
root.xpath("/root/level1/*/@class")
title:warning
collapse:open
 
xpath索引顺序从一开始
`root.xpath("/root/level1/*/child[1]/text()")`

关系图谱

最近的笔记

反向链接