Skip to content

Releases: GeneralNewsExtractor/GeneralNewsExtractor

Bug fix

17 Apr 14:48
@kingname kingname
v0.3.1
4da1eb7
Compare
Choose a tag to compare
Bug fix Latest
Latest

0.3.1 (2024-04-17)

Bug fix

有一些网站源代码不规范，在html中间突然出现。末尾又出现一次.这种情况下，会导致解析出错。现在已经修复。

Assets 3
Loading

支持基于可视化区域精确查找正文

07 Oct 07:03
@kingname kingname
v0.3.0
7067925
Compare
Choose a tag to compare
支持基于可视化区域精确查找正文

0.3.0 (2021-10-07)

New Feature

  1. 基于可视化区域，更准确地识别正文

Bug fix

  • 修复下面这种情况时，无法正确寻找正文的 bug
<div>
我是正文我是正文我是正文<a href="xxx">关键词1</a>我是正文我是正文我是正文我是正文
我是正文我是正文我是正文我是正文我是正文<a href="xxx">关键词2</a>我是正文我是正文
我是正文
</div>
  • 统计一个标签下面的 p 标签的时候，应该把这个标签下面的直接文档数也统计进去
Loading

限制 h 标签与 title 的最小公共子串长度

17 Feb 03:04
@kingname kingname
0.2.6
5c5093d
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
限制 h 标签与 title 的最小公共子串长度
  1. 修复 extract_by_htag_and_title 在发现 H 标签中的文本与 title 标签的文本在最小公共子串长度小于4时被认为是标题的问题。
Loading

修复从 title 标签提取标题部分失败的问题

21 Dec 15:21
@kingname kingname
0.2.5
de4a338
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
修复从 title 标签提取标题部分失败的问题
  1. 如果标题中含有-|，且在较为靠前的地方，可能导致标题只提取了半截。需要判断-|左侧字符串的长度。如果小于4，那么返回整个标题字符串。
Loading

预处理时，移除 footer 标签

06 Oct 14:43
@kingname kingname
0.2.4
fa52c52
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
预处理时，移除 footer 标签
  1. 预处理时，移除 footer 标签。
Loading

现在 useless_attr 中的属性必须完全匹配才能删除节点

15 Sep 04:37
@kingname kingname
0.2.3
f80ab69
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
现在 useless_attr 中的属性必须完全匹配才能删除节点

0.2.3 (2020-09-15)

Bug fix

  1. USELESS_ATTR对应的节点，只有 class 完全匹配才需要删除。之前包含就删除的匹配方式会导致 ifeng 的正文被删除。
Loading

自动提取新闻列表页

02 Aug 09:28
@kingname kingname
0.2.2
e4bdccf
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
自动提取新闻列表页

新闻列表页自动提取功能测试版已经上线，用法如下：

>>> from gne import ListPageExtractor
>>> html = '''经过渲染的网页 HTML 代码'''
>>> list_extractor = ListPageExtractor()
>>> result = list_extractor.extract(html,
                                    feature='列表中任意元素的 XPath")
>>> print(result)
Loading

修复提取节点中的文本时只能提取最后一个节点的问题

27 Jun 08:00
@kingname kingname
0.2.1
d68e894
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
修复提取节点中的文本时只能提取最后一个节点的问题 
Merge pull request #84 from kingname/develop

修复提取标签中的文本时，只能提取最后一个节点的问题
Loading

优化标题提取算法，增加 body_xpath 参数

06 Jun 04:16
@kingname kingname
0.1.9
815d839
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
优化标题提取算法，增加 body_xpath 参数
  1. 优化标题提取逻辑，根据@止水 和 @asyncins 的建议，通过对比 //title/text()中的文本与 标签中的文本，提取出标题。
  2. 增加 body_xpath参数，精确定义正文所在的位置，强力避免干扰。

例如对于澎湃新闻，在不设置body_xpath参数时：

result = extractor.extract(html,
                           host='https://www.xxx.com',
                           noise_node_list=['//div[@class="comment-list"]',
                                            '//*[@style="display:none"]',
                                            '//div[@class="statement"]'
                                            ])

提取效果如下：

设置了body_xpath以后：

result = extractor.extract(html,
                           host='https://www.xxx.com',
                           body_xpath='//div[@class="news_txt"]',  # 缩小正文提取范围
                           noise_node_list=['//div[@class="comment-list"]',
                                            '//*[@style="display:none"]',
                                            '//div[@class="statement"]'
                                            ])

结果如下：

Loading

修复由于预处理导致自定义 XPath 失效的问题

11 Mar 13:01
@kingname kingname
0.18
5461b6d
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
修复由于预处理导致自定义 XPath 失效的问题
  1. 预处理逻辑可能会破坏原有 HTML 结构，导致用户自定义的 XPath 失效。因此需要再预处理之前提取 title、author 和 publish_time。
Loading