Java实现DFA算法实现敏感词过滤,并且支持跳词，如中 国，中*国等，并包含了比较全的敏感词库，也摘抄了其它几种敏感词过滤方法，只对DFA方式的敏感词做了优化，支持跳词
2022-04-05 16:44:51
3.41MB
1
Author: Richard Zhang.
Mail: 89205975@qq.com
This library filters sensitive phrases by user's configuration.
Currently, only support UTF8 & ANSI encoded strings.
The matching rule is max-length-matching, the library tries to match sensitive phrase as long as possible.
For example:
"damn fucker" and "damn" are all in sensitive dictionary, the sentence "he's a damn fucker" will be processed to "he's a ***********".
Even user insert some spaces or non-letter characters between sensitive words, the library is also able to deal with it.
For example:
"Bad boy" is added to sensitive dictionary, "Bad.boy", "Bad boy", "Bad/boy" can also be filtered.
"你去死" is added to sensitive dictionary, "你 去 死", "你/去 死", "你 去 .死" can also be filtered.
Compiling requirement:
1. STL C++11
2. BOOST multi_index_container
Performance test condition:
1. Giving a sentence around 100 bytes (English & Chinese mixed)
2. Dirty phrases around 10,000
3. Do 1,000 loop test
4. Intel I7 CPU
Test result:
For each loop, it cost around 100us
2022-04-02 17:47:14
4KB
1
java使用dfa算法实现敏感词过滤，此算法效率最高，附带了一个敏感词库，轻松搞定论坛网站的敏感词过滤问题。
2022-03-30 13:14:46
1.39MB
1
这是一个微博敏感词搜索程序,把需要搜索的敏感词存'weibo.positive.train'里
，每隔一定时间从微博API接口申请chunk，利用positive和negative结合的方法，完全结果输出到'weibo.censorshipOutput'
2022-03-21 11:01:52
5.91MB
1
如下所示：
#!/usr/bin/python2.6
# -*- coding: utf-8 -*-
import time
class Node(object):
def __init__(self):
self.children = None
# The encode of word is UTF-8
def add_word(root,word):
node = root
for i in range(len(word)):
if node.children == None:
node.children = {}
node.chil
2022-03-01 10:49:08
52KB
1