OpenWRT - site and content filtering #2 privoxy

CONTENT FILTERING #2

WORKING IN PROGRESS

Router NameOpenWrt
Router ModelBuffalo WZR-HP-G450H
Firmware VersionOpenWrt Attitude Adjustment 12.09 / LuCI 0.11.1 Release (0.11.1)
Kernel Version3.3.8

ssh into your router.


root@OpenWrt:~# opkg update
root@OpenWrt:~# opkg install privoxy
root@OpenWrt:~# cat /etc/privoxy/config 
confdir /etc/privoxy
logdir /var/log
filterfile default.filter
logfile privoxy
actionsfile match-all.action # Actions that are applied to all sites and maybe overruled later on.
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
listen-address 192.168.1.1:8118
toggle  1
enable-remote-toggle  1
enable-remote-http-toggle  0
enable-edit-actions 1
enforce-blocks 0
buffer-limit 4096
forwarded-connect-retries  0
accept-intercepted-requests 1
allow-cgi-request-crunching 0
split-large-forms 0
keep-alive-timeout 300
socket-timeout 300
permit-access  192.168.1.0/24
debug   2    # show each GET/POST/CONNECT request
debug   4096 # Startup banner and warnings
debug   8192 # Errors - *we highly recommended enabling this*
debug   65536 # Log the applying actions
debug   1024 # Log the destination for requests Privoxy didn't let through, and the reason why.
#admin-address privoxy-admin@example.com
#proxy-info-url http://www.example.com/proxy-service.html
Add +filter{shockwave-flash} \
root@OpenWrt:~# cat /etc/privoxy/match-all.action 
#############################################################################
# $Id: match-all.action,v 1.3 2010/03/27 18:48:38 fabiankeil Exp $
#
# This file contains the actions that are applied to all requests and
# may be overruled later on by other actions files. Less experienced
# users should only edit this file through the actions file editor.
#
#############################################################################
{ \
+change-x-forwarded-for{block} \
+client-header-tagger{css-requests} \
+client-header-tagger{image-requests} \
+hide-from-header{block} \
+set-image-blocker{pattern} \
+filter{shockwave-flash} \
+set-image-blocker{pattern} \
}
/ # Match all URLs
ONLY THE shockwage-flash filter




Append the following to /etc/privoxy/user.action

{+block{Block video extensions.}}                                                
/.*\.(mov|mp4|wmv|swf|avi|mpg|rms|wpl|tp|fbr|mpeg|otrkey|veg|m21|msdvd|mswmm|nvc|
                                                                                 
{-block}                                                                         
.lsmwebcast.com                                                                  
                                                                                 
{+block{Site-specific block pattern matches.}}                                   
# Blocked sites added by Jake He                                                 
.pptv.com                                                                        
.steam*.com                                                                      
.battle.net                                                                      
.youku.com                                                                       
.newsarama.com                                                                   
                                                                                 
# block movie sites                                                              
# source : shalla list http://www.shallalist.de/                                 
.cokeandpopcorn.ch                                                               
.56.com  
         

OpenWRT - site and content filtering #1 tinyproxy and dansguardian

CONTENT FILTERING #1

This week, I was given the task of filtering all gaming and video streaming in our network. After doing some research, I find out this is not as straight forward as it seems. Of course, I do not want to spend any money on purchasing any expensive software or hardware. 

The general direction I took is to look for light weight proxy that can be installed on our existing OpenWRT router. Initially, I tried tinyproxy and dansguardian. This seems to be a promising solution. However after few hours of testing, users complained that lots of sites throws this error message "ERR_CONTENT_DECODING_FAILED". After a hard search, I cannot find out the course of this error. I decided to ditch tinyproxy and try privoxy. I noticed privoxy while trying to solve the error message. 

Anyway, since I went into the trouble of getting tinyproxy and dansguardian up and running. Here are the steps I took to get it running. 
Router NameOpenWrt
Router ModelBuffalo WZR-HP-G450H
Firmware VersionOpenWrt Attitude Adjustment 12.09 / LuCI 0.11.1 Release (0.11.1)
Kernel Version3.3.8

ssh into your router.

Install the software

opkg update
opkg install tinyproxy luci-app-tinyproxy dansguardian

Delete luci cache 

rm /tmp/luci-indexcache

Configure tinyproxy

You can use either tinyproxy config file or luci to configure tinyproxy. Luci config is at Service->Tinyproxy. I used the config file.
My router LAN IP is 192.168.1.1. Tinyproxy port is 3128.
root@OpenWrt:~# cat /etc/config/tinyproxy 

config tinyproxy
 option User 'nobody'
 option Group 'nogroup'
 option Port '3128'
 option Listen '127.0.0.1'
 option Timeout '600'
 option DefaultErrorFile '/usr/share/tinyproxy/default.html'
 option StatFile '/usr/share/tinyproxy/stats.html'
 option MaxClients '100'
 option MinSpareServers '5'
 option MaxSpareServers '20'
 option StartServers '10'
 option MaxRequestsPerChild '0'
 option ViaProxyName 'tinyproxy'
 list ConnectPort '443'
 list ConnectPort '563'
 option enabled '1'
 option FilterExtended '1'
 option FilterURLs '1'
 option LogLevel 'Connect'
 option Syslog '1'
 option Allow '127.0.0.1'
 option Allow '192.168.1.0/24'

Configure firewall for transparent proxy

 config redirect                                        
    option name 'transparent proxy'                    
    option src 'lan'                                   
    option proto 'tcp'                                 
    option src_dport '80'                              
    option dest_port '3128'                            
    option src_dip '!192.168.1.1'                      
    option dest_ip '192.168.1.1'

Initialise tinyproxy's log file

root@OpenWrt:~# touch /var/log/tinyproxy.log 
root@OpenWrt:~# chown nobody.nogroup /var/log/tinyproxy.log                        

Restart tinyproxy every night to refresh memory

root@OpenWrt:~# crontab -e<
0 22 * * * root /etc/init.d/tinyproxy restart

Initialise dansguardianty's log files

root@OpenWrt:~# mkdir /var/log/dansguardian
root@OpenWrt:~# touch /var/log/dansguardian/access.log
root@OpenWrt:~# touch /var/log/dansguardian/stats

Configure dansguardian

root@OpenWrt:~# root@OpenWrt:~# grep ^[^#] /etc/dansguardian/dansguardian.conf 
reportinglevel = 2
languagedir = '/usr/share/dansguardian/languages'
language = 'ukenglish'
loglevel = 2
logexceptionhits = 2
logfileformat = 1
loglocation = '/var/log/dansguardian_access.log'
statlocation = '/var/log/dansguardian_stats'
filterip =
filterport = 8888
proxyip = 127.0.0.1
proxyport = 3128   #this have to match tinyproxy port  
accessdeniedaddress = 'http://YOURSERVER.YOURDOMAIN/cgi-bin/dansguardian.pl'
nonstandarddelimiter = on
usecustombannedimage = off
custombannedimagefile = '/usr/share/dansguardian/transparent1x1.gif'
filtergroups = 1
filtergroupslist = '/etc/dansguardian/lists/filtergroupslist'
bannediplist = '/etc/dansguardian/lists/bannediplist'
exceptioniplist = '/etc/dansguardian/lists/exceptioniplist'
showweightedfound = on
weightedphrasemode = 2
urlcachenumber = 1000
urlcacheage = 900
scancleancache = on
phrasefiltermode = 2
preservecase = 0
hexdecodecontent = off
forcequicksearch = off
reverseaddresslookups = off
reverseclientiplookups = off
logclienthostnames = off
createlistcachefiles = on
maxuploadsize = -1
maxcontentfiltersize = 256
maxcontentramcachescansize = 2000
maxcontentfilecachescansize = 20000
filecachedir = '/tmp'
deletedownloadedtempfiles = on
initialtrickledelay = 20
trickledelay = 10
downloadmanager = '/etc/dansguardian/downloadmanagers/fancy.conf'
downloadmanager = '/etc/dansguardian/downloadmanagers/default.conf'
contentscannertimeout = 60
contentscanexceptions = off
recheckreplacedurls = off
forwardedfor = off
usexforwardedfor = off
logconnectionhandlingerrors = on
logchildprocesshandling = off
maxchildren = 120
minchildren = 8
minsparechildren = 4
preforkchildren = 6
maxsparechildren = 32
maxagechildren = 500
maxips = 0
ipcfilename = '/tmp/.dguardianipc'
urlipcfilename = '/tmp/.dguardianurlipc'
ipipcfilename = '/tmp/.dguardianipipc'
nodaemon = off
nologger = off
logadblocks = off
loguseragent = off
daemonuser = 'root'
daemongroup = 'root'
softrestart = off

root@OpenWrt:# grep ^[^#] /etc/dansguardian/dansguardianf1.conf 
groupmode = 1
groupname = 'group_one'
bannedphraselist = '/etc/dansguardian/lists/bannedphraselist'
weightedphraselist = '/etc/dansguardian/lists/weightedphraselist'
exceptionphraselist = '/etc/dansguardian/lists/exceptionphraselist'
bannedsitelist = '/etc/dansguardian/lists/bannedsitelist'
greysitelist = '/etc/dansguardian/lists/greysitelist'
exceptionsitelist = '/etc/dansguardian/lists/exceptionsitelist'
bannedurllist = '/etc/dansguardian/lists/bannedurllist'
greyurllist = '/etc/dansguardian/lists/greyurllist'
exceptionurllist = '/etc/dansguardian/lists/exceptionurllist'
exceptionregexpurllist = '/etc/dansguardian/lists/exceptionregexpurllist'
bannedregexpurllist = '/etc/dansguardian/lists/bannedregexpurllist'
picsfile = '/etc/dansguardian/lists/pics'
contentregexplist = '/etc/dansguardian/lists/contentregexplist'
urlregexplist = '/etc/dansguardian/lists/urlregexplist'
blockdownloads = off
exceptionextensionlist = '/etc/dansguardian/lists/exceptionextensionlist'
exceptionmimetypelist = '/etc/dansguardian/lists/exceptionmimetypelist'
bannedextensionlist = '/etc/dansguardian/lists/bannedextensionlist'
bannedmimetypelist = '/etc/dansguardian/lists/bannedmimetypelist'
exceptionfilesitelist = '/etc/dansguardian/lists/exceptionfilesitelist'
exceptionfileurllist = '/etc/dansguardian/lists/exceptionfileurllist'
headerregexplist = '/etc/dansguardian/lists/headerregexplist'
bannedregexpheaderlist = '/etc/dansguardian/lists/bannedregexpheaderlist'
naughtynesslimit = 160
categorydisplaythreshold = 0
embeddedurlweight = 0
enablepics = off
bypass = 0
bypasskey = ''
infectionbypass = 0
infectionbypasskey = ''
infectionbypasserrorsonly = on
disablecontentscan = off
deepurlanalysis = off

Run tinyproxy and dansguardian

root@OpenWrt:~# /etc/init.d/tinyproxy enable
root@OpenWrt:~# /etc/init.d/tinyproxy start
root@OpenWrt:~# /etc/init.d/firewall restart
For dansguardian, I realised there is no startup scripts is installed. I just run it from the command by typing dansguardian. Reload it by dansguardian -r. For dansguardian I get this error: "Error reading custom image file: /usr/share/dansguardian/transparent1x1.gif" I just ignore this. It runs fine. If both tineyproxy and dansguardian is running, you should see in ps. There should be lots of messages in their log files.

Content filtering

All the content filtering files are in /etc/dansguardian/lists. You can edit them to suit your requirements. 

Debugging

If you want to see what the tinyproxy and dansguardian is doing, check their log files in /var/log.
root@OpenWrt:~# tinyproxy -d -c /var/etc/tinyproxy.conf

References:


Traffic Analysis


ntop