OpenWRT - site and content filtering #2 privoxy



Router NameOpenWrt
Router ModelBuffalo WZR-HP-G450H
Firmware VersionOpenWrt Attitude Adjustment 12.09 / LuCI 0.11.1 Release (0.11.1)
Kernel Version3.3.8

ssh into your router.

root@OpenWrt:~# opkg update
root@OpenWrt:~# opkg install privoxy
root@OpenWrt:~# cat /etc/privoxy/config 
confdir /etc/privoxy
logdir /var/log
filterfile default.filter
logfile privoxy
actionsfile match-all.action # Actions that are applied to all sites and maybe overruled later on.
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
toggle  1
enable-remote-toggle  1
enable-remote-http-toggle  0
enable-edit-actions 1
enforce-blocks 0
buffer-limit 4096
forwarded-connect-retries  0
accept-intercepted-requests 1
allow-cgi-request-crunching 0
split-large-forms 0
keep-alive-timeout 300
socket-timeout 300
debug   2    # show each GET/POST/CONNECT request
debug   4096 # Startup banner and warnings
debug   8192 # Errors - *we highly recommended enabling this*
debug   65536 # Log the applying actions
debug   1024 # Log the destination for requests Privoxy didn't let through, and the reason why.
Add +filter{shockwave-flash} \
root@OpenWrt:~# cat /etc/privoxy/match-all.action 
# $Id: match-all.action,v 1.3 2010/03/27 18:48:38 fabiankeil Exp $
# This file contains the actions that are applied to all requests and
# may be overruled later on by other actions files. Less experienced
# users should only edit this file through the actions file editor.
{ \
+change-x-forwarded-for{block} \
+client-header-tagger{css-requests} \
+client-header-tagger{image-requests} \
+hide-from-header{block} \
+set-image-blocker{pattern} \
+filter{shockwave-flash} \
+set-image-blocker{pattern} \
/ # Match all URLs
ONLY THE shockwage-flash filter

Append the following to /etc/privoxy/user.action

{+block{Block video extensions.}}                                                
{+block{Site-specific block pattern matches.}}                                   
# Blocked sites added by Jake He                                                                                                                
# block movie sites                                                              
# source : shalla list                                                                                

OpenWRT - site and content filtering #1 tinyproxy and dansguardian


This week, I was given the task of filtering all gaming and video streaming in our network. After doing some research, I find out this is not as straight forward as it seems. Of course, I do not want to spend any money on purchasing any expensive software or hardware. 

The general direction I took is to look for light weight proxy that can be installed on our existing OpenWRT router. Initially, I tried tinyproxy and dansguardian. This seems to be a promising solution. However after few hours of testing, users complained that lots of sites throws this error message "ERR_CONTENT_DECODING_FAILED". After a hard search, I cannot find out the course of this error. I decided to ditch tinyproxy and try privoxy. I noticed privoxy while trying to solve the error message. 

Anyway, since I went into the trouble of getting tinyproxy and dansguardian up and running. Here are the steps I took to get it running. 
Router NameOpenWrt
Router ModelBuffalo WZR-HP-G450H
Firmware VersionOpenWrt Attitude Adjustment 12.09 / LuCI 0.11.1 Release (0.11.1)
Kernel Version3.3.8

ssh into your router.

Install the software

opkg update
opkg install tinyproxy luci-app-tinyproxy dansguardian

Delete luci cache 

rm /tmp/luci-indexcache

Configure tinyproxy

You can use either tinyproxy config file or luci to configure tinyproxy. Luci config is at Service->Tinyproxy. I used the config file.
My router LAN IP is Tinyproxy port is 3128.
root@OpenWrt:~# cat /etc/config/tinyproxy 

config tinyproxy
 option User 'nobody'
 option Group 'nogroup'
 option Port '3128'
 option Listen ''
 option Timeout '600'
 option DefaultErrorFile '/usr/share/tinyproxy/default.html'
 option StatFile '/usr/share/tinyproxy/stats.html'
 option MaxClients '100'
 option MinSpareServers '5'
 option MaxSpareServers '20'
 option StartServers '10'
 option MaxRequestsPerChild '0'
 option ViaProxyName 'tinyproxy'
 list ConnectPort '443'
 list ConnectPort '563'
 option enabled '1'
 option FilterExtended '1'
 option FilterURLs '1'
 option LogLevel 'Connect'
 option Syslog '1'
 option Allow ''
 option Allow ''

Configure firewall for transparent proxy

 config redirect                                        
    option name 'transparent proxy'                    
    option src 'lan'                                   
    option proto 'tcp'                                 
    option src_dport '80'                              
    option dest_port '3128'                            
    option src_dip '!'                      
    option dest_ip ''

Initialise tinyproxy's log file

root@OpenWrt:~# touch /var/log/tinyproxy.log 
root@OpenWrt:~# chown nobody.nogroup /var/log/tinyproxy.log                        

Restart tinyproxy every night to refresh memory

root@OpenWrt:~# crontab -e<
0 22 * * * root /etc/init.d/tinyproxy restart

Initialise dansguardianty's log files

root@OpenWrt:~# mkdir /var/log/dansguardian
root@OpenWrt:~# touch /var/log/dansguardian/access.log
root@OpenWrt:~# touch /var/log/dansguardian/stats

Configure dansguardian

root@OpenWrt:~# root@OpenWrt:~# grep ^[^#] /etc/dansguardian/dansguardian.conf 
reportinglevel = 2
languagedir = '/usr/share/dansguardian/languages'
language = 'ukenglish'
loglevel = 2
logexceptionhits = 2
logfileformat = 1
loglocation = '/var/log/dansguardian_access.log'
statlocation = '/var/log/dansguardian_stats'
filterip =
filterport = 8888
proxyip =
proxyport = 3128   #this have to match tinyproxy port  
accessdeniedaddress = 'http://YOURSERVER.YOURDOMAIN/cgi-bin/'
nonstandarddelimiter = on
usecustombannedimage = off
custombannedimagefile = '/usr/share/dansguardian/transparent1x1.gif'
filtergroups = 1
filtergroupslist = '/etc/dansguardian/lists/filtergroupslist'
bannediplist = '/etc/dansguardian/lists/bannediplist'
exceptioniplist = '/etc/dansguardian/lists/exceptioniplist'
showweightedfound = on
weightedphrasemode = 2
urlcachenumber = 1000
urlcacheage = 900
scancleancache = on
phrasefiltermode = 2
preservecase = 0
hexdecodecontent = off
forcequicksearch = off
reverseaddresslookups = off
reverseclientiplookups = off
logclienthostnames = off
createlistcachefiles = on
maxuploadsize = -1
maxcontentfiltersize = 256
maxcontentramcachescansize = 2000
maxcontentfilecachescansize = 20000
filecachedir = '/tmp'
deletedownloadedtempfiles = on
initialtrickledelay = 20
trickledelay = 10
downloadmanager = '/etc/dansguardian/downloadmanagers/fancy.conf'
downloadmanager = '/etc/dansguardian/downloadmanagers/default.conf'
contentscannertimeout = 60
contentscanexceptions = off
recheckreplacedurls = off
forwardedfor = off
usexforwardedfor = off
logconnectionhandlingerrors = on
logchildprocesshandling = off
maxchildren = 120
minchildren = 8
minsparechildren = 4
preforkchildren = 6
maxsparechildren = 32
maxagechildren = 500
maxips = 0
ipcfilename = '/tmp/.dguardianipc'
urlipcfilename = '/tmp/.dguardianurlipc'
ipipcfilename = '/tmp/.dguardianipipc'
nodaemon = off
nologger = off
logadblocks = off
loguseragent = off
daemonuser = 'root'
daemongroup = 'root'
softrestart = off

root@OpenWrt:# grep ^[^#] /etc/dansguardian/dansguardianf1.conf 
groupmode = 1
groupname = 'group_one'
bannedphraselist = '/etc/dansguardian/lists/bannedphraselist'
weightedphraselist = '/etc/dansguardian/lists/weightedphraselist'
exceptionphraselist = '/etc/dansguardian/lists/exceptionphraselist'
bannedsitelist = '/etc/dansguardian/lists/bannedsitelist'
greysitelist = '/etc/dansguardian/lists/greysitelist'
exceptionsitelist = '/etc/dansguardian/lists/exceptionsitelist'
bannedurllist = '/etc/dansguardian/lists/bannedurllist'
greyurllist = '/etc/dansguardian/lists/greyurllist'
exceptionurllist = '/etc/dansguardian/lists/exceptionurllist'
exceptionregexpurllist = '/etc/dansguardian/lists/exceptionregexpurllist'
bannedregexpurllist = '/etc/dansguardian/lists/bannedregexpurllist'
picsfile = '/etc/dansguardian/lists/pics'
contentregexplist = '/etc/dansguardian/lists/contentregexplist'
urlregexplist = '/etc/dansguardian/lists/urlregexplist'
blockdownloads = off
exceptionextensionlist = '/etc/dansguardian/lists/exceptionextensionlist'
exceptionmimetypelist = '/etc/dansguardian/lists/exceptionmimetypelist'
bannedextensionlist = '/etc/dansguardian/lists/bannedextensionlist'
bannedmimetypelist = '/etc/dansguardian/lists/bannedmimetypelist'
exceptionfilesitelist = '/etc/dansguardian/lists/exceptionfilesitelist'
exceptionfileurllist = '/etc/dansguardian/lists/exceptionfileurllist'
headerregexplist = '/etc/dansguardian/lists/headerregexplist'
bannedregexpheaderlist = '/etc/dansguardian/lists/bannedregexpheaderlist'
naughtynesslimit = 160
categorydisplaythreshold = 0
embeddedurlweight = 0
enablepics = off
bypass = 0
bypasskey = ''
infectionbypass = 0
infectionbypasskey = ''
infectionbypasserrorsonly = on
disablecontentscan = off
deepurlanalysis = off

Run tinyproxy and dansguardian

root@OpenWrt:~# /etc/init.d/tinyproxy enable
root@OpenWrt:~# /etc/init.d/tinyproxy start
root@OpenWrt:~# /etc/init.d/firewall restart
For dansguardian, I realised there is no startup scripts is installed. I just run it from the command by typing dansguardian. Reload it by dansguardian -r. For dansguardian I get this error: "Error reading custom image file: /usr/share/dansguardian/transparent1x1.gif" I just ignore this. It runs fine. If both tineyproxy and dansguardian is running, you should see in ps. There should be lots of messages in their log files.

Content filtering

All the content filtering files are in /etc/dansguardian/lists. You can edit them to suit your requirements. 


If you want to see what the tinyproxy and dansguardian is doing, check their log files in /var/log.
root@OpenWrt:~# tinyproxy -d -c /var/etc/tinyproxy.conf


Traffic Analysis


Internet Quota Management

If you are interested to know how I manage internet quota, please leave a commit. When I have time, I can slowly write a post to explain it.

Dynamic Clients with FreeRADIUS Using MAC-Address

This blog will show how to setup dynamic clients that do no have static ip address.

Details are coming soon!

802.1X/EAP User Authentication

This blog will how to configure FreeRADIUS to authenticate wireless users via 802.1X/EAP.  MySQL will be used an user store.
Wireless AP will be a MikroTik (MT) router. RADIUS server will be FreeRADIUS 2.2.5 running on Linux Mint Debian. This tutorial is only an instruction to setup a 802.1X/EAP wireless network. It will not explain how 802.1X/EAP wireless network works.

Radius Terminologies:
Clients: A client refers to a NAS, like an access point (AP). In this tutorial, it is the MT router.
NAS: Network access server.

Change Ownership:
Default FreeRADIUS configuration files are in /etc/freeradius. To work in this folder easier change the folder ownership.
$ sudo chown -R jake:freerad /etc/freeradius
Configure NAS:
For clients to work probably, NAS requires to have a static ip address. To enable dynamic clients with no static ip address, please see this post. This tutorial will assume all the clients have a static ip address. Client configuration can be stored in a file or MySQL. Below show how to configure the client file.
$ cd /etc/freeradius
Open the client file with an editor.
$ sudo vi clients.conf
This file is well documented. Read through it for your own understanding. Add a new client by copy and paste the following to the end of the clients.conf. Client's ip address is the ip that the RADIUS have to communicate with. If RADIUS is within the same local network, client's ip address is its local ip address. If RADIUS have to speak the client via the internet, client's ip address is the public ip address.
client 127.313.28.15{
 secret    = radius
 shortname = client_home
 nas_type  = mikrotik
 limit {
  max_connections = 0
  lifetime = 0
  idle_timeout = 30
MySQL Configuration:
Setting up the MySQL database: Username: radius. Password: radpass.
 mysql -uroot -p
   GRANT ALL ON radius.* TO radius@localhost IDENTIFIED BY "radpass";
Create the tables.
$sudo  mysql -u root -p radius < /etc/freeradius/sql/mysql/schema.sql
$sudo mysql -u root -p radius < /etc/freeradius/sql/mysql/nas.sql
Add the radius database information. 
$vi sql.conf
    driver = "rlm_sql_mysql"
    # Connection info:
    server = "localhost"
    port = 3306
    login = "radius"
    password = "radpass"
    read_groups = yes    
    read_clients = yes
Uncomment all sql in  /sites-enabled/default and /sites-enabled/inner-tunnel.
$ sudo sed -i s/^#.*sql$/sql/g sites-enabled/default
$ sudo sed -i s/^#.*sql$/sql/g /etc/freeradius/sites-enabled/inner-tunnel 
Add $INCLUDE sql.conf below $INCLUDE sites-enabled/ in /radiusd.conf.
Link sql.conf to modules.
$ cd modules 
$ ln -s sql.conf modules/sql
Disable Proxy:
sudo vi radiusd.conf
proxy_requests  = no
Configure EAP:
$ cd /etc/freeradius
$ sed -i s/use_tunneled_reply\ =\ no/use_tunneled_reply\ =\ yes/g  eap.conf
Disable PAP:
$ sed -i s/pap$/#pap/g  sites-enabled/default 
Run it:
$ sudo radiusd -X
Listening on authentication address * port 1812
Listening on accounting address * port 1813
Listening on authentication address port 18120 as server inner-tunnel
Ready to process requests.
Setup Test Environment:
Add Radius Client to Mk Router:
Setup MK Router to authenticate wireless network via EAP.
Add test username and password to MySQL. Try it.

FreeRADIUS build and install (Debian packages with rlm_raw patch)

This post will show how to build and install Debian packages for FreeRADIUS 2.2.5 with rlm_raw patch.

Machine: Linux Mint Debian Edition (LMDE) 64 bit

Download the source code:
Download the FreeRADIUS source file.
Extract the source code.
tar jxf freeradius-server-2.2.5.tar.bz2

Patch the code with rlm_raw:
This patch enable dynamic clients using mac address rather than static ip address.
Download the rlm_raw patch.
wget -O rlm_raw_patch
Add the rlm_raw patch
cd freeradius-server-2.2.5
patch -p1 < rlm_raw_patch
If all goes according to plan the following output should show on your screen:
$ patch -p1 < rlm_raw_patch 
patching file src/modules/rlm_raw/
patching file src/modules/rlm_raw/configure
patching file src/modules/rlm_raw/
patching file src/modules/rlm_raw/
patching file src/modules/rlm_raw/rlm_raw.c
patching file src/modules/stable
Hunk #1 FAILED at 39.
1 out of 1 hunk FAILED -- saving rejects to file src/modules/stable.rej
This simply means the last part of the patch did not happen. You can manually fix it by adding the rlm_raw to the bottom of the src/modules/stable file.
echo rlm_raw >> src/modules/stable

Install dependencies. This might take a while.
sudo apt-get install debhelper quilt autotools-dev libtool libltdl3-dev libssl-dev libpam0g-dev libmysqlclient-dev libgdbm-dev libldap2-dev libsasl2-dev libiodbc2-dev libkrb5-dev libperl-dev libpcap-dev python-dev libsnmp-dev libpq-dev
Build FreeRADIUS packages from the source. Be patient this will take some time too.
sudo fakeroot dpkg-buildpackage -b -uc
A successful build will show the following files:
ls ../*.deb

Install the newly built FreeRADIUS packages in the following sequence.
cd ..
sudo dpkg -i libfreeradius2_2.2.5+git_amd64.deb freeradius-common_2.2.5+git_all.deb
sudo dpkg -i freeradius_2.2.5+git_amd64.deb freeradius-krb5_2.2.5+git_amd64.deb freeradius-ldap_2.2.5+git_amd64.deb freeradius-dbg_2.2.5+git_amd64.deb freeradius-iodbc_2.2.5+git_amd64.deb 
sudo dpkg -i freeradius-mysql_2.2.5+git_amd64.deb freeradius-utils_2.2.5+git_amd64.deb 
A successful install should show the following:
$ sudo /etc/init.d/freeradius restart
[ ok ] Checking FreeRADIUS daemon configuration...done (Configuration appears to be OK.).
[ ok ] Stopping FreeRADIUS daemon: freeradius.
[ ok ] Starting FreeRADIUS daemon: freeradius.

  1. rlm_raw patch
  2. Build Debian packages


Router Model: Asus WL-500G Premium V2
Firmware Version: DD-WRT v24-sp2 (05/20/09) std-nokaid-usb - build 12171M NEWD Eko
~ # mkdir /mnt/opt
~ # mount -o bind /mnt/opt /opt

lamsao: the above links seem to be dead.  I found this link and it works.[dead links deleted]

wget -O - | tr -d '\r' > /tmp/
sh /tmp/

Search Path
Adding /opt/bin to search PATH will save us from tedious typing of /opt/bin for each command from /opt/bin. For running applications from /opt tree search path should be expanded with
~# export PATH=/opt/bin:/opt/sbin:$PATH 

Installing Samba 2
/opt/bin/ipkg-opt update 
/opt/bin/ipkg-opt remove samba
/opt/bin/ipkg-opt install samba2
/opt/bin/ipkg-opt install xinetd

Startup script for external hard disk

/opt/bin/ipkg-opt install e2fsprogs