Jake's Blog

OpenWRT - site and content filtering #2 privoxy

in Networking, OpenWRT - on 4:31 pm - 1 comment

CONTENT FILTERING #2

WORKING IN PROGRESS

Router Name	OpenWrt
Router Model	Buffalo WZR-HP-G450H
Firmware Version	OpenWrt Attitude Adjustment 12.09 / LuCI 0.11.1 Release (0.11.1)
Kernel Version	3.3.8

ssh into your router.

root@OpenWrt:~# opkg update
root@OpenWrt:~# opkg install privoxy

root@OpenWrt:~# cat /etc/privoxy/config 
confdir /etc/privoxy
logdir /var/log
filterfile default.filter
logfile privoxy
actionsfile match-all.action # Actions that are applied to all sites and maybe overruled later on.
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
listen-address 192.168.1.1:8118
toggle  1
enable-remote-toggle  1
enable-remote-http-toggle  0
enable-edit-actions 1
enforce-blocks 0
buffer-limit 4096
forwarded-connect-retries  0
accept-intercepted-requests 1
allow-cgi-request-crunching 0
split-large-forms 0
keep-alive-timeout 300
socket-timeout 300
permit-access  192.168.1.0/24
debug   2    # show each GET/POST/CONNECT request
debug   4096 # Startup banner and warnings
debug   8192 # Errors - *we highly recommended enabling this*
debug   65536 # Log the applying actions
debug   1024 # Log the destination for requests Privoxy didn't let through, and the reason why.
#admin-address privoxy-admin@example.com
#proxy-info-url http://www.example.com/proxy-service.html

Add +filter{shockwave-flash} \

root@OpenWrt:~# cat /etc/privoxy/match-all.action 
#############################################################################
# $Id: match-all.action,v 1.3 2010/03/27 18:48:38 fabiankeil Exp $
#
# This file contains the actions that are applied to all requests and
# may be overruled later on by other actions files. Less experienced
# users should only edit this file through the actions file editor.
#
#############################################################################
{ \
+change-x-forwarded-for{block} \
+client-header-tagger{css-requests} \
+client-header-tagger{image-requests} \
+hide-from-header{block} \
+set-image-blocker{pattern} \
+filter{shockwave-flash} \
+set-image-blocker{pattern} \
}
/ # Match all URLs

ONLY THE shockwage-flash filter

Append the following to /etc/privoxy/user.action

{+block{Block video extensions.}}                                                
/.*\.(mov|mp4|wmv|swf|avi|mpg|rms|wpl|tp|fbr|mpeg|otrkey|veg|m21|msdvd|mswmm|nvc|
                                                                                 
{-block}                                                                         
.lsmwebcast.com                                                                  
                                                                                 
{+block{Site-specific block pattern matches.}}                                   
# Blocked sites added by Jake He                                                 
.pptv.com                                                                        
.steam*.com                                                                      
.battle.net                                                                      
.youku.com                                                                       
.newsarama.com                                                                   
                                                                                 
# block movie sites                                                              
# source : shalla list http://www.shallalist.de/                                 
.cokeandpopcorn.ch                                                               
.56.com

OpenWRT - site and content filtering #1 tinyproxy and dansguardian

in Networking, OpenWRT - on 9:31 pm - 5 comments

CONTENT FILTERING #1

This week, I was given the task of filtering all gaming and video streaming in our network. After doing some research, I find out this is not as straight forward as it seems. Of course, I do not want to spend any money on purchasing any expensive software or hardware.

The general direction I took is to look for light weight proxy that can be installed on our existing OpenWRT router. Initially, I tried tinyproxy and dansguardian. This seems to be a promising solution. However after few hours of testing, users complained that lots of sites throws this error message "ERR_CONTENT_DECODING_FAILED". After a hard search, I cannot find out the course of this error. I decided to ditch tinyproxy and try privoxy. I noticed privoxy while trying to solve the error message.

Anyway, since I went into the trouble of getting tinyproxy and dansguardian up and running. Here are the steps I took to get it running.

Router Name	OpenWrt
Router Model	Buffalo WZR-HP-G450H
Firmware Version	OpenWrt Attitude Adjustment 12.09 / LuCI 0.11.1 Release (0.11.1)
Kernel Version	3.3.8

ssh into your router.

Install the software

opkg update
opkg install tinyproxy luci-app-tinyproxy dansguardian

Delete luci cache

rm /tmp/luci-indexcache

Configure tinyproxy

You can use either tinyproxy config file or luci to configure tinyproxy. Luci config is at Service->Tinyproxy. I used the config file.

My router LAN IP is 192.168.1.1. Tinyproxy port is 3128.

root@OpenWrt:~# cat /etc/config/tinyproxy 

config tinyproxy
 option User 'nobody'
 option Group 'nogroup'
 option Port '3128'
 option Listen '127.0.0.1'
 option Timeout '600'
 option DefaultErrorFile '/usr/share/tinyproxy/default.html'
 option StatFile '/usr/share/tinyproxy/stats.html'
 option MaxClients '100'
 option MinSpareServers '5'
 option MaxSpareServers '20'
 option StartServers '10'
 option MaxRequestsPerChild '0'
 option ViaProxyName 'tinyproxy'
 list ConnectPort '443'
 list ConnectPort '563'
 option enabled '1'
 option FilterExtended '1'
 option FilterURLs '1'
 option LogLevel 'Connect'
 option Syslog '1'
 option Allow '127.0.0.1'
 option Allow '192.168.1.0/24'

Configure firewall for transparent proxy

 config redirect                                        
    option name 'transparent proxy'                    
    option src 'lan'                                   
    option proto 'tcp'                                 
    option src_dport '80'                              
    option dest_port '3128'                            
    option src_dip '!192.168.1.1'                      
    option dest_ip '192.168.1.1'

Initialise tinyproxy's log file

root@OpenWrt:~# touch /var/log/tinyproxy.log 
root@OpenWrt:~# chown nobody.nogroup /var/log/tinyproxy.log

Restart tinyproxy every night to refresh memory

root@OpenWrt:~# crontab -e<
0 22 * * * root /etc/init.d/tinyproxy restart

Initialise dansguardianty's log files

root@OpenWrt:~# mkdir /var/log/dansguardian
root@OpenWrt:~# touch /var/log/dansguardian/access.log
root@OpenWrt:~# touch /var/log/dansguardian/stats

Configure dansguardian

root@OpenWrt:~# root@OpenWrt:~# grep ^[^#] /etc/dansguardian/dansguardian.conf 
reportinglevel = 2
languagedir = '/usr/share/dansguardian/languages'
language = 'ukenglish'
loglevel = 2
logexceptionhits = 2
logfileformat = 1
loglocation = '/var/log/dansguardian_access.log'
statlocation = '/var/log/dansguardian_stats'
filterip =
filterport = 8888
proxyip = 127.0.0.1
proxyport = 3128   #this have to match tinyproxy port  
accessdeniedaddress = 'http://YOURSERVER.YOURDOMAIN/cgi-bin/dansguardian.pl'
nonstandarddelimiter = on
usecustombannedimage = off
custombannedimagefile = '/usr/share/dansguardian/transparent1x1.gif'
filtergroups = 1
filtergroupslist = '/etc/dansguardian/lists/filtergroupslist'
bannediplist = '/etc/dansguardian/lists/bannediplist'
exceptioniplist = '/etc/dansguardian/lists/exceptioniplist'
showweightedfound = on
weightedphrasemode = 2
urlcachenumber = 1000
urlcacheage = 900
scancleancache = on
phrasefiltermode = 2
preservecase = 0
hexdecodecontent = off
forcequicksearch = off
reverseaddresslookups = off
reverseclientiplookups = off
logclienthostnames = off
createlistcachefiles = on
maxuploadsize = -1
maxcontentfiltersize = 256
maxcontentramcachescansize = 2000
maxcontentfilecachescansize = 20000
filecachedir = '/tmp'
deletedownloadedtempfiles = on
initialtrickledelay = 20
trickledelay = 10
downloadmanager = '/etc/dansguardian/downloadmanagers/fancy.conf'
downloadmanager = '/etc/dansguardian/downloadmanagers/default.conf'
contentscannertimeout = 60
contentscanexceptions = off
recheckreplacedurls = off
forwardedfor = off
usexforwardedfor = off
logconnectionhandlingerrors = on
logchildprocesshandling = off
maxchildren = 120
minchildren = 8
minsparechildren = 4
preforkchildren = 6
maxsparechildren = 32
maxagechildren = 500
maxips = 0
ipcfilename = '/tmp/.dguardianipc'
urlipcfilename = '/tmp/.dguardianurlipc'
ipipcfilename = '/tmp/.dguardianipipc'
nodaemon = off
nologger = off
logadblocks = off
loguseragent = off
daemonuser = 'root'
daemongroup = 'root'
softrestart = off

root@OpenWrt:# grep ^[^#] /etc/dansguardian/dansguardianf1.conf 
groupmode = 1
groupname = 'group_one'
bannedphraselist = '/etc/dansguardian/lists/bannedphraselist'
weightedphraselist = '/etc/dansguardian/lists/weightedphraselist'
exceptionphraselist = '/etc/dansguardian/lists/exceptionphraselist'
bannedsitelist = '/etc/dansguardian/lists/bannedsitelist'
greysitelist = '/etc/dansguardian/lists/greysitelist'
exceptionsitelist = '/etc/dansguardian/lists/exceptionsitelist'
bannedurllist = '/etc/dansguardian/lists/bannedurllist'
greyurllist = '/etc/dansguardian/lists/greyurllist'
exceptionurllist = '/etc/dansguardian/lists/exceptionurllist'
exceptionregexpurllist = '/etc/dansguardian/lists/exceptionregexpurllist'
bannedregexpurllist = '/etc/dansguardian/lists/bannedregexpurllist'
picsfile = '/etc/dansguardian/lists/pics'
contentregexplist = '/etc/dansguardian/lists/contentregexplist'
urlregexplist = '/etc/dansguardian/lists/urlregexplist'
blockdownloads = off
exceptionextensionlist = '/etc/dansguardian/lists/exceptionextensionlist'
exceptionmimetypelist = '/etc/dansguardian/lists/exceptionmimetypelist'
bannedextensionlist = '/etc/dansguardian/lists/bannedextensionlist'
bannedmimetypelist = '/etc/dansguardian/lists/bannedmimetypelist'
exceptionfilesitelist = '/etc/dansguardian/lists/exceptionfilesitelist'
exceptionfileurllist = '/etc/dansguardian/lists/exceptionfileurllist'
headerregexplist = '/etc/dansguardian/lists/headerregexplist'
bannedregexpheaderlist = '/etc/dansguardian/lists/bannedregexpheaderlist'
naughtynesslimit = 160
categorydisplaythreshold = 0
embeddedurlweight = 0
enablepics = off
bypass = 0
bypasskey = ''
infectionbypass = 0
infectionbypasskey = ''
infectionbypasserrorsonly = on
disablecontentscan = off
deepurlanalysis = off

Run tinyproxy and dansguardian

root@OpenWrt:~# /etc/init.d/tinyproxy enable
root@OpenWrt:~# /etc/init.d/tinyproxy start
root@OpenWrt:~# /etc/init.d/firewall restart

For dansguardian, I realised there is no startup scripts is installed. I just run it from the command by typing dansguardian. Reload it by dansguardian -r. For dansguardian I get this error: "Error reading custom image file: /usr/share/dansguardian/transparent1x1.gif" I just ignore this. It runs fine. If both tineyproxy and dansguardian is running, you should see in ps. There should be lots of messages in their log files.

Content filtering

All the content filtering files are in /etc/dansguardian/lists. You can edit them to suit your requirements.

Debugging

If you want to see what the tinyproxy and dansguardian is doing, check their log files in /var/log.

root@OpenWrt:~# tinyproxy -d -c /var/etc/tinyproxy.conf

References:

Traffic Analysis

in FreeRADIUS, Networking, OpenWRT - on 2:03 pm - No comments

ntop

Internet Quota Management

in FreeRADIUS, Networking - on 2:01 pm - 1 comment

If you are interested to know how I manage internet quota, please leave a commit. When I have time, I can slowly write a post to explain it.

Dynamic Clients with FreeRADIUS Using MAC-Address

in FreeRADIUS, Networking - on 1:57 pm - No comments

This blog will show how to setup dynamic clients that do no have static ip address.

Details are coming soon!

802.1X/EAP User Authentication

in FreeRADIUS, Networking - on 10:16 am - 3 comments

This blog will how to configure FreeRADIUS to authenticate wireless users via 802.1X/EAP. MySQL will be used an user store.

Wireless AP will be a MikroTik (MT) router. RADIUS server will be FreeRADIUS 2.2.5 running on Linux Mint Debian. This tutorial is only an instruction to setup a 802.1X/EAP wireless network. It will not explain how 802.1X/EAP wireless network works.

Radius Terminologies:
Clients: A client refers to a NAS, like an access point (AP). In this tutorial, it is the MT router.
NAS: Network access server.

Change Ownership:
Default FreeRADIUS configuration files are in /etc/freeradius. To work in this folder easier change the folder ownership.

$ sudo chown -R jake:freerad /etc/freeradius

Configure NAS:
For clients to work probably, NAS requires to have a static ip address. To enable dynamic clients with no static ip address, please see this post. This tutorial will assume all the clients have a static ip address. Client configuration can be stored in a file or MySQL. Below show how to configure the client file.

$ cd /etc/freeradius

Open the client file with an editor.

$ sudo vi clients.conf

This file is well documented. Read through it for your own understanding. Add a new client by copy and paste the following to the end of the clients.conf. Client's ip address is the ip that the RADIUS have to communicate with. If RADIUS is within the same local network, client's ip address is its local ip address. If RADIUS have to speak the client via the internet, client's ip address is the public ip address.

client 127.313.28.15{
 secret    = radius
 shortname = client_home
 nas_type  = mikrotik
 limit {
  max_connections = 0
  lifetime = 0
  idle_timeout = 30
 }
}

MySQL Configuration:
Setting up the MySQL database: Username: radius. Password: radpass.

 mysql -uroot -p
   CREATE DATABASE radius;
   GRANT ALL ON radius.* TO radius@localhost IDENTIFIED BY "radpass";
   exit

Create the tables.

$sudo  mysql -u root -p radius < /etc/freeradius/sql/mysql/schema.sql
$sudo mysql -u root -p radius < /etc/freeradius/sql/mysql/nas.sql

Configuration:
Add the radius database information.

$vi sql.conf
    driver = "rlm_sql_mysql"
    # Connection info:
    server = "localhost"
    port = 3306
    login = "radius"
    password = "radpass"
    read_groups = yes    
    read_clients = yes

Uncomment all sql in /sites-enabled/default and /sites-enabled/inner-tunnel.

$ sudo sed -i s/^#.*sql$/sql/g sites-enabled/default

$ sudo sed -i s/^#.*sql$/sql/g /etc/freeradius/sites-enabled/inner-tunnel

Add $INCLUDE sql.conf below $INCLUDE sites-enabled/ in /radiusd.conf.
Link sql.conf to modules.

$ cd modules

$ ln -s sql.conf modules/sql

Disable Proxy:

sudo vi radiusd.conf
proxy_requests  = no

Configure EAP:

$ cd /etc/freeradius
$ sed -i s/use_tunneled_reply\ =\ no/use_tunneled_reply\ =\ yes/g  eap.conf

Disable PAP:

$ sed -i s/pap$/#pap/g  sites-enabled/default

Run it:

$ sudo radiusd -X
Listening on authentication address * port 1812
Listening on accounting address * port 1813
Listening on authentication address 127.0.0.1 port 18120 as server inner-tunnel
Ready to process requests.

Setup Test Environment:
Add Radius Client to Mk Router:
Setup MK Router to authenticate wireless network via EAP.
Add test username and password to MySQL. Try it.

FreeRADIUS build and install (Debian packages with rlm_raw patch)

in FreeRADIUS, Networking - on 2:31 pm - No comments

This post will show how to build and install Debian packages for FreeRADIUS 2.2.5 with rlm_raw patch.

Machine: Linux Mint Debian Edition (LMDE) 64 bit

Download the source code:
Download the FreeRADIUS source file.

wget ftp://ftp.freeradius.org/pub/freeradius/freeradius-server-2.2.5.tar.bz2

Extract the source code.

tar jxf freeradius-server-2.2.5.tar.bz2

Patch the code with rlm_raw:
This patch enable dynamic clients using mac address rather than static ip address.
Download the rlm_raw patch.

wget -O rlm_raw_patch http://sourceforge.net/p/radiusdesk/code/HEAD/tree/trunk/rd_cake/Setup/Radius/rlm_raw_patch?format=raw

Add the rlm_raw patch

cd freeradius-server-2.2.5
patch -p1 < rlm_raw_patch

If all goes according to plan the following output should show on your screen:

$ patch -p1 < rlm_raw_patch 
patching file src/modules/rlm_raw/config.h.in
patching file src/modules/rlm_raw/configure
patching file src/modules/rlm_raw/configure.in
patching file src/modules/rlm_raw/Makefile.in
patching file src/modules/rlm_raw/rlm_raw.c
patching file src/modules/stable
Hunk #1 FAILED at 39.
1 out of 1 hunk FAILED -- saving rejects to file src/modules/stable.rej

This simply means the last part of the patch did not happen. You can manually fix it by adding the rlm_raw to the bottom of the src/modules/stable file.

echo rlm_raw >> src/modules/stable

Build
Install dependencies. This might take a while.

sudo apt-get install debhelper quilt autotools-dev libtool libltdl3-dev libssl-dev libpam0g-dev libmysqlclient-dev libgdbm-dev libldap2-dev libsasl2-dev libiodbc2-dev libkrb5-dev libperl-dev libpcap-dev python-dev libsnmp-dev libpq-dev

Build FreeRADIUS packages from the source. Be patient this will take some time too.

sudo fakeroot dpkg-buildpackage -b -uc

A successful build will show the following files:

ls ../*.deb
freeradius_2.2.5+git_amd64.deb
freeradius-common_2.2.5+git_all.deb
freeradius-dbg_2.2.5+git_amd64.deb
freeradius-dialupadmin_2.2.5+git_all.deb
freeradius-iodbc_2.2.5+git_amd64.deb
freeradius-krb5_2.2.5+git_amd64.deb
freeradius-ldap_2.2.5+git_amd64.deb
freeradius-mysql_2.2.5+git_amd64.deb
freeradius-postgresql_2.2.5+git_amd64.deb
freeradius-server-2.2.5
freeradius-server-2.2.5.tar.bz2
freeradius-utils_2.2.5+git_amd64.deb
libfreeradius2_2.2.5+git_amd64.deb
libfreeradius-dev_2.2.5+git_amd64.deb

Install
Install the newly built FreeRADIUS packages in the following sequence.

cd ..
sudo dpkg -i libfreeradius2_2.2.5+git_amd64.deb freeradius-common_2.2.5+git_all.deb
sudo dpkg -i freeradius_2.2.5+git_amd64.deb freeradius-krb5_2.2.5+git_amd64.deb freeradius-ldap_2.2.5+git_amd64.deb freeradius-dbg_2.2.5+git_amd64.deb freeradius-iodbc_2.2.5+git_amd64.deb 
sudo dpkg -i freeradius-mysql_2.2.5+git_amd64.deb freeradius-utils_2.2.5+git_amd64.deb

A successful install should show the following:

$ sudo /etc/init.d/freeradius restart
[ ok ] Checking FreeRADIUS daemon configuration...done (Configuration appears to be OK.).
[ ok ] Stopping FreeRADIUS daemon: freeradius.
[ ok ] Starting FreeRADIUS daemon: freeradius.

References: