Recently I was confronted with the situation of many users Avast Antivirus installations updating from the internet, and downloading many MB's of files on a limited internet connection. Also, due to the speed of the connection, and the fact that they are going through a captival portal, the downloads take considerable time, and are counted towards their download quota. Due to a limitation in the captive portal, I can't change it so that the Avast downloads aren't counted in their download limit, but if they don't login to the portal, then the download won't count towards their quota.
So part of the solution was going to be to make the downloads happen quicker. It was already going through a Squid proxy, which should have made this happen, but it still meant the downloads would be slow the first time, so one user would be charged for the downloads, and not subsequent users. And due to the fact that it load balances the servers, the chances of all users hitting the same avast download server is next to nothing. My searches on the net lead me to http://files.avast.com/files/eng/mirror.zip which is an application from the Avast Distributed Network Manager that sets up a central server for managing avast in a corporate setting. After playing around with the mirror.exe file that generates a mirror, and discovering that adding "restrict_products=av_pro" to the mirror.ini file, I could limit the mirror generation to just the normal avast files for program and virus definition updates.
So now I had an Avast mirror, I now needed to do 2 things, find a way to generate an avast mirror, and update it without using the windows only, mirror.exe program, and find a way to make Avast clients update from the mirror, and not the internet.
The first problem was solved with a bit of poking around in the files downloaded by mirror.exe, and discovering that it checks a couple of "stamp" files on the server, if the stamp files are new, it downloads the file it is a stamp for, in particular, mirror.def, which contains a complete list of all the files needed for a mirror, with the MD5 sum of the file, and filesize, as well as the "products" that utilise the file. This file makes it easy to generate a mirror!! Below is the script that generates a mirror. There are a few things it doesn't do that mirror.exe does. It doesn't generate the 400.vps files, which Linux clients use to update from (all other clients should use the partial update files to save downloading 20Mb+ every time). It also doesn't generate a new mirror.def file that only contains the files that this partial contains, instead it uses the same mirror.def file that it used to generate the mirror from the master servers. It also doesn't modify the servers.def file. The main reason for this is that these files are signed, and we don't yet know how to sign them ourselves, so the clients would reject any modifications. Also, unless someone who will be behind the proxy, and thus using your mirror, attempts to create their own more complete mirror, mirror.def being wrong doesn't matter. Also, changing the servers.def file to point to your mirror as mirror.exe does, would prevent users from later updating Avast on another network that doesn't contain your mirror.
The next part of the solution is how to redirect requests for Avast updates to your mirror, without Avast knowing it isn't getting it's updates from the real servers, so that no modifications to Avast are needed.
So this part actually worked out a lot easier than initially thought. All users currently go through a Squid proxy, transparently, due to filtering and the obvious benefit of caching on a limited internet connection. I thought that with Avast attempting it's updates from a variety of servers throughout the world, and with both IP and domain name attempts (it alternates between the 2 so that DNS outages, or DNS hijacks, can't prevent it from updating), it would be difficult to write a generic filter to redirect them. Then I realised that every request from avast, was to a directory, iavs4x. Suddenly I didn't need to worry about which server it was attempting to connect to, just the directory. So I wrote a Squid Redirector that rewrites any request that is to the directory, iavs4x, to go to the server that hosts the mirror. You can setup ACL's so that not every URL goes through the redirector, but as I already use a redirector for adzapping, I didn't pursue this option. The script is below, change the server variable to reflect your mirror server.
avast_mirror.sh
#!/bin/bash
#### Avast Mirror
#
# Files that should be in mirror but won't be
# 400.vps 400.vps.md5
#
# Extra files we need that aren't in mirror.def
# jollyroger.vpu.stamp jollyroger.vpu servers.def servers.def.stamp servers.def.vpu.stamp servers.def.lkg servers.def.vpu
# BASEURL (Select a random from mirrors file? with fallback, currently just default)
BASEURL="http://files.avast.com/iavs4x/"
mirror_def=mirror.def
BASEDIR=/home/mirror
TMP_URL_LIST=/tmp/mirror_files
# Filter for mirror.def
restrict_products='av_pro'
# Set DEBUG to anything other than '' to get debugging info
DEBUG=''
cd $BASEDIR
# Files updated from stamp files
function stamp_update {
file=$1
stampfile=$1.stamp
old_stamp=$(cat $stampfile)
file_stamp=$file_stamp.new
wget -q -O $file_stamp $BASEURL$stampfile
new_stamp=$(cat $file_stamp)
if test "x$old_stamp" != "x$new_stamp" ; then
echo "$file out of date."
echo "Updating..."
mv $file_stamp $stampfile
mv -f $file $file.old
wget -nv $BASEURL$file
echo "$file updated"
else
if test -n "$DEBUG" ; then echo "$file up to date.";fi
fi
}
stamp_update mirror.def
stamp_update jollyroger.vpu
stamp_update servers.def
stamp_update servers.def.vpu
# Read in mirror.def, check all files and add out of date ones to queue
rm -f $TMP_URL_LIST
# Add "extra" files that are always downloaded here
#echo $BASEURL"servers.def.lkg" > $TMP_URL_LIST
filelist=$(cat $mirror_def |grep $restrict_products|cut -f 2 -d =|cut -f 1-3 -d ,)
for file in $filelist
do
info=( `echo "$file" | tr -s ',' ' '` )
filename=${info[0]}
filesize=${info[1]}
filemd5=${info[2]}
if test ! -f $filename
then
if test -n "$DEBUG" ; then echo;fi
echo "Adding new $filename to download queue"
echo $BASEURL$filename >> $TMP_URL_LIST
continue
fi
curfilesize=$(stat -c%s "$filename")
if test "0$curfilesize" -lt "0$filesize"
then
if test -n "$DEBUG" ; then echo;fi
echo "Adding incomplete $filename to download queue"
echo $BASEURL$filename >> $TMP_URL_LIST
continue
fi
curfilemd5=$(md5sum "$filename" | cut -f 1 -d ' '|tr [:lower:] [:upper:])
if test "x$filemd5" != "x$curfilemd5"
then
mv -f $filename $filename.old
if test -n "$DEBUG" ; then echo;fi
echo "Adding corrupt $filename to download queue"
echo $BASEURL$filename >> $TMP_URL_LIST
continue
fi
if test -n "$DEBUG" ; then echo -n ".";fi
#echo "$filename already downloaded and complete"
done
if test -n "$DEBUG" ; then echo;fi
if test -s $TMP_URL_LIST
then
echo "Downloading out of date mirror files"
#Download files
# --no-dns-cache means it'll load balance randomly accross different servers for each file
wget -c -nv -N --no-dns-cache -i /tmp/mirror_files --progress=dot |tee -a mirror.log
fiavast_redirect.pl
#!/usr/bin/perl
$|=1;
$server='10.1.0.1';
while (<>) {
@X = split;
$url = $X[0];
if ($url =~ /^http:\/\/[^\/]*\/iavs4x\// && $url !~ m/$server/o) {
$url =~ s/^http:\/\/[^\/]*\/iavs4x\//http:\/\/$server\/iavs4x\//o;
print "$url\n";
} else {
print "\n";
}
}
February 1st, 2009 | Tags: ADNM, Antivirus, Avast, Bash, Distributed Network Manager, Linux, Mirror, mirror.zip, Perl, Programming, Proxy, Redirect, Redirector, Scripting, Squid, Src, Transparent | Category: Blog, Computers, Internet, New Technology's | Comments (2)
