Zip up stats for disk space conservation

[Arf]

Please note that this code has been revised as of 12/17/05. Look for my updated code created on that date.

Here are some commands that you can use to reduce your disk space usage. The first one zips up the stats in all of the client's accounts for a particular year (Example shows 2002. If you want a different year, just change that number throughout):

Code: Select all

find /home/*/*-www/stats/ -maxdepth 1 -name www2002 | xargs -i zip -r www2002stats.zip {}

Now you can remove the directories that contain the stats:

Code: Select all

rm -Rf /home/*/www/stats/www2002/

Enjoy!

[rldev]

Thanks Arf, this is good stuff.

[naplesdave]

Many of my customers don't want stats at all. We frown on turning off stats, because we want to sell them on adding more space for keeping stats.

But when we do turn them off, we simply remove the entire stats directory. It will be recreated by the stats software, so what we do is just create a touch file named "stats" and this keeps the software from creating a directory.

rm -rf stats

The command to create the placeholder file is:

touch stats

That creates a file of zero bytes named stats. When the stats program tries to create a directory named stats, it will error out because a file with that name is already in place.

[knopfler]

Very useful, thanks a lot!

[sixpackmx]

Dave:

Nice solution, still, there is another workaround. Create a .NoStats file on the user's home.

We have packages that does not include stats because of their limited size. We run a script that compares user quota and creats the .NoStats file accordingly.

Also, our clients have to option to disable stats from a CP Custom Tool.

Best wishes,

SixpackMX

[Anonymous]

Cool! I would be interested in both.

[naplesdave]

I have tried .NoStats on several accounts and it doesn't seem to be working. What's the trick?

[Arf]

see Ala KB #411 for information.

[naplesdave]

So according to KB 411 it should stop updating /stats and shouldn't even create a dir named /stats if you have deleted it.

I suppose I should get with Support. Several of mine are continuing to pump out data.

[Heather]

Good tips!

Some clients do want their stats, but when you have a fairly busy website, I've found it's the referrers files that really bulk up the stats. It is fairly easy to instruct clients to go into their stats via the file manager and remove all of the lrefers****.html files. It's easy to see these are usually the largest files if you sort the files according to size. Most people won't miss these much, or the lagents****.html ones either which are normally the second largest group of files. I have a client whose lrefers files are about 10MB for each year of stats.
Heather

[RobW]

If you just want to see how much of your server disk space is used by the stats for a particular year, you can run the following command:

Code: Select all

find /home*/*/*-www/stats/ -maxdepth 1 -name www2002  | xargs du -hc

This produces a size for each client plus a total at the bottom

Cheers
Rob

[flaguy]

Worked great Rob! thanks!

[Sapphyre]

I did this

Code: Select all

cd /home/$user/$domain-www/stats/www2004
zip -r ../www2004.zip ./*

It created the file just fine.
I go by browser to http://domain.com/stats/www2004.zip
I have tried both opening the archive directly or saving it to my desktop first and then opening it (Win XP) - both times the archive shows 0 objects.

OTOH, I try unzip on the server and all the files are retrieved just fine.

How do I make a Windows compatible zip ?

[Sapphyre]

..supposed to prevent opening them from untrusted sites; however I have found that even if I add the site I am downloading the zip from to 'trusted sites', I still can't open the zip. The other suggestion was to use to navigate to the file, right click - properties - and click the 'Unblock' button. Still doesn't open.

looking on Microsoft's newsgroup - post entitled "Blocking" dated 1.24.05
http://www.microsoft.com/windowsxp/expe ... me+extract
Got an MVP telling people to get a third party unzip tool!

This is just outrageous. MS has really gone too far with the security precuations.

[Anonymous]

Sapphyre I had no problem zipping the stats and opening on either win 2000 pro or win xp. On win 2000 I use winzip with xp I use the default windows app.

I did use Arf's method of zipping up the stats...

[Sapphyre]

Sorry, I neglected to add - it's a problem as of Win XP SP2 - they added a so-called "attachment manager" which prevents you from opening emailed or downloaded files from untrusted sources. Also using the default app provided.

I was able to 'unblock' the zip file itself, but I think the items within the zip file (stats folder for a particular year) are subject to further blocking - but you can't unblock them because winzip won't even list them out until they are unblocked.

It's insane - some files are high risk unless they are opened in specific programs!
http://support.microsoft.com/default.as ... -us;883260

I did download zipgenius.it and it opened fine.

[flaguy]

yes the win xp i used has the sp2 ...

[nada]

Here are some commands that you can use to reduce your disk space usage. The first one zips up the stats in all of the client's accounts for a particular year (Example shows 2002. If you want a different year, just change that number throughout):

Code: Select all

find /home/*/*-www/stats/ -maxdepth 1 -name www2002 | xargs -i zip -r www2002stats.zip {}

Now you can remove the directories that contain the stats:

Code: Select all

rm -Rf /home/*/www/stats/www2002/

Enjoy!

Greetings

I did the above, ie zipping 2004 stats

using:
find /home/*/*-www/stats/ -maxdepth 1 -name www2004 | xargs -i zip -r www2004stats.zip {}

It showed for an hour or so that it was deflating files, but in the end it gave countless errors like:
zip error: Zip file structure invalid (www2004stats.zip)
zip warning: count in end of central directory incorrect



and then when trying to delete them using :
rm -Rf /home/*/www/stats/www2004/

I am getting this error for all the zipped files.

rm: cannot open directory `/home/zoomsite/www/stats/www2004/': Too many open files

Any suggestions?

[Arf]

This is some very old code. I've learned a lot since writing it. So, here's the updated code. Just Change the "2002" at the beginning to any year you wish to manage amd run this is one very long command.

Code: Select all

 YEAR="2002";for STATS in `find /home/*/*-www/stats/ -maxdepth 1 -name www$YEAR`;do cd $STATS; cd ..; zip -r www$YEAR.zip www$YEAR; rm -Rf ./www$YEAR;done

To make sure this worked, I ran it on my server for 2002 and 2003 and it worked like a charm.

[Arf]

Variation on the above theme. I ran the above and then wondered if it had really done it's job (it did but I'm paranoid). So below is the same script except that it shows a directory listing of the stats directory so you can "see" the wwwYEAR.zip file for yourself.

Code: Select all

YEAR="2004";for STATS in `find /home/*/*-www/stats/ -maxdepth 1 -name www$YEAR`;do cd $STATS; cd ..; zip -r www$YEAR.zip www$YEAR; rm -Rf ./www$YEAR;ls -al; pwd;sleep 2;done

[Arf]

Welcome to 2009. It's that time once again where you run the above command to clear our last year's stats. Happy new year!

[Arf]

Looks like navisite never cleared out the December 2009 stats directory. So you'll need this too.

Code: Select all

for directory in `find /home/*/*-logs/old/200912 -maxdepth 0`; do echo $directory; rm -Rf $directory;done