Zip up stats for disk space conservation

If you have a program or script that you'd like to give away or sell, this is the place to do it. All offers should include contact info.

Moderators: BBear, theunknownhost, flaguy

Post Reply
User avatar
Arf
Official Test Penquin
Posts: 9103
Joined: Tue Apr 09, 2002 12:00 am
Location: IDAHO, USA
Contact:

Zip up stats for disk space conservation

Post by Arf » Sat Sep 27, 2003 12:12 am

Please note that this code has been revised as of 12/17/05. Look for my updated code created on that date.

Here are some commands that you can use to reduce your disk space usage. The first one zips up the stats in all of the client's accounts for a particular year (Example shows 2002. If you want a different year, just change that number throughout):

Code: Select all

find /home/*/*-www/stats/ -maxdepth 1 -name www2002 | xargs -i zip -r www2002stats.zip {}


Now you can remove the directories that contain the stats:

Code: Select all

rm -Rf /home/*/www/stats/www2002/


Enjoy!
Last edited by Arf on Sat Dec 17, 2005 9:24 pm, edited 2 times in total.

rldev
Hosting Superstar
Posts: 1067
Joined: Tue Aug 05, 2003 10:17 pm

Post by rldev » Sat Sep 27, 2003 2:05 am

Thanks Arf, this is good stuff.

naplesdave
Nothing better to do.
Posts: 243
Joined: Mon Apr 14, 2003 6:18 pm
Location: Naples, FL
Contact:

Post by naplesdave » Tue Nov 18, 2003 10:56 pm

Many of my customers don't want stats at all. We frown on turning off stats, because we want to sell them on adding more space for keeping stats.

But when we do turn them off, we simply remove the entire stats directory. It will be recreated by the stats software, so what we do is just create a touch file named "stats" and this keeps the software from creating a directory.

rm -rf stats

The command to create the placeholder file is:

touch stats

That creates a file of zero bytes named stats. When the stats program tries to create a directory named stats, it will error out because a file with that name is already in place.
Kind regards,
Dave Jackson
World Wide Mart, Inc.

User avatar
knopfler
Nothing better to do.
Posts: 163
Joined: Wed Nov 26, 2003 6:27 pm

Post by knopfler » Thu Nov 27, 2003 3:40 pm

Very useful, thanks a lot!

sixpackmx
Hard Drive Crasher
Posts: 640
Joined: Fri Nov 01, 2002 9:44 am
Location: Mexico City, Mexico

Post by sixpackmx » Thu Nov 27, 2003 4:20 pm

Dave:

Nice solution, still, there is another workaround. Create a .NoStats file on the user's home.

We have packages that does not include stats because of their limited size. We run a script that compares user quota and creats the .NoStats file accordingly.

Also, our clients have to option to disable stats from a CP Custom Tool.

Best wishes,

SixpackMX

Anonymous

Post by Anonymous » Sat Nov 29, 2003 9:23 pm

Cool! I would be interested in both.

naplesdave
Nothing better to do.
Posts: 243
Joined: Mon Apr 14, 2003 6:18 pm
Location: Naples, FL
Contact:

Post by naplesdave » Sat Jan 17, 2004 3:43 pm

I have tried .NoStats on several accounts and it doesn't seem to be working. What's the trick?
Kind regards,
Dave Jackson
World Wide Mart, Inc.

User avatar
Arf
Official Test Penquin
Posts: 9103
Joined: Tue Apr 09, 2002 12:00 am
Location: IDAHO, USA
Contact:

Post by Arf » Sat Jan 17, 2004 9:48 pm

see Ala KB #411 for information.

naplesdave
Nothing better to do.
Posts: 243
Joined: Mon Apr 14, 2003 6:18 pm
Location: Naples, FL
Contact:

Post by naplesdave » Sat Jan 17, 2004 11:52 pm

So according to KB 411 it should stop updating /stats and shouldn't even create a dir named /stats if you have deleted it.

I suppose I should get with Support. Several of mine are continuing to pump out data.
Kind regards,
Dave Jackson
World Wide Mart, Inc.

Heather
Alahosts.com-Newbie
Posts: 6
Joined: Tue Sep 21, 2004 8:27 pm
Location: London, Ontario, Canada
Contact:

Post by Heather » Wed Sep 22, 2004 4:11 am

Good tips!

Some clients do want their stats, but when you have a fairly busy website, I've found it's the referrers files that really bulk up the stats. It is fairly easy to instruct clients to go into their stats via the file manager and remove all of the lrefers****.html files. It's easy to see these are usually the largest files if you sort the files according to size. Most people won't miss these much, or the lagents****.html ones either which are normally the second largest group of files. I have a client whose lrefers files are about 10MB for each year of stats.
Heather

RobW
Propeller head licensee
Posts: 45
Joined: Wed Nov 10, 2004 11:35 am

Post by RobW » Thu Jan 06, 2005 2:47 pm

If you just want to see how much of your server disk space is used by the stats for a particular year, you can run the following command:

Code: Select all

find /home*/*/*-www/stats/ -maxdepth 1 -name www2002  | xargs du -hc


This produces a size for each client plus a total at the bottom

Cheers
Rob

flaguy
Hard Drive Crasher
Posts: 725
Joined: Wed Nov 24, 2004 4:52 pm
Location: Florida

Post by flaguy » Fri Jan 07, 2005 1:23 pm

Worked great Rob! thanks!

User avatar
Sapphyre
Official Test Penquin
Posts: 3337
Joined: Tue Nov 19, 2002 2:50 am
Location: Canada

Won't open in winzip ?

Post by Sapphyre » Tue Mar 15, 2005 1:50 am

I did this

Code: Select all

cd /home/$user/$domain-www/stats/www2004
zip -r ../www2004.zip ./*


It created the file just fine.
I go by browser to http://domain.com/stats/www2004.zip
I have tried both opening the archive directly or saving it to my desktop first and then opening it (Win XP) - both times the archive shows 0 objects.

OTOH, I try unzip on the server and all the files are retrieved just fine.

How do I make a Windows compatible zip ?

User avatar
Sapphyre
Official Test Penquin
Posts: 3337
Joined: Tue Nov 19, 2002 2:50 am
Location: Canada

SP2 prevents opening zips

Post by Sapphyre » Tue Mar 15, 2005 3:21 am

..supposed to prevent opening them from untrusted sites; however I have found that even if I add the site I am downloading the zip from to 'trusted sites', I still can't open the zip. The other suggestion was to use to navigate to the file, right click - properties - and click the 'Unblock' button. Still doesn't open.

looking on Microsoft's newsgroup - post entitled "Blocking" dated 1.24.05
http://www.microsoft.com/windowsxp/expe ... me+extract
Got an MVP telling people to get a third party unzip tool!

This is just outrageous. MS has really gone too far with the security precuations.

Anonymous

Post by Anonymous » Tue Mar 15, 2005 1:16 pm

Sapphyre I had no problem zipping the stats and opening on either win 2000 pro or win xp. On win 2000 I use winzip with xp I use the default windows app.

I did use Arf's method of zipping up the stats...

User avatar
Sapphyre
Official Test Penquin
Posts: 3337
Joined: Tue Nov 19, 2002 2:50 am
Location: Canada

Sorry, the problem is with Win XP SP2

Post by Sapphyre » Tue Mar 15, 2005 3:17 pm

Sorry, I neglected to add - it's a problem as of Win XP SP2 - they added a so-called "attachment manager" which prevents you from opening emailed or downloaded files from untrusted sources. Also using the default app provided.

I was able to 'unblock' the zip file itself, but I think the items within the zip file (stats folder for a particular year) are subject to further blocking - but you can't unblock them because winzip won't even list them out until they are unblocked.

It's insane - some files are high risk unless they are opened in specific programs!
http://support.microsoft.com/default.as ... -us;883260

I did download zipgenius.it and it opened fine.

flaguy
Hard Drive Crasher
Posts: 725
Joined: Wed Nov 24, 2004 4:52 pm
Location: Florida

Post by flaguy » Tue Mar 15, 2005 3:30 pm

yes the win xp i used has the sp2 ...

User avatar
nada
Propeller head licensee
Posts: 85
Joined: Mon Aug 25, 2003 2:33 pm
Location: Cyprus, Lebanon, Nigeria

Re: Zip up stats for disk space conservation

Post by nada » Sat Dec 17, 2005 4:02 pm

Arf wrote:Here are some commands that you can use to reduce your disk space usage. The first one zips up the stats in all of the client's accounts for a particular year (Example shows 2002. If you want a different year, just change that number throughout):

Code: Select all

find /home/*/*-www/stats/ -maxdepth 1 -name www2002 | xargs -i zip -r www2002stats.zip {}


Now you can remove the directories that contain the stats:

Code: Select all

rm -Rf /home/*/www/stats/www2002/


Enjoy!


Greetings

I did the above, ie zipping 2004 stats

using:
find /home/*/*-www/stats/ -maxdepth 1 -name www2004 | xargs -i zip -r www2004stats.zip {}

It showed for an hour or so that it was deflating files, but in the end it gave countless errors like:
zip error: Zip file structure invalid (www2004stats.zip)
zip warning: count in end of central directory incorrect



and then when trying to delete them using :
rm -Rf /home/*/www/stats/www2004/

I am getting this error for all the zipped files.

rm: cannot open directory `/home/zoomsite/www/stats/www2004/': Too many open files

Any suggestions?
Rami El-Zein
admin@practicalhost.com
PracticalHost.com

User avatar
Arf
Official Test Penquin
Posts: 9103
Joined: Tue Apr 09, 2002 12:00 am
Location: IDAHO, USA
Contact:

Post by Arf » Sat Dec 17, 2005 9:21 pm

This is some very old code. I've learned a lot since writing it. So, here's the updated code. Just Change the "2002" at the beginning to any year you wish to manage amd run this is one very long command.

Code: Select all

 YEAR="2002";for STATS in `find /home/*/*-www/stats/ -maxdepth 1 -name www$YEAR`;do cd $STATS; cd ..; zip -r www$YEAR.zip www$YEAR; rm -Rf ./www$YEAR;done


To make sure this worked, I ran it on my server for 2002 and 2003 and it worked like a charm.

User avatar
Arf
Official Test Penquin
Posts: 9103
Joined: Tue Apr 09, 2002 12:00 am
Location: IDAHO, USA
Contact:

Post by Arf » Tue Jan 10, 2006 5:45 pm

Variation on the above theme. I ran the above and then wondered if it had really done it's job (it did but I'm paranoid). So below is the same script except that it shows a directory listing of the stats directory so you can "see" the wwwYEAR.zip file for yourself.

Code: Select all

YEAR="2004";for STATS in `find /home/*/*-www/stats/ -maxdepth 1 -name www$YEAR`;do cd $STATS; cd ..; zip -r www$YEAR.zip www$YEAR; rm -Rf ./www$YEAR;ls -al; pwd;sleep 2;done

User avatar
Arf
Official Test Penquin
Posts: 9103
Joined: Tue Apr 09, 2002 12:00 am
Location: IDAHO, USA
Contact:

Post by Arf » Thu Feb 04, 2010 4:59 pm

Welcome to 2009. It's that time once again where you run the above command to clear our last year's stats. Happy new year!

User avatar
Arf
Official Test Penquin
Posts: 9103
Joined: Tue Apr 09, 2002 12:00 am
Location: IDAHO, USA
Contact:

Post by Arf » Thu Feb 04, 2010 7:30 pm

Looks like navisite never cleared out the December 2009 stats directory. So you'll need this too.

Code: Select all

for directory in `find /home/*/*-logs/old/200912 -maxdepth 0`; do echo $directory; rm -Rf $directory;done

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest