| View previous topic :: View next topic |
| Author |
Message |
Arf Official Test Penquin

Joined: 09 Apr 2002 Posts: 8327 Location: IDAHO, USA
|
Posted: Sat Sep 27, 2003 12:12 am Post subject: Zip up stats for disk space conservation |
|
|
Please note that this code has been revised as of 12/17/05. Look for my updated code created on that date.
Here are some commands that you can use to reduce your disk space usage. The first one zips up the stats in all of the client's accounts for a particular year (Example shows 2002. If you want a different year, just change that number throughout):
| Code: | | find /home/*/*-www/stats/ -maxdepth 1 -name www2002 | xargs -i zip -r www2002stats.zip {} |
Now you can remove the directories that contain the stats:
| Code: | | rm -Rf /home/*/www/stats/www2002/ |
Enjoy! _________________ Thomas
ez2ba.com (easy to be a dot-com)
Last edited by Arf on Sat Dec 17, 2005 9:24 pm; edited 2 times in total |
|
| Back to top |
|
 |
rldev Hosting Superstar
Joined: 05 Aug 2003 Posts: 1067
|
Posted: Sat Sep 27, 2003 2:05 am Post subject: |
|
|
| Thanks Arf, this is good stuff. |
|
| Back to top |
|
 |
naplesdave Nothing better to do.
Joined: 14 Apr 2003 Posts: 243 Location: Naples, FL
|
Posted: Tue Nov 18, 2003 10:56 pm Post subject: |
|
|
Many of my customers don't want stats at all. We frown on turning off stats, because we want to sell them on adding more space for keeping stats.
But when we do turn them off, we simply remove the entire stats directory. It will be recreated by the stats software, so what we do is just create a touch file named "stats" and this keeps the software from creating a directory.
rm -rf stats
The command to create the placeholder file is:
touch stats
That creates a file of zero bytes named stats. When the stats program tries to create a directory named stats, it will error out because a file with that name is already in place. _________________ Kind regards,
Dave Jackson
World Wide Mart, Inc. |
|
| Back to top |
|
 |
knopfler Nothing better to do.

Joined: 26 Nov 2003 Posts: 163
|
Posted: Thu Nov 27, 2003 3:40 pm Post subject: |
|
|
| Very useful, thanks a lot! |
|
| Back to top |
|
 |
sixpackmx Hard Drive Crasher
Joined: 01 Nov 2002 Posts: 639 Location: Mexico City, Mexico
|
Posted: Thu Nov 27, 2003 4:20 pm Post subject: |
|
|
Dave:
Nice solution, still, there is another workaround. Create a .NoStats file on the user's home.
We have packages that does not include stats because of their limited size. We run a script that compares user quota and creats the .NoStats file accordingly.
Also, our clients have to option to disable stats from a CP Custom Tool.
Best wishes,
SixpackMX |
|
| Back to top |
|
 |
Anonymous Nothing better to do.
Joined: 12 May 1978 Posts: 307
|
Posted: Sat Nov 29, 2003 9:23 pm Post subject: |
|
|
| Cool! I would be interested in both. |
|
| Back to top |
|
 |
naplesdave Nothing better to do.
Joined: 14 Apr 2003 Posts: 243 Location: Naples, FL
|
Posted: Sat Jan 17, 2004 3:43 pm Post subject: |
|
|
I have tried .NoStats on several accounts and it doesn't seem to be working. What's the trick? _________________ Kind regards,
Dave Jackson
World Wide Mart, Inc. |
|
| Back to top |
|
 |
Arf Official Test Penquin

Joined: 09 Apr 2002 Posts: 8327 Location: IDAHO, USA
|
Posted: Sat Jan 17, 2004 9:48 pm Post subject: |
|
|
| see Ala KB #411 for information. |
|
| Back to top |
|
 |
naplesdave Nothing better to do.
Joined: 14 Apr 2003 Posts: 243 Location: Naples, FL
|
Posted: Sat Jan 17, 2004 11:52 pm Post subject: |
|
|
So according to KB 411 it should stop updating /stats and shouldn't even create a dir named /stats if you have deleted it.
I suppose I should get with Support. Several of mine are continuing to pump out data. _________________ Kind regards,
Dave Jackson
World Wide Mart, Inc. |
|
| Back to top |
|
 |
Heather Navihost.com-Newbie
Joined: 21 Sep 2004 Posts: 6 Location: London, Ontario, Canada
|
Posted: Wed Sep 22, 2004 4:11 am Post subject: |
|
|
Good tips!
Some clients do want their stats, but when you have a fairly busy website, I've found it's the referrers files that really bulk up the stats. It is fairly easy to instruct clients to go into their stats via the file manager and remove all of the lrefers****.html files. It's easy to see these are usually the largest files if you sort the files according to size. Most people won't miss these much, or the lagents****.html ones either which are normally the second largest group of files. I have a client whose lrefers files are about 10MB for each year of stats.
Heather |
|
| Back to top |
|
 |
RobW Propeller head licensee
Joined: 10 Nov 2004 Posts: 45
|
Posted: Thu Jan 06, 2005 2:47 pm Post subject: |
|
|
If you just want to see how much of your server disk space is used by the stats for a particular year, you can run the following command:
| Code: | | find /home*/*/*-www/stats/ -maxdepth 1 -name www2002 | xargs du -hc |
This produces a size for each client plus a total at the bottom
Cheers
Rob |
|
| Back to top |
|
 |
flaguy Hard Drive Crasher
Joined: 24 Nov 2004 Posts: 712 Location: Florida
|
Posted: Fri Jan 07, 2005 1:23 pm Post subject: |
|
|
| Worked great Rob! thanks! |
|
| Back to top |
|
 |
Sapphyre Official Test Penquin

Joined: 19 Nov 2002 Posts: 2916 Location: Canada
|
Posted: Tue Mar 15, 2005 1:50 am Post subject: Won't open in winzip ? |
|
|
I did this
| Code: | cd /home/$user/$domain-www/stats/www2004
zip -r ../www2004.zip ./* |
It created the file just fine.
I go by browser to http://domain.com/stats/www2004.zip
I have tried both opening the archive directly or saving it to my desktop first and then opening it (Win XP) - both times the archive shows 0 objects.
OTOH, I try unzip on the server and all the files are retrieved just fine.
How do I make a Windows compatible zip ? |
|
| Back to top |
|
 |
Sapphyre Official Test Penquin

Joined: 19 Nov 2002 Posts: 2916 Location: Canada
|
Posted: Tue Mar 15, 2005 3:21 am Post subject: SP2 prevents opening zips |
|
|
..supposed to prevent opening them from untrusted sites; however I have found that even if I add the site I am downloading the zip from to 'trusted sites', I still can't open the zip. The other suggestion was to use to navigate to the file, right click - properties - and click the 'Unblock' button. Still doesn't open.
looking on Microsoft's newsgroup - post entitled "Blocking" dated 1.24.05
http://www.microsoft.com/windowsxp/expertzone/newsgroups/reader.mspx?query=XP+would+not+let+me+extract
Got an MVP telling people to get a third party unzip tool!
This is just outrageous. MS has really gone too far with the security precuations. |
|
| Back to top |
|
 |
Anonymous Nothing better to do.
Joined: 12 May 1978 Posts: 307
|
Posted: Tue Mar 15, 2005 1:16 pm Post subject: |
|
|
Sapphyre I had no problem zipping the stats and opening on either win 2000 pro or win xp. On win 2000 I use winzip with xp I use the default windows app.
I did use Arf's method of zipping up the stats... |
|
| Back to top |
|
 |
Sapphyre Official Test Penquin

Joined: 19 Nov 2002 Posts: 2916 Location: Canada
|
Posted: Tue Mar 15, 2005 3:17 pm Post subject: Sorry, the problem is with Win XP SP2 |
|
|
Sorry, I neglected to add - it's a problem as of Win XP SP2 - they added a so-called "attachment manager" which prevents you from opening emailed or downloaded files from untrusted sources. Also using the default app provided.
I was able to 'unblock' the zip file itself, but I think the items within the zip file (stats folder for a particular year) are subject to further blocking - but you can't unblock them because winzip won't even list them out until they are unblocked.
It's insane - some files are high risk unless they are opened in specific programs!
http://support.microsoft.com/default.aspx?scid=kb;en-us;883260
I did download zipgenius.it and it opened fine. |
|
| Back to top |
|
 |
flaguy Hard Drive Crasher
Joined: 24 Nov 2004 Posts: 712 Location: Florida
|
Posted: Tue Mar 15, 2005 3:30 pm Post subject: |
|
|
| yes the win xp i used has the sp2 ... |
|
| Back to top |
|
 |
nada Propeller head licensee

Joined: 25 Aug 2003 Posts: 85 Location: Cyprus, Lebanon, Nigeria
|
Posted: Sat Dec 17, 2005 4:02 pm Post subject: Re: Zip up stats for disk space conservation |
|
|
| Arf wrote: | Here are some commands that you can use to reduce your disk space usage. The first one zips up the stats in all of the client's accounts for a particular year (Example shows 2002. If you want a different year, just change that number throughout):
| Code: | | find /home/*/*-www/stats/ -maxdepth 1 -name www2002 | xargs -i zip -r www2002stats.zip {} |
Now you can remove the directories that contain the stats:
| Code: | | rm -Rf /home/*/www/stats/www2002/ |
Enjoy! |
Greetings
I did the above, ie zipping 2004 stats
using:
find /home/*/*-www/stats/ -maxdepth 1 -name www2004 | xargs -i zip -r www2004stats.zip {}
It showed for an hour or so that it was deflating files, but in the end it gave countless errors like:
zip error: Zip file structure invalid (www2004stats.zip)
zip warning: count in end of central directory incorrect
and then when trying to delete them using :
rm -Rf /home/*/www/stats/www2004/
I am getting this error for all the zipped files.
rm: cannot open directory `/home/zoomsite/www/stats/www2004/': Too many open files
Any suggestions? _________________ Rami El-Zein
admin@practicalhost.com
PracticalHost.com |
|
| Back to top |
|
 |
Arf Official Test Penquin

Joined: 09 Apr 2002 Posts: 8327 Location: IDAHO, USA
|
Posted: Sat Dec 17, 2005 9:21 pm Post subject: |
|
|
This is some very old code. I've learned a lot since writing it. So, here's the updated code. Just Change the "2002" at the beginning to any year you wish to manage amd run this is one very long command.
| Code: | | YEAR="2002";for STATS in `find /home/*/*-www/stats/ -maxdepth 1 -name www$YEAR`;do cd $STATS; cd ..; zip -r www$YEAR.zip www$YEAR; rm -Rf ./www$YEAR;done |
To make sure this worked, I ran it on my server for 2002 and 2003 and it worked like a charm. _________________ Thomas
ez2ba.com (easy to be a dot-com) |
|
| Back to top |
|
 |
Arf Official Test Penquin

Joined: 09 Apr 2002 Posts: 8327 Location: IDAHO, USA
|
Posted: Tue Jan 10, 2006 5:45 pm Post subject: |
|
|
Variation on the above theme. I ran the above and then wondered if it had really done it's job (it did but I'm paranoid). So below is the same script except that it shows a directory listing of the stats directory so you can "see" the wwwYEAR.zip file for yourself.
| Code: | | YEAR="2004";for STATS in `find /home/*/*-www/stats/ -maxdepth 1 -name www$YEAR`;do cd $STATS; cd ..; zip -r www$YEAR.zip www$YEAR; rm -Rf ./www$YEAR;ls -al; pwd;sleep 2;done |
|
|
| Back to top |
|
 |
Arf Official Test Penquin

Joined: 09 Apr 2002 Posts: 8327 Location: IDAHO, USA
|
Posted: Thu Feb 04, 2010 4:59 pm Post subject: |
|
|
| Welcome to 2009. It's that time once again where you run the above command to clear our last year's stats. Happy new year! |
|
| Back to top |
|
 |
Arf Official Test Penquin

Joined: 09 Apr 2002 Posts: 8327 Location: IDAHO, USA
|
Posted: Thu Feb 04, 2010 7:30 pm Post subject: |
|
|
Looks like navisite never cleared out the December 2009 stats directory. So you'll need this too.
| Code: | | for directory in `find /home/*/*-logs/old/200912 -maxdepth 0`; do echo $directory; rm -Rf $directory;done |
|
|
| Back to top |
|
 |
|