The Unofficial Alabanza Forums Forum Index The Unofficial Alabanza Forums
To access Hidden Members' forums, fill out membership page
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Zip up stats for disk space conservation

 
Post new topic   Reply to topic    The Unofficial Alabanza Forums Forum Index -> Server Candy (Public)
View previous topic :: View next topic  
Author Message
Arf
Official Test Penquin


Joined: 09 Apr 2002
Posts: 8327
Location: IDAHO, USA

PostPosted: Sat Sep 27, 2003 12:12 am    Post subject: Zip up stats for disk space conservation Reply with quote

Please note that this code has been revised as of 12/17/05. Look for my updated code created on that date.

Here are some commands that you can use to reduce your disk space usage. The first one zips up the stats in all of the client's accounts for a particular year (Example shows 2002. If you want a different year, just change that number throughout):

Code:
find /home/*/*-www/stats/ -maxdepth 1 -name www2002 | xargs -i zip -r www2002stats.zip {}


Now you can remove the directories that contain the stats:

Code:
rm -Rf /home/*/www/stats/www2002/


Enjoy!
_________________
Thomas
ez2ba.com (easy to be a dot-com)


Last edited by Arf on Sat Dec 17, 2005 9:24 pm; edited 2 times in total
Back to top
View user's profile Send private message
rldev
Hosting Superstar


Joined: 05 Aug 2003
Posts: 1067

PostPosted: Sat Sep 27, 2003 2:05 am    Post subject: Reply with quote

Thanks Arf, this is good stuff.
Back to top
View user's profile Send private message
naplesdave
Nothing better to do.


Joined: 14 Apr 2003
Posts: 243
Location: Naples, FL

PostPosted: Tue Nov 18, 2003 10:56 pm    Post subject: Reply with quote

Many of my customers don't want stats at all. We frown on turning off stats, because we want to sell them on adding more space for keeping stats.

But when we do turn them off, we simply remove the entire stats directory. It will be recreated by the stats software, so what we do is just create a touch file named "stats" and this keeps the software from creating a directory.

rm -rf stats

The command to create the placeholder file is:

touch stats

That creates a file of zero bytes named stats. When the stats program tries to create a directory named stats, it will error out because a file with that name is already in place.
_________________
Kind regards,
Dave Jackson
World Wide Mart, Inc.
Back to top
View user's profile Send private message AIM Address Yahoo Messenger
knopfler
Nothing better to do.


Joined: 26 Nov 2003
Posts: 163

PostPosted: Thu Nov 27, 2003 3:40 pm    Post subject: Reply with quote

Very useful, thanks a lot!
Back to top
View user's profile Send private message
sixpackmx
Hard Drive Crasher


Joined: 01 Nov 2002
Posts: 639
Location: Mexico City, Mexico

PostPosted: Thu Nov 27, 2003 4:20 pm    Post subject: Reply with quote

Dave:

Nice solution, still, there is another workaround. Create a .NoStats file on the user's home.

We have packages that does not include stats because of their limited size. We run a script that compares user quota and creats the .NoStats file accordingly.

Also, our clients have to option to disable stats from a CP Custom Tool.

Best wishes,

SixpackMX
Back to top
View user's profile Send private message
Anonymous
Nothing better to do.


Joined: 12 May 1978
Posts: 307

PostPosted: Sat Nov 29, 2003 9:23 pm    Post subject: Reply with quote

Cool! I would be interested in both.
Back to top
View user's profile Send private message
naplesdave
Nothing better to do.


Joined: 14 Apr 2003
Posts: 243
Location: Naples, FL

PostPosted: Sat Jan 17, 2004 3:43 pm    Post subject: Reply with quote

I have tried .NoStats on several accounts and it doesn't seem to be working. What's the trick?
_________________
Kind regards,
Dave Jackson
World Wide Mart, Inc.
Back to top
View user's profile Send private message AIM Address Yahoo Messenger
Arf
Official Test Penquin


Joined: 09 Apr 2002
Posts: 8327
Location: IDAHO, USA

PostPosted: Sat Jan 17, 2004 9:48 pm    Post subject: Reply with quote

see Ala KB #411 for information.
Back to top
View user's profile Send private message
naplesdave
Nothing better to do.


Joined: 14 Apr 2003
Posts: 243
Location: Naples, FL

PostPosted: Sat Jan 17, 2004 11:52 pm    Post subject: Reply with quote

So according to KB 411 it should stop updating /stats and shouldn't even create a dir named /stats if you have deleted it.

I suppose I should get with Support. Several of mine are continuing to pump out data.
_________________
Kind regards,
Dave Jackson
World Wide Mart, Inc.
Back to top
View user's profile Send private message AIM Address Yahoo Messenger
Heather
Navihost.com-Newbie


Joined: 21 Sep 2004
Posts: 6
Location: London, Ontario, Canada

PostPosted: Wed Sep 22, 2004 4:11 am    Post subject: Reply with quote

Good tips!

Some clients do want their stats, but when you have a fairly busy website, I've found it's the referrers files that really bulk up the stats. It is fairly easy to instruct clients to go into their stats via the file manager and remove all of the lrefers****.html files. It's easy to see these are usually the largest files if you sort the files according to size. Most people won't miss these much, or the lagents****.html ones either which are normally the second largest group of files. I have a client whose lrefers files are about 10MB for each year of stats.
Heather
Back to top
View user's profile Send private message Visit poster's website
RobW
Propeller head licensee


Joined: 10 Nov 2004
Posts: 45

PostPosted: Thu Jan 06, 2005 2:47 pm    Post subject: Reply with quote

If you just want to see how much of your server disk space is used by the stats for a particular year, you can run the following command:

Code:
find /home*/*/*-www/stats/ -maxdepth 1 -name www2002  | xargs du -hc


This produces a size for each client plus a total at the bottom

Cheers
Rob
Back to top
View user's profile Send private message
flaguy
Hard Drive Crasher


Joined: 24 Nov 2004
Posts: 712
Location: Florida

PostPosted: Fri Jan 07, 2005 1:23 pm    Post subject: Reply with quote

Worked great Rob! thanks!
Back to top
View user's profile Send private message
Sapphyre
Official Test Penquin


Joined: 19 Nov 2002
Posts: 2916
Location: Canada

PostPosted: Tue Mar 15, 2005 1:50 am    Post subject: Won't open in winzip ? Reply with quote

I did this

Code:
cd /home/$user/$domain-www/stats/www2004
zip -r ../www2004.zip ./*


It created the file just fine.
I go by browser to http://domain.com/stats/www2004.zip
I have tried both opening the archive directly or saving it to my desktop first and then opening it (Win XP) - both times the archive shows 0 objects.

OTOH, I try unzip on the server and all the files are retrieved just fine.

How do I make a Windows compatible zip ?
Back to top
View user's profile Send private message
Sapphyre
Official Test Penquin


Joined: 19 Nov 2002
Posts: 2916
Location: Canada

PostPosted: Tue Mar 15, 2005 3:21 am    Post subject: SP2 prevents opening zips Reply with quote

..supposed to prevent opening them from untrusted sites; however I have found that even if I add the site I am downloading the zip from to 'trusted sites', I still can't open the zip. The other suggestion was to use to navigate to the file, right click - properties - and click the 'Unblock' button. Still doesn't open.

looking on Microsoft's newsgroup - post entitled "Blocking" dated 1.24.05
http://www.microsoft.com/windowsxp/expertzone/newsgroups/reader.mspx?query=XP+would+not+let+me+extract
Got an MVP telling people to get a third party unzip tool!

This is just outrageous. MS has really gone too far with the security precuations.
Back to top
View user's profile Send private message
Anonymous
Nothing better to do.


Joined: 12 May 1978
Posts: 307

PostPosted: Tue Mar 15, 2005 1:16 pm    Post subject: Reply with quote

Sapphyre I had no problem zipping the stats and opening on either win 2000 pro or win xp. On win 2000 I use winzip with xp I use the default windows app.

I did use Arf's method of zipping up the stats...
Back to top
View user's profile Send private message
Sapphyre
Official Test Penquin


Joined: 19 Nov 2002
Posts: 2916
Location: Canada

PostPosted: Tue Mar 15, 2005 3:17 pm    Post subject: Sorry, the problem is with Win XP SP2 Reply with quote

Sorry, I neglected to add - it's a problem as of Win XP SP2 - they added a so-called "attachment manager" which prevents you from opening emailed or downloaded files from untrusted sources. Also using the default app provided.

I was able to 'unblock' the zip file itself, but I think the items within the zip file (stats folder for a particular year) are subject to further blocking - but you can't unblock them because winzip won't even list them out until they are unblocked.

It's insane - some files are high risk unless they are opened in specific programs!
http://support.microsoft.com/default.aspx?scid=kb;en-us;883260

I did download zipgenius.it and it opened fine.
Back to top
View user's profile Send private message
flaguy
Hard Drive Crasher


Joined: 24 Nov 2004
Posts: 712
Location: Florida

PostPosted: Tue Mar 15, 2005 3:30 pm    Post subject: Reply with quote

yes the win xp i used has the sp2 ...
Back to top
View user's profile Send private message
nada
Propeller head licensee


Joined: 25 Aug 2003
Posts: 85
Location: Cyprus, Lebanon, Nigeria

PostPosted: Sat Dec 17, 2005 4:02 pm    Post subject: Re: Zip up stats for disk space conservation Reply with quote

Arf wrote:
Here are some commands that you can use to reduce your disk space usage. The first one zips up the stats in all of the client's accounts for a particular year (Example shows 2002. If you want a different year, just change that number throughout):

Code:
find /home/*/*-www/stats/ -maxdepth 1 -name www2002 | xargs -i zip -r www2002stats.zip {}


Now you can remove the directories that contain the stats:

Code:
rm -Rf /home/*/www/stats/www2002/


Enjoy!


Greetings

I did the above, ie zipping 2004 stats

using:
find /home/*/*-www/stats/ -maxdepth 1 -name www2004 | xargs -i zip -r www2004stats.zip {}

It showed for an hour or so that it was deflating files, but in the end it gave countless errors like:
zip error: Zip file structure invalid (www2004stats.zip)
zip warning: count in end of central directory incorrect



and then when trying to delete them using :
rm -Rf /home/*/www/stats/www2004/

I am getting this error for all the zipped files.

rm: cannot open directory `/home/zoomsite/www/stats/www2004/': Too many open files

Any suggestions?
_________________
Rami El-Zein
admin@practicalhost.com
PracticalHost.com
Back to top
View user's profile Send private message Send e-mail MSN Messenger
Arf
Official Test Penquin


Joined: 09 Apr 2002
Posts: 8327
Location: IDAHO, USA

PostPosted: Sat Dec 17, 2005 9:21 pm    Post subject: Reply with quote

This is some very old code. I've learned a lot since writing it. So, here's the updated code. Just Change the "2002" at the beginning to any year you wish to manage amd run this is one very long command.

Code:
 YEAR="2002";for STATS in `find /home/*/*-www/stats/ -maxdepth 1 -name www$YEAR`;do cd $STATS; cd ..; zip -r www$YEAR.zip www$YEAR; rm -Rf ./www$YEAR;done


To make sure this worked, I ran it on my server for 2002 and 2003 and it worked like a charm.
_________________
Thomas
ez2ba.com (easy to be a dot-com)
Back to top
View user's profile Send private message
Arf
Official Test Penquin


Joined: 09 Apr 2002
Posts: 8327
Location: IDAHO, USA

PostPosted: Tue Jan 10, 2006 5:45 pm    Post subject: Reply with quote

Variation on the above theme. I ran the above and then wondered if it had really done it's job (it did but I'm paranoid). So below is the same script except that it shows a directory listing of the stats directory so you can "see" the wwwYEAR.zip file for yourself.

Code:
YEAR="2004";for STATS in `find /home/*/*-www/stats/ -maxdepth 1 -name www$YEAR`;do cd $STATS; cd ..; zip -r www$YEAR.zip www$YEAR; rm -Rf ./www$YEAR;ls -al; pwd;sleep 2;done
Back to top
View user's profile Send private message
Arf
Official Test Penquin


Joined: 09 Apr 2002
Posts: 8327
Location: IDAHO, USA

PostPosted: Thu Feb 04, 2010 4:59 pm    Post subject: Reply with quote

Welcome to 2009. It's that time once again where you run the above command to clear our last year's stats. Happy new year!
Back to top
View user's profile Send private message
Arf
Official Test Penquin


Joined: 09 Apr 2002
Posts: 8327
Location: IDAHO, USA

PostPosted: Thu Feb 04, 2010 7:30 pm    Post subject: Reply with quote

Looks like navisite never cleared out the December 2009 stats directory. So you'll need this too.

Code:
for directory in `find /home/*/*-logs/old/200912 -maxdepth 0`; do echo $directory; rm -Rf $directory;done
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    The Unofficial Alabanza Forums Forum Index -> Server Candy (Public) All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group