Forum Moderators: phranque

Message Too Old, No Replies

Linux server running out of space

         

csdude55

5:19 am on Oct 24, 2020 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I'm running CentOS 6.10 xen hvm with WHM / cPanel. It has a 100G SSD, but I'm constantly running out of space.

It's partitioned like:

Filesystem   Size Used Avail Use% Mounted on
/dev/xvda2 99G 92G 2.1G 98% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/xvda1 248M 79M 157M 34% /boot
/usr/tmpDSK 3.2G 206M 2.8G 7% /tmp


I'm constantly removing the .GZ files from /var/log/apache2/archive, and there doesn't seem to be an automatic way to just not have them.

I also have a few .GZ files at /var/log, but they're only about 10M combined so not as major.

Any suggestions on what else I can remove to clear up some space?

I'm also pretty confused by the /domlogs/ section. When I go to /var/log/apache2/domlogs I have files like "domain-ssl_log". But then I go to /var/log/apache2/domlogs/ACCOUNT and have the same, identical file! In my biggest site's case that one file is 300M, and I obviously don't need it twice. But I can't figure out how to change it to only have one copy.

csdude55

6:25 am on Oct 28, 2020 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I've kinda gotten a little gun shy over the years. Back in 2006 I had signed on with a cheaper company, but the server was CONSTANTLY going down. At one point, it was down for 2 WEEKS! And they swore the whole time that it wasn't on their end. They almost put me out of business when I was still kinda new.

I forget where it was suggested, but someone recommended The Planet (which I can safely say here because they no longer exist). They were more expensive, but their support was stellar! The staff remembered me and were friendly, they replied to tickets within minutes, they were constantly throwing discounts my way, and I never had a minute of down time.

They eventually sold out to another company, and service went way downhill. They THEY sold out to another company, and now it's horrible. But I still have no problems with hardware or down time, so I'm hesitant to move away.

Besides, it's a HUGE pain to change servers! So right now I'm looking at other providers, and I'm gonna try to twist my provider's arm a little and see if they'll price match.

Just curious ... have you run an optimize/compact on your database recently?

Noooo, last time I did it took FOREVER to finish! phpMyAdmin shows that I have 1.2M in "overhead", am I correct in understanding that all it would do is free up that 1.2M?

graeme_p

10:58 am on Oct 28, 2020 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Besides, it's a HUGE pain to change servers!


What happens if your server dies. Say a dead SSD which cannot be easily fixed? If you have server backups in the hands of the people runnig your server, I hop you trust them to have off-site backups recoverable in all circumstances. What about hack that affects some sites and users, but not others? A whole server restore is not ideal.

I've kinda gotten a little gun shy over the years. Back in 2006 I had signed on with a cheaper company, but the server was CONSTANTLY going down. At one point, it was down for 2 WEEKS!


I have managed my own servers and many clients servers for many years and we have never come anywhere near that. One cheap provider had on an off connectivity issues for an hour or so at a time, several times.

Self manage, have backups, have a way to restore quickly. The best approach IMO is to have tested scripted deployment. You can also backup /etc (even put it in version control - but be careful to exclude sensitive files) and content separately.

One advantage of VPSs is you can usually take automated baclup images of the whole VPS. I would still prefer to my own copy elsewhere.

JorgeV

12:41 pm on Oct 28, 2020 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month



I know what you mean, about worrying to change host. I perfectly understand this.

When I began using dedicated servers, in the early 2000s, i had mixed experiences, and then I found a host, which I am still with since 15+ years now. It happens they have offers from insanely cheap, to ultra expensive. So they have ranges for everybody, and every budget. I know, I wouldn't try anywhere else now.

By the way, I think we should be allowed to name companies/businesses, when they are ultra famous. We do when it comes to Google, Amazon, etc... so why not for big hosts too. This would not be spam.

csdude55

6:51 am on Oct 29, 2020 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



What happens if your server dies. Say a dead SSD which cannot be easily fixed?

In all honesty, @graeme_p... I'd be SOL. A few years back I had multiple servers and ran regular backups from one to the other, but as revenue started drying up (2017-ish) I've had to cut, cut, cut. I won't lie, a lot of my bills have been paid on credit cards for the last year or so. I have to try to balance between what's best, and what's the best that I can afford :-/

4GB for a dedicated server, this is really low today..

I meant to reply to this one earlier, @JorgeV. Before around November 2019 I was always floating between 3.5 and 4G, so it was high but OK. Then one day, out of the blue, BAM! 8G spike that never really went down. I went back and forth with cPanel and my provider for WEEKS, but no one could ever find an explanation for it.

Even right now, at 2:28am, I'm red lining. CPU load is practically non-existent, but it's still using 4G of RAM. MySQL seems like the only thing using memory, but there are literally no processing running (per "SHOW FULL PROCESSLIST;"). I KNOW there's a software reason that I'm swapping so much, I'm just at a loss on figuring out why.

If you're interested, here are my results from top:

top - 02:28:06 up 179 days, 44 min, 1 user, load average: 0.03, 0.14, 0.19
Tasks: 168 total, 1 running, 166 sleeping, 0 stopped, 1 zombie
Cpu(s): 2.0%us, 2.9%sy, 0.0%ni, 94.4%id, 0.0%wa, 0.0%hi, 0.5%si, 0.2%st
Mem: 3974360k total, 3930132k used, 44228k free, 41900k buffers
Swap: 2096440k total, 601804k used, 1494636k free, 1281652k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19979 nobody 20 0 0 0 0 Z 2.6 0.0 0:00.22 [httpd] <defunct>
27358 mysql 20 0 2350m 604m 4492 S 1.3 15.6 1692:54 /usr/sbin/mysqld --
4117 cpanelco 20 0 11540 5188 4336 S 1.0 0.1 17:37.59 /usr/local/cpanel/3
20098 root 20 0 13132 1288 936 R 1.0 0.0 0:00.22 top -c
18034 nobody 20 0 474m 26m 8708 S 0.7 0.7 0:00.80 /usr/sbin/httpd -k
18893 nobody 20 0 407m 25m 7424 S 0.7 0.7 0:00.72 /usr/sbin/httpd -k
19327 nobody 20 0 407m 24m 6612 S 0.7 0.6 0:00.39 /usr/sbin/httpd -k
203 root 20 0 0 0 0 S 0.3 0.0 400:02.54 [kjournald]
1287 root 20 0 18268 424 320 S 0.3 0.0 49:43.22 irqbalance --pid=/v
18125 nobody 20 0 475m 28m 8536 S 0.3 0.7 0:00.69 /usr/sbin/httpd -k
18895 nobody 20 0 407m 24m 7288 S 0.3 0.6 0:00.54 /usr/sbin/httpd -k
19011 nobody 20 0 474m 27m 8428 S 0.3 0.7 0:00.53 /usr/sbin/httpd -k
19016 nobody 20 0 473m 25m 8004 S 0.3 0.7 0:00.50 /usr/sbin/httpd -k
19036 nobody 20 0 407m 25m 7472 S 0.3 0.6 0:00.46 /usr/sbin/httpd -k
19243 nobody 20 0 474m 26m 8040 S 0.3 0.7 0:00.32 /usr/sbin/httpd -k
19244 nobody 20 0 474m 26m 8696 S 0.3 0.7 0:00.40 /usr/sbin/httpd -k


And mysqltuner didn't really suggest anything other than increasing query_cache_size (currently 128M):

>> MySQLTuner 1.3.0 - Major Hayden <major@mhtx.net>
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering
[OK] Currently running supported MySQL version 5.5.62-cll
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[--] Data in MyISAM tables: 8G (Tables: 639)
[--] Data in InnoDB tables: 43M (Tables: 140)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[--] Data in MEMORY tables: 0B (Tables: 2)
[!] Total fragmented tables: 61

-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 11d 22h 50m 54s (77M q [75.044 qps], 9M conn, TX: 73B, RX: 8B)
[--] Reads / Writes: 37% / 63%
[--] Total buffers: 1.3G global + 8.4M per thread (150 max threads)
[OK] Maximum possible memory usage: 2.5G (65% of installed RAM)
[OK] Slow queries: 0% (731/77M)
[OK] Highest usage of available connections: 38% (58/150)
[OK] Key buffer size / total MyISAM indexes: 1.0G/5.4G
[OK] Key buffer hit rate: 100.0% (2B cached / 483K reads)
[OK] Query cache efficiency: 62.8% (19M cached / 31M selects)
[!] Query cache prunes per day: 18272
[OK] Sorts requiring temporary tables: 0% (607 temp sorts / 6M sorts)
[OK] Temporary tables created on disk: 3% (245K on disk / 6M total)
[OK] Thread cache hit rate: 99% (219 created / 9M connections)
[OK] Table cache hit rate: 99% (969 open / 976 opened)
[OK] Open file limit used: 14% (1K/10K)
[OK] Table locks acquired immediately: 99% (31M immediate / 31M locks)
[OK] InnoDB buffer pool / data size: 120.0M/43.2M
[OK] InnoDB log waits: 0
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
Variables to adjust:
query_cache_size (> 128M)


My "max Apache processes" for the month is 131, so I have the limit set to 150. I have MySQL's max_connections set to 150 to match, but I see that it's only used 58 of them. Maybe I could lower that to save some RAM? But I changed that in 2017, so that wouldn't explain the jump in November 2019.

csdude55

7:00 am on Oct 29, 2020 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



By the way, I think we should be allowed to name companies/businesses, when they are ultra famous. We do when it comes to Google, Amazon, etc... so why not for big hosts too. This would not be spam.

FWIW, I gave that a little thought today... as a moderator of my own board, I feel their pain on this one. How do you define "ultra famous"? And do you really want to encourage people to only promote big companies and never promote small companies? As a small business owner myself, the mere idea hurts my soul! :-O LOL

If there were to be a change at all (which I hope there is), I think it would have to be a technical one; eg, members that are X years old, have made Y number of posts, and/or have had Z number of votes could be allowed to recommend hosting companies, but new or unestablished users couldn't. There would have to be a defined value, though, so that the system could control it instead of relying on human mods to do it.

Or an easier option, they could create a subforum for hosting and then just allow open discussion on it. Spammers would be annoying, of course, but once they've spammed a few times and gotten on everyone's nerves then they would be hurting themselves more than anything else. And who knows, one or two of them might actually take part in conversations and become an asset... I know, I know, but it could happen! LOL

That's just my $0.02, though. I've been in those same shoes trying to create a fair and detailed policy change, and it can get really complicated once you start pulling threads.

phranque

8:26 am on Oct 29, 2020 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



By the way, I think we should be allowed to name companies/businesses, when they are ultra famous.

btw regarding discussion of hosting companies, from the Webmaster General forum Charter [webmasterworld.com]:
Hosting Questions:
Public discussion and/or recommendation of hosting companies is no longer allowed at WebmasterWorld. We are closing the floodgates to get a better handle on the spam that always crops up in hosting discussions.

robzilla

9:40 am on Oct 29, 2020 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



My "max Apache processes" for the month is 131, so I have the limit set to 150. I have MySQL's max_connections set to 150 to match, but I see that it's only used 58 of them. Maybe I could lower that to save some RAM? But I changed that in 2017, so that wouldn't explain the jump in November 2019.

In the short term, a solution is more important than an explanation :-)

It's not the full process list, so I can't tell how many httpd processes are actually running here, but one thing you could do is put a lightweight web server like nginx in front of Apache to handle all static file requests, and pass everything else to Apache. Then you can probably reduce the number of active Apache workers and/or reduce their idle time to save RAM. Apache with modules for PHP etc. built-in is much heavier than nginx.

Try the atop utility to see accumulated memory usage for each service.

yum install atop
atop

Press 'M' to sort by memory usage, press 'P' to see consumption per program (as opposed to per process). On the far right, you'll see the program (CMD) and the percentage of memory it uses (MEM).

MEM CMD 
8% mysqld
7% php-fpm
3% sshd
3% nginx
1% php
1% atop
1% rsyslogd
0% munin-node

tangor

5:16 am on Oct 30, 2020 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Time involved or not, still believe you should compact your mySQL database! WEEKLY (will run faster each time you do so!).

I get between 25% to 45% compaction each time on a highly active add/delete database. Makes a difference.

YMMV

JorgeV

12:18 pm on Oct 31, 2020 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month



I meant to reply to this one earlier, @JorgeV. Before around November 2019 I was always floating between 3.5 and 4G, so it was high but OK. Then one day, out of the blue, BAM! 8G spike that never really went down. I went back and forth with cPanel and my provider for WEEKS, but no one could ever find an explanation for it.

Hum, that deserve investigations indeed.

I am NOT recommending it, but, if I was in such situation, I would reboot my server, and study how the memory usage is growing.(keeping in mind, that, as long as memory is "really" available, the system will use it for "buffering" and "caching").

Your client, who has a huge mailbox, I wonder if, the process is reading the whole file(s), each time she connects.
This 39 message thread spans 2 pages: 39