SysAdmins are like Auto Mechanics
Posted by Steven in Work on March 16th, 2006 at 08:16am
I feel like an auto mechanic working on a lemon. I’ve never encountered so many problems with my server. All within 12 hours:
- MySQL’s user table corrupted, MySQL crashed, /tmp/mysql.sock link vanished, MySQL wouldn’t start
(this ruined our plan to feed some ducks at Mission Bay)
- Apache crashed for no good reason, but restarted on second try
- My apache error log reached 2 GB and started overflowing
- I found that my HTTPS service was dead all night long!
- My datacenter decided they won’t do kernel upgrades anymore, so I did it myself, borked it and then had to learn to fix grub.conf
I usually don’t have this many problems in a month! I’ve learned a few things though and that’s good.
4 Comments Add your own
1. Dan | March 16th, 2006 at 10:55 am
That sucks. You know, I used to have the /tmp/mysql.sock missing link problem. Mark helped me with it and it turns out some service designed to clean up old temp files or something was deleting my /tmp/mysql.sock link. So we added a cron job which recreates the link every day. So it never gets old and it never gets deleted anymore. LOL.
2GB for a LOG???!!!??? WOW. Filled with what? 2 billion lines of the same error? You don’t suppose someone is trying to attack your server do you?
Now go wash off that grease of your hands, don’t wana get your keyboard all gunky.
2. Steven | March 16th, 2006 at 11:52 am
It actually wasn’t my error log, it was my digitalart.org log. When it reached 2 GB (I think) my error_log started filling up with messages saying my da.o log reached its limit. This happens every month or so and I really should setup a cronjob to keep things in check every now and then. I don’t run any log analyzers, which usually clean this stuff up on their own.
3. Mark | March 16th, 2006 at 10:54 pm
If you don’t run log analyzers, why don’t you turn off general loging? Just keep the error log.
Also, might look into some log rotate utilities. You can probably find an RPM for one.
As for Dan’s situation, he has a cron which empties the /tmp directory. So I just appended that cron with a line that recreated the link. On second thought, I should have modified the command to sipmly omit the link.
4. Dan | March 16th, 2006 at 11:23 pm
But that would make too much sense.
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
Subscribe to the comments via RSS Feed