High Server Load

Discussion in 'Open Source' started by Deep, Nov 14, 2004.

Thread Status:
Not open for further replies.
  1. Deep

    Deep Version 2.0

    Joined:
    Jan 23, 2004
    Messages:
    977
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Mumbai
    Hi,
    We are facing High Server load problem for last few days..

    and it happens at perticular time only (around 1 AM IST ) but not everyday..

    we checked the cron jobs running but did not find any job running at that time...

    somtimes mysql connections goes upto 150 and sometimes 20 but the server load is always 1+ at that time...

    any ideas about it ?

    can it be bcoz of zombie processes ? I think DDOS attack is not possible coz server is totally managed and we have our own hardware firewall too...

    i googled a lot about it but couldnt find proper reason for it...

    Server config :
    OS - Redhat Linux 3 Enterprise
    AMD 2400+
    1 Gig RAM
    73 GB SCSI HDD i suppose (Not very sure about the HDD size but its SCSI for sure)

    CICSO Hardware firewall

    Tape backups are being taken twice a week by Rackapce.com (can that cause the high load? )

    ideas please...
    Deep
     
  2. GNUrag

    GNUrag FooBar Guy

    Joined:
    Jun 22, 2004
    Messages:
    1,246
    Likes Received:
    5
    Trophy Points:
    0
    Location:
    Interwebs
    could you get the output of this command [ # ps aux ]when the load is at its highest ?

    ps: Rackspace should be having competant sys admins ... did you talk to them also ? Tape backups dont cause any server overload.... our HBCSE server is also backed up every week.... never faced any sluggishness...
     
  3. OP
    OP
    Deep

    Deep Version 2.0

    Joined:
    Jan 23, 2004
    Messages:
    977
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Mumbai
    actually i had done that when it had happend in the day time and that time we knew the reason...

    that was one mysql script which was causing the problem...

    but this is kind of odd time...and we have given ssh access to only One IP i.e. our office IP for security reasons

    I havent asked rackspace about it coz before asking then I want to check what can be the possibilties....

    I am going to write them tmrw...and will also find out the things from our side...

    u know any possibilities which can cause this kind of problem.. ?

    Deep
     
  4. Guest

    Guest Guest

    Turn on system accouting and use sa command to find out this is the only way to get snapshot of today, y’day etc!
     
  5. OP
    OP
    Deep

    Deep Version 2.0

    Joined:
    Jan 23, 2004
    Messages:
    977
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Mumbai
    thanks for the info..

    gonna try it soon :)

    Regards
    Deep
     
  6. firewall

    firewall New Member

    Joined:
    Mar 31, 2004
    Messages:
    299
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Kolkata - INDIA
    hi, Deep u have a nice config. ! btw Server load is nothing to do with Zombie. As Zombies are already dead... they dont have much to do with server loadd.

    now on the reason of that.. well i think i cant say anything on that by not seeing the server... ;)
    Btw you ca nchec kthe mySQL.. coz thwere is a latest exploits is released.

    A .diff file, applied to the MySQL 5.0.0-alpha source distribution will allow building a MySQL client that can be used to connect to a remote MySQL server with no password.

    This could be a reason.. some one may applying this.... ;) as mySQL AB hasn't released a patch you may try to close the remote port to get rid of that...

    btw here is that exploit code....



    :)
    Code:
    1941,1956c1941,1942
    <   if (passwd[0])
    <   {
    <     if (mysql->server_capabilities & CLIENT_SECURE_CONNECTION)
    <     {
    <       *end++= SCRAMBLE_LENGTH;
    <       scramble(end, mysql->scramble, passwd);
    <       end+= SCRAMBLE_LENGTH;
    <     }
    <     else
    <     {
    <       scramble_323(end, mysql->scramble, passwd);
    <       end+= SCRAMBLE_LENGTH_323 + 1;
    <     }
    <   }
    <   else
    <     *end++= '\0';                               /* empty password */
    ---
    >   sprintf(end,"\x14\x00");
    >   end+=2; 

    :)
     
  7. OP
    OP
    Deep

    Deep Version 2.0

    Joined:
    Jan 23, 2004
    Messages:
    977
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Mumbai
    I suppose remote port is closed coz when u write telnet ip 3306 it shows connection to the server lost

    i said zombie coz i have heard that if too many dead processes are stored in zombie then it might take more ram and make the server slow or something...

    for example 2 many unclosed mysql connection in zombie might lead fo more use of memory or something but i am not very sure about it..just a guess..

    anyways i am going to write to racksapce tmrw i.e. after checking few things on server today and see what they are saying..

    EDIT : woha you posted code also..lol we will try it somtime on our server..haha

    Deep
     
  8. firewall

    firewall New Member

    Joined:
    Mar 31, 2004
    Messages:
    299
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Kolkata - INDIA
    kool... you have ur solution man.... ! Rackspace.... :)

    ;)
     
Thread Status:
Not open for further replies.

Share This Page