Columbia University Information Technology

Service Alerts

RESOLVED SERVICE ALERTS

Resolved Service Alerts

Resolved Service Alerts

  • LAMP Servers
    06/05/2009 - 03:43pm to 07/27/2009 - 01:55pm

    Summary: Begining the afternoon of June 5th, many of Columbia University's Interactice Production web sites through CUIT's LAMP service were intermittently or completely unavailable due to an aggressive and sustained 'denial of services' effect whereby intruders generated massive number of access requests.  This delays or prevents legitimate webpage requests.  CUIT administrators have been working on the issue throughout the night. This type of attack is extremely difficult to resolve due to the instigating computers are so widespread, but some changes were made approx. 12am that were believed to have alleviated the impact.

    UPDATE 6/19/09:  Recent attempts to improve site security are causing intermittent issues when updating web content.  In order to resolve this problem, 3rd-party hosting partners may need to briefly update and restart CUIT's LAMP servers.  While this is expected NOT to result in any downtime, this cannot be guaranteed.  If there is a problem, it is anticipated that no site will be offline for longer than 30 minutes.

    UPDATE 6/12/09:  We continued to have intermittent problems with CUIT LAMP-hosted Columbia sites over the course of the week, but at 12:30pm today the issue spiked and a number of sites again became unavailable.  Most significantly impacted were the Columbia News and the Engineering department websites.  Steps were taken to alleviate the effects; full functionality resumed at approximately 3pm, with the known exception of Columbia News.  CUIT administrators continue their work on a permanent solution.

    UPDATE 6/8/09:   While issues were alleviated for the overnight, it recurred early this morning.  Administrators are testing various solutions, and as of 5:25pm you may find most sites are accessible (known exceptions: news, engineering).  This will continue to be monitored closely as a more permanent solution is implemented.

    UPDATE 7/1/09:  All issues being experienced by customers is now believed to be resolved.

    For reference, affected sites include:
    http://cees.columbia.edu
    http://cuitalerts.columbia.edu
    http://cupop.columbia.edu
    http://engineering.columbia.edu/web/faculty
    http://evpr.columbia.edu
    http://facets.columbia.edu
    http://gallery.columbia.edu
    http://isso.columbia.edu
    http://mailservices.columbia.edu
    http://qmss.columbia.edu
    http://researchinitiatives.columbia.edu
    http://sof.columbia.edu
    http://spa.columbia.edu
    http://sponsoredprojectshandbook.columbia.edu
    http://universityprograms.columbia.edu
    http://webservices.lamp.columbia.edu/drupal
    http://worldhistory.columbia.edu
    http://worldleaders.columbia.edu
    http://warmemorial.columbia.edu
    http://hr.columbia.edu
    http://facilities.columbia.edu

    For a complete list of recent and upcoming scheduled service changes, please visit http://www.columbia.edu/acis/alerts-list.html.

  • UNIX Hosting/Application Service
    07/02/2009 - 07:26pm to 07/03/2009 - 03:05am

    Begining 7/2/09 7:26pm, most online services requiring a login became unavailable due to some changes made to address a web page loading issue.  A successful resolution was implemented at 7/3/09 12:15am.   We recommend that if you are still experiencing an issue to restart your browser or email client and try again.   There may be temporary slowness, especially with Cubmail and other email clients as people start to access services at the same time.

    Services affected include almost all services requiring a WIND or UNI login (e.g., CubMail, Libraries restricted online resources, MyColumbia (including People@Columbia), Courseworks, etc.).   Other affected services include Cyrus email, Alpha Exchange web-accessed email, Dial-up connections, VPN access, CUNIX login and other authenticated web pages.

    Please note that Columbia web pages that do NOT require UNI logins may still be intermittently inaccessible due to the ongoing web issue discussed in another Service Alert.

    We sincerely apologize for this service interruption.

    UPDATE:  Cubmail was experiencing some instability from 2:30 until 3:15.  Users may experience some sluggishness for a period of time, but it seems to have stabilized.

     

    For a complete list of recent and upcoming scheduled service changes, please visit http://www.columbia.edu/acis/alerts-list.html.

  • CUNIX Web Servers
    07/02/2009 - 11:25am to 07/13/2009 - 10:43am

    We continue to see intermittent non-responsiveness or failover to the backup provider and are attempting various methods of handling or eliminating the increased traffic.  Users will see one of two error states:

    - www.columbia.edu is not responding
    - www.columbia.edurewrites to a www-fail.columbia.edu URL which indicates that the requested page is not available (on the backup provider).

    In some cases, users will be redirected to the backup provider successfully and not notice any problems.  This is because the backup provider has the most visible or most popular pages from www.columbia.edu , but does not have the full site.

    The System Administrators continue to work on resolving this problem.

     

    UPDATE:  This issue is believed to have been resolved.

  • Network - Ethernet and Wireless
    (Approximate) 07/09/2009 - 10:30am to 07/10/2009 - 05:00pm

    Due to a water main break, the electricity supply to our switches at 601 West 113th Street has been unstable since Thursday 7/9/2009 8:30am.

    The switch on the 12th floor, which provides network connectivity to 6th-12th floors, has been down since Thursday 7/9/2009 10:30am.

    The switch in the Basement, which provides network connectivity to Basement-5th floors, was down on Thursday 7/9/2009 10:30am-6:15pm. We anticipate it to go down again today (Friday 7/10/2009).

    Facilities informed that electricity should be restored to normal by 5pm today (Friday 7/10/2009).

    For a complete list of recent and upcoming scheduled service changes, please visit http://www.columbia.edu/acis/alerts-list.html.

  • Telephony - Third Party Service Provider
    08/04/2009 - 10:15am to 08/04/2009 - 11:15am
    Verizon is performing necessary maintenance on the underground wiring that feeds UTS. This scheduled maintenance is expected to be completed on Tuesday, August 4, between 10:15AM and 11:15AM.

    During this maintenance UTS will lose inbound and outbound Voice Service.