Columbia University Information Technology

Service Alerts

ISSUES WITH SOME COLUMBIA SITES - 6/19/09 UPDATE

Service: 
Service Impact: 
Alert Date: 
06/05/2009 - 03:43pm to 07/27/2009 - 01:55pm
Main Content
Alert Message: 

Summary: Begining the afternoon of June 5th, many of Columbia University's Interactice Production web sites through CUIT's LAMP service were intermittently or completely unavailable due to an aggressive and sustained 'denial of services' effect whereby intruders generated massive number of access requests.  This delays or prevents legitimate webpage requests.  CUIT administrators have been working on the issue throughout the night. This type of attack is extremely difficult to resolve due to the instigating computers are so widespread, but some changes were made approx. 12am that were believed to have alleviated the impact.

UPDATE 6/19/09:  Recent attempts to improve site security are causing intermittent issues when updating web content.  In order to resolve this problem, 3rd-party hosting partners may need to briefly update and restart CUIT's LAMP servers.  While this is expected NOT to result in any downtime, this cannot be guaranteed.  If there is a problem, it is anticipated that no site will be offline for longer than 30 minutes.

UPDATE 6/12/09:  We continued to have intermittent problems with CUIT LAMP-hosted Columbia sites over the course of the week, but at 12:30pm today the issue spiked and a number of sites again became unavailable.  Most significantly impacted were the Columbia News and the Engineering department websites.  Steps were taken to alleviate the effects; full functionality resumed at approximately 3pm, with the known exception of Columbia News.  CUIT administrators continue their work on a permanent solution.

UPDATE 6/8/09:   While issues were alleviated for the overnight, it recurred early this morning.  Administrators are testing various solutions, and as of 5:25pm you may find most sites are accessible (known exceptions: news, engineering).  This will continue to be monitored closely as a more permanent solution is implemented.

UPDATE 7/1/09:  All issues being experienced by customers is now believed to be resolved.

For reference, affected sites include:
http://cees.columbia.edu
http://cuitalerts.columbia.edu
http://cupop.columbia.edu
http://engineering.columbia.edu/web/faculty
http://evpr.columbia.edu
http://facets.columbia.edu
http://gallery.columbia.edu
http://isso.columbia.edu
http://mailservices.columbia.edu
http://qmss.columbia.edu
http://researchinitiatives.columbia.edu
http://sof.columbia.edu
http://spa.columbia.edu
http://sponsoredprojectshandbook.columbia.edu
http://universityprograms.columbia.edu
http://webservices.lamp.columbia.edu/drupal
http://worldhistory.columbia.edu
http://worldleaders.columbia.edu
http://warmemorial.columbia.edu
http://hr.columbia.edu
http://facilities.columbia.edu

For a complete list of recent and upcoming scheduled service changes, please visit http://www.columbia.edu/acis/alerts-list.html.