Passenger Trains > Amtrak computers now out for almost 24 hours

At approximately 100pm EDT on Saturday August 25, Amtrak's ARROW computer system went completely down.

No reservations, no ticketing, no Train Status, no Quik-Trak machines. Nothing.

A major power supply outage at the vendor to whom Amtrak has outsourced its data center is blamed. Amtrak anticipated that the system would resume operation at approx. 1200N EDT on Sunday August 26...45 minutes ago.

Amtrak outsourced ARROW's data operations to a vendor a few years ago. The hardware is good, contemporary stuff but apparently a system power outage occurred and there was no backup for that failure.

So far, only one newssource, COX newspapers, has picked on this story and apparently has circulated it to only one of its newspapers, the Austin American Statesman;

http://www.statesman.com/blogs/content/shared-blogs/washington/washington/entries/2007/08/25/amtrak_suffers.html

Here is a bio on the Staff Reporter filing the story;

http://www.coxwashington.com/reporters/content/reporters/guroian_r.html

I thought Amtrak outsourced ARROW to IBM a few years ago?

I am still trying to book a Surfliner trip via Amtrak.com since yesterday.

EDIT: ARROW is back!! Just spoke with someone at the Riverside call center. All systems go.

#11, which was about 6 hours late through Gilroy, CA yesterday, just went through 100% on time today. Maybe Amtrak should have computer problems more often!

Amtrak Amtrak Reservations back online
Mainframe reportedly hit by lightning

Austin American-Statesman
Sunday, August 26, 2007, 01:27 PM

By Rafi Guroian

> As a followup to our posting yesterday regarding Amtrak’s massive
> ticketing system failure, the pasenger railroad’s reservation and
> ticketing systems have now come back online. We have received an
> unconfirmed report that the cause was a bolt of lightning that struck an
> IBM mainframe facility in northern Virginia during some severe weather
> yesterday. We’ll post more information as it becomes available.


http://www.statesman.com/blogs/content/shared-blogs/washington/washington/entries/2007/08/26/amtrak_reservat.html

westcoaster Wrote:
-------------------------------------------------------
> #11, which was about 6 hours late through Gilroy,
> CA yesterday, just went through 100% on time
> today. Maybe Amtrak should have computer problems
> more often!

I was down in Pennsylvania this morning to photograph #48, and it was on time also!

Nobody can complain that they're late because nobody knew where the trains were/status of em!!!

An AP report late Sunday afternoon says it was due to a "recently installed software change." Here's a link to the report:
http://www.forbes.com/feeds/ap/2007/08/26/ap4054700.html

Which am I to consider was the cause of the incident? A recently installed software change or a lightning strike? Was it IBM that did the software change?

jb

jdb Wrote:
-------------------------------------------------------
> Which am I to consider was the cause of the
> incident? A recently installed software change or
> a lightning strike? Was it IBM that did the
> software change?
>
> jb


Amtrak's official statement to employees was it was a shorted Power Circuit, located at a major point which prevented even backup systems from starting up. Replacement part was sent overnight on the Night Owl (Overnight Regional) and installed by technicians. Major systems started to come up around 10:15a West Coast Time, but not all internal systems were up until early in the evening. I'm sure each computer was handled separately; started, booted, tested, checked, and approved before being linked to the network.

I'm actually very impressed with how well it came together in the face of having no communication, automated tickets, a busy weekend... Agents kept each other up-to-date by phone, tickets were hand-written by clerks and crews... And most importantly the trains still ran as they do every day.

GenePoon Wrote:
-------------------------------------------------------
> No reservations, no ticketing, no Train Status, no
> Quik-Trak machines. Nothing.


Not as obvious to the public, the outage also affected the computerized time slip system for T & E crews AND the crew management systems. Crew callers in Delaware were working over 24 hours with just pencils and a print-out of the extra board from the day before. A testament to the herculean effort put in by the crew callers is that no trains went unstaffed.

Yesterday, August 26th, I stopped by the Davis Station early to purchase tickets to go to Berkeley. The ticket clerk had to handwrite tickets, and she was told that the reservation system would not be up and running until at least a week. Good grief! All the other trains, #11, #6 and a Capitol were all running on time, in spite of the computer glitch. When I returned with my friends to board Train #733, the system was up and running.

On the return trip, Train #744 was delayed some for some time due to a grass fire near Martinez. Not a good day for the Capitol Corridor!

Some years back SABRE, American Airline's computer system (now out-sourced) experienced a total system failure and the word went out -> EMERGENCY TO ALL AMERICAN AIRLINE FLIGHTS AND CREWS.


"LAND. RIGHT #$%^&^%$# NOW. LAND!! Declare an emergency and get the bird on the ground RIGHT NOW. We don't know who you are, WHERE you are, how much gas is on board, who the crew is, what the passenger data is - nothing. If you are over water, divert to the nearest "friendly" airfield but do it NOW!!!"

Thank heaven Amtrak didn't need to do that.

ChrisJ Wrote:
-------------------------------------------------------
> An AP report late Sunday afternoon says it was due
> to a "recently installed software change." Here's
> a link to the report:
> http://www.forbes.com/feeds/ap/2007/08/26/ap405470
> 0.html


That report was false.

The outage has been confirmed as being due to a power supply failure in the building where the data mainframe is located. A circuit breaker tripped and primary backup systems could not be used due to the location of the failure. Plan B backup...fixing the problem...was delayed for many hours because the vendor did not have the parts needed; they had to be shipped from about 300 miles away and in fact were transported via Amtrak train to Washington.

It had to be global warming if any idiot listens to one named Gore