Passenger Trains > Amtrak's Website

The website has been down for maintenance since early Saturday evening and is still down this morning. Anyone know when it might be back up?

about 7:00

It is actually broke down and it is a mess at 30th St. Station, Philly!

Arrow System had a Fatal Shutdown affecting the Entire Amtrak System. Crew Management Ticketing ETC.. This shutdown happened on Sat and the system should be in service sometime today...

Apparently, Amtrak's ticketing system was down all or most of Saturday. Agents were having to hand-write tickets, and Quick-Trak machines were out of service. A lot of tickets were being sold by conductors onboard (passengers were given special slips by station agents allowing the onboard cash penalty to be waived). It made the national network news (at least the ABC network radio news, which is where I heard about it on Saturday afternoon).

I admit that I am not too computer savvy in my old age but wouldn't there be a back up system to Arrow? Even I backup any writing that I do on my computer!

stone23 Wrote:
-------------------------------------------------------
> I admit that I am not too computer savvy in my old
> age but wouldn't there be a back up system to
> Arrow? Even I backup any writing that I do on my
> computer!


Considering the age of Arrow I would expect that they have to find replacement equipment on Ebay or a computer museum.

Todd Clark
Canyon Country, CA
Trainorders.com

I was told by an Amtrak agent in Fullerton yesterday that there was a fire in Washington causing the computer failure.

Keith

KD6PYD Wrote:
-------------------------------------------------------
> I was told by an Amtrak agent in Fullerton
> yesterday that there was a fire in Washington
> causing the computer failure.
>
> Keith

A fire in DC still does not explain why there is not a backup for this obviously necessary computer system!

stone23 Wrote:
-------------------------------------------------------
> I admit that I am not too computer savvy in my old
> age but wouldn't there be a back up system to
> Arrow? Even I backup any writing that I do on my
> computer!


ARROW itself is backed up, but if the servers were damaged then they can't transmit anything, even a backed up copy of the system.

Todd, while ARROW itself is old the hardware is constantly being updated, just last year new computers were installed everywhere and the system was down twice for an extended period to add new Hardware or replace old stuff. ARROW is old because nothing has come along with the capability of replacing what thirty years of programming and tweaking has enabled it to do. ARROW will be here for a long time.

It's causing havoc everywhere, as was noted above, though; not just ARROW is affected... this could have far more serious consequences if it goes on too long.

http://www.taborcommunications.com/dsstar/02/0423/104191.html

My guess is that the hardware is NOT the problem; it too was likely upgraded throughout the years; and since Amtrak outsources I.T. management (including that of the ARROW system) it probably runs on a current (or fairly recent) model S/390 (Z-Series Server).

stone23 Wrote:
-------------------------------------------------------
> KD6PYD Wrote:
> --------------------------------------------------
> -----
> > I was told by an Amtrak agent in Fullerton
> > yesterday that there was a fire in Washington
> > causing the computer failure.
> >
> > Keith
>
> A fire in DC still does not explain why there is
> not a backup for this obviously necessary computer
> system!


They DO have a backup. It's called hand-written tickets. Best backup system in existance. Absolutey, 100% guaranteed to work every time (unless they run out of pens and paper).

I've been an IT professional for 35 years and am currently working as a IT System Manager for an extensive, nationwide government hospital system (who's name I won't mention but you can probably guess) and can tell you that full 100%, "never-goes-down" redundancy is a tad bit more expensive and complicated that simply clicking a "Backup" button on your home pc.

First off, a backup system needs to be located some distance (minimum of 50 miles) from the primary system to guard against failures caused by natural or man-made disasters. If the backup system is expected to function adequately, it will have to pretty much be a duplicate of the primary site, so double your expenses both for acquisition and maintenance. You can save SOME money by reducing cpu power, etc., but the manpower and maintenance to staff the backup site will be the same so you won't save much.

Next, if you expect the backup system to be able to take over for the primary within a reasonable amount of time, it's databases will have to be updated real-time from the primary system. This requires a lot of network bandwidth and that doesn't come cheap.

Next, if the backup system is to be fully redundant to the primary, every network connection to the "outside world" (stations, amtrak.com, etc.) will have to be duplicated to mirror the primary site. Again, double your network expenses.

On top of all of this, there is overhead the equipment and maintenance for the redundancy itself. Figure that true redundancy will cost you about 2.5 times cost of the primary-only design.

Folks, this is Amtrak. We all know that they don't have the financial resources to do this. Even the hospital system I work at doesn't have that kind of redundancy. We have contingency plans in place, most of which are simply going to paper records until the outage is resolved. We certainly have redundancy in our network connections and computer systems but if there were a fire in the computer room, there would still be down time until a temporary replacement system were to arrive (I'm told there is one in a semi-truck somewhere that can respond within 24 hours). Until then, we implement our contingencies.

Amtrak certainly backs up their databases regularly however if database corruption has occurred or if their disk arrays experienced catastrophic failure, it can take a considerable amount of time to restore the data. Recently, at the hospital, we had a vendor field service engineer come in to replace a failing disk controller. This controller was a "hot-swap" item that can be replaced without shutting down the system. The FSE, however, entered the wrong command at the wrong time and wiped the entire array. 24 hours later, we finally had replaced the contoller (and the FSE), rebuilt the array and restored the data.

Also, keep in mind that the fire might not have been at Amtrak's facility but possibly at a network carrier (Sprint, etc.). Amtrak might have redundant network lines but as we found out recently, network carriers many times sub-contract network connections to other carriers and your "redundant" network might end up going through the same fiber cable as your primary, but they never tell you this and you won't find out until a cut happens and you lose both paths. In the incident we had, our primary carrier was S****, our backup connection went to someone else who sub-contracted it to another carrier who sub-contracted it back to S*****. We now write in our contracts that primary and backup network connections must be routed through physically different fiber cables.

Anyway, even eBay experiences outages so cut them some slack. I feel the pain that their IT people are going through right now.

Ken

Neither primes nor backups work very well if the power goes out. Failure can happen anywhere along the chain.

webmaster Wrote:
-------------------------------------------------------
> stone23 Wrote:
> --------------------------------------------------
> -----
> > I admit that I am not too computer savvy in my
> old
> > age but wouldn't there be a back up system to
> > Arrow? Even I backup any writing that I do on
> my
> > computer!
>
>
> Considering the age of Arrow I would expect that
> they have to find replacement equipment on Ebay or
> a computer museum.

==================================================

Amtrak outsourced ARROW's data operations to a vendor a few years ago. The hardware is good, contemporary stuff but apparently a system power outage occurred and there was no backup for that failure.

So far, only one newssource, COX newspapers, has picked on this story and apparently has circulated it to only one of its newspapers, the Austin American Statesman;

http://www.statesman.com/blogs/content/shared-blogs/washington/washington/entries/2007/08/25/amtrak_suffers.html

Here is a bio on the Staff Reporter filing the story;

http://www.coxwashington.com/reporters/content/reporters/guroian_r.html

Material parallel posted at the related topic originated by Mr. Poon.

krapplem Wrote:
-------------------------------------------------------

> They DO have a backup. It's called hand-written
> tickets. Best backup system in existance.
> Absolutey, 100% guaranteed to work every time
> (unless they run out of pens and paper).

Agree. And I'm seeing it done.

But...........

This will work for cash only. They can't read credit cards with the system down.

Maybe they could hand write the credit card number also if they feel the card is good.

Other problems show up hand writing tickets. More tickets could be written than there is space available. However, space available may not be a problem because they can't hand write all the tickets before the train leaves.

jb

jdb, I was thinking the same thing...

If Amtrak allows passengers to use credit cards while hand writing the tickets, it could create a serious risk of fraud.

I am curious to see what is going on at the larger Amtrak stations today. It must be quite "interesting".


EDIT: ARROW is back!! Just spoke with someone at the Riverside call center. All systems go.

They have fixed the power problem and went back online at about 100pm PDT, 400pm EDT.

jdb Wrote:
-------------------------------------------------------
> krapplem Wrote:
> --------------------------------------------------
> -----
>
> > They DO have a backup. It's called
> hand-written
> > tickets. Best backup system in existance.
> > Absolutey, 100% guaranteed to work every time
> > (unless they run out of pens and paper).
>
> Agree. And I'm seeing it done.
>
> But...........
>
> This will work for cash only. They can't read
> credit cards with the system down.
>
> Maybe they could hand write the credit card number
> also if they feel the card is good.
>
> Other problems show up hand writing tickets. More
> tickets could be written than there is space
> available. However, space available may not be a
> problem because they can't hand write all the
> tickets before the train leaves.
>
> jb


That's why I say "Paper Tickets Forever!" It's hard for anyone to argue that you don't belong on a train when you have a pre-printed, official ticket in your hand. Never did trust e-tickets.

I guess everyone has settled on "power outage" as the problem. I was hoping that with the good press Amtrak received in the past few days that so many people were booking trips, the system crashed. :>)

Ken

GenePoon Wrote:
-------------------------------------------------------
> They have fixed the power problem and went back
> online at about 100pm PDT, 400pm EDT.


Must be something wrong with my watch Gene, It is only 321pm EDT in New Jersey!