AVSIM down--they needed Jesse

TangoWhiskey

TangoWhiskey

Touchdown! Greaser!
Joined
Feb 23, 2005
Messages
14,210
Location
Midlothian, TX
Display Name
Display name:
3Green
http://linux.myalbemarle.org/forums/viewtopic.php?f=32&t=10

Sounds like they didn't have any backups!? They needed a REAL sys admin like Jesse. There's talk on Aero-News.net that it might have been an "inside job" by a disgruntled person... sad!

They were saying "we don't know if we CAN recover..." :nonod:

Folks,

Today, AVSIM was destroyed by a hacker. Both of our primary servers were hacked and the partition tables destroyed, and that will be very difficult to overcome. I won’t speculate on who did it, but we will have a forensic done on our disks and scrutinize the logs to validate our suspicions.

I would appreciate if you could post the following as a news item or announcement on your websites or forums. Nels and Francois, I know we have been competitors, but I would very much appreciate you posting this on your web sites, if you could.

PRESS RELEASE: AVSIM Hacked!!

Tom Allensworth, CEO and Publisher of AVSIM, today issued the following announcement; “We regret to inform the flight simulation community that on Tuesday, May 12, AVSIM was hacked and effectively destroyed. The method of the hack makes recovery difficult, if not impossible, to recover from. Both servers, that is the library / email and web site / forum servers were attacked. AVSIM is totally offline at this time and we expect to be so for some time to come. We are not able to predict when we will be back online, if we can come back at all. We will post more news as we are able to in the coming days and weeks. END

Best Regards,
Tom Allensworth
Click to expand...
 
Last edited:
Supposedly they were backing up their servers; just from one server to the other. Woops.
 
Can you say....oops? That does indeed suck. Pull the drives and send them to some folks that are good at recovery.
 
sucks for AVSIM ... gee, back to my post on backups ... disk to disk backups are great, but no substitute for off-site storage of your backups!!!
 
gkainz said:
sucks for AVSIM ... gee, back to my post on backups ... disk to disk backups are great, but no substitute for off-site storage of your backups!!!
Click to expand...
There's always one who says "I told ya so!"
Backups are fine until you need them. If you've never tested out a restore, you are doomed. When you need them is NOT the time to find out they're no good OR that you can't use the tool OR that you can't read the tape OR that the person with the key is half way around the world on an extended sabbatical. If you take them, make sure you can restore them.

And practice practice practice.
 
silver-eagle said:
There's always one who says "I told ya so!"
Backups are fine until you need them. If you've never tested out a restore, you are doomed. When you need them is NOT the time to find out they're no good OR that you can't use the tool OR that you can't read the tape OR that the person with the key is half way around the world on an extended sabbatical. If you take them, make sure you can restore them.

And practice practice practice.
Click to expand...

Yup! I work in the Storage and Backup team (infrastructure-side) in a major data center. We backup, encrypt, and write to tape 220Tb of data per night. I always tell my team that we're incorrectly named the "backup" team--we should be called the "restore" team. Nobody cares if we backup data. All they care is if we can restore it.
 
TangoWhiskey said:
Yup! I work in the Storage and Backup team (infrastructure-side) in a major data center. We backup, encrypt, and write to tape 220Tb of data per night. I always tell my team that we're incorrectly named the "backup" team--we should be called the "restore" team. Nobody cares if we backup data. All they care is if we can restore it.
Click to expand...

When I went on job interviews 20 years ago..
1) The Arthur Andriod that Steve Fossett hired who hired me...They kept talking about some job where they couldn't read the tapes. Now that I think about it, that might be why they were available for Steve to hire them.

2) Fannie Mae: "How do you know your backups are good?"
-> My script rewinds and reads the whole tape back every night.
(I got a "doesn't suck" on my backup scripts from my mentor on that job after they tossed me out with the cutbacks.)
 
That really does suck :(

We found out a few weeks ago that data backups mean nothing if you have a hardware failure. That was interesting. I don't think I'd heard that many curse words come out of our sysadmin's mouth to Dell when the guy was forcing him to go through the scripted diagnosis in order to get the replacement part under warranty.
 
wbarnhill said:
That really does suck :(

We found out a few weeks ago that data backups mean nothing if you have a hardware failure. That was interesting. I don't think I'd heard that many curse words come out of our sysadmin's mouth to Dell when the guy was forcing him to go through the scripted diagnosis in order to get the replacement part under warranty.
Click to expand...

Redundancy, clusters, failover, VMs...

Intel hardware is so relatively cheap buy two. ..which of course rewards Dell for making it....
 
wbarnhill said:
We found out a few weeks ago that data backups mean nothing if you have a hardware failure.
Click to expand...

Backups can mean nothing due to hardware and software obsolescence, too. Marketing data will show an LTO tape has a 30 year life span. Yeah, in perfect conditions. Move it back and forth between offsite storage and your data center, in a humid truck, a few dozen times as it goes scratch (data reaches retention age and tape can be overwritten). Then write a long-term (7-10 year retention) backup to it and stick it on the shelf for several years. Legal calls and needs a restore. Then try to read that 12 or 13 year old tape. Do you still have the software you wrote to the tape with? Will that software still run on the hardware and, more importantly, operating system version you are now running? Say you backed it up with REEL, were running it on AIX 4.x; you're up to AIX 5.3 now and the original software hasn't been updated by the manufacturer, or you've moved onto a new product (Legato, Veritas, CommVault, take your pick) and no longer maintain a contract with vendor x from way back when... can you still restore the tape? Does anybody in your organization even remember how?

It gets fun! ;-)
 
TangoWhiskey said:
Yup! I work in the Storage and Backup team (infrastructure-side) in a major data center. We backup, encrypt, and write to tape 220Tb of data per night. I always tell my team that we're incorrectly named the "backup" team--we should be called the "restore" team. Nobody cares if we backup data. All they care is if we can restore it.
Click to expand...

A-Men!

I can't believe there wasn't an offline copy somewhere. That just seems so... basic. It might be a month old, but so what?

At work, we write to tapes that stay in the library, and transactions are written using Oracle DataGuard to a disaster recovery site, where they are also backed up to tape. We've gone through two tape tech refreshes so far but fortunately NetBackup does a good job of copying data from the old library to the new one. Next refresh will probably to use disk instead of tape for most backup jobs, with a small library in case we need to write stuff that will be physically relocated, rather than shipping the offsite stuff across the net.

At home, it's all disk based, but I take Acronis images every quarter or after any major system change, and those are kept on an offline eSata disk. Normal weekly/nightlys are written to NAS. Worst case is I fall back 90 days.
 
Last edited:
gkainz said:
sucks for AVSIM ... gee, back to my post on backups ... disk to disk backups are great, but no substitute for off-site storage of your backups!!!
Click to expand...
There is nothing that says you cannot pull hard-disks off-site. Also, an inside man, with the right knowledge can destroy a company in a very short amount of time. This was likely the case.
 
wbarnhill said:
We found out a few weeks ago that data backups mean nothing if you have a hardware failure. That was interesting. I don't think I'd heard that many curse words come out of our sysadmin's mouth to Dell when the guy was forcing him to go through the scripted diagnosis in order to get the replacement part under warranty.
Click to expand...

This is why I've always operated under the philosophy that ordering servers with warranties mean absolutely nothing. Some folks like warranties and get all warm and fuzzy, personally, on servers..they're pretty much a waste.

The best warranty you can really get is a 4-hour warranty from IBM. They will have someone on site with the part within 4 hours. Well a critical system being down for 4 hours is absolutely unacceptable.

You're way better off just building your own **** and stocking your own spare parts. Build out all your systems to be redundant and clustered where possible and fix **** yourself.

Warranties can make sense on some things, like Cisco gear, but even then you could stock replacements.
 
jesse said:
There is nothing that says you cannot pull hard-disks off-site. Also, an inside man, with the right knowledge can destroy a company in a very short amount of time. This was likely the case.
Click to expand...

The guy gets to laugh.


Then the guy gets to go to jail.
 
You must log in or register to reply here.
Back
Top