hackers

[SixPapaCharlie]

So I have a handful of websites for various things.
2 of them have upload features.

Seems someone uploaded something to one of my sites on the 18th.
I just caught it yesterday evening.

There were 4 new folders on my site all containing jpeg images.
All were identified as invalid jpeg. I tried opening them with scripting tools but it was all encrypted.

I have no idea what they were doing but they crippled the site and I lost all permissions to do anything through FTP. I can't read, write, change permissions.

I pulled the logs and there was a ton of data going through on the 18th
scripts being uploaded. Several references to robots and spiders but this does not seem automated. There were multiple uploaded files and folders.

I have removed the site and domain, and am still unable to do anything via FTP even to files / folders I created months ago.

The site itself is not important but I use scripts that are not a part of the actual website as a 3rd party for handling some important transactions for my business.

That seems intact but I am recreating that functionality elsewhere.

In 15 years of building websites, I have never been hacked and I don't think i have gotten a computer virus in that amount of time either.

I will post the logs if anyone is interested. I know some of you are off the charts smart at IT stuff.

If you allow a visitor to upload a file, how can you prevent them from doing harm and what is it they are able to do? I restrict file type and I change the name of the file in the upload process and I generate the folder where it will be uploaded.

What could someone have done with a fake jpeg file?

 

[EdFred]

You better brush up on your nun-chuck skills and bo staff stills.

 

[schmookeeg]

If you lost permissions to your files, the issue wasn't the upload -- they used some other facility or exploit to execute things on your server. Just letting any schmoe upload files is harmless... it's when they find a way to execute things that hacks occur.

Imagine going through this with a voip server (something I'm dabbling in right now, for fun), and each time you fail to harden a part of the site, you end up with $30 in calls to Liberia going through your system by the time you wake up and fix it.

I'm down $60 so far in Liberia tax... but of course, I've not paid Charter or the telco some $360 in the same timeframe, so I'm still money ahead.

 

[wsuffa]

I am aware of a site that the ISP shut down because a hacker broke in and used it for DDOS attacks as part of a botnet. Got in through a zero-day exploit.

 

[RJM62]

Anonymous uploads, by default, should be jailed; and FTP uploads by users that you create should be jailed by design. It's possible that neither were the cause of the compromise.

Post-mortems aside, who controls the actual server? Root should be able to change the password to something strong so you can get your access back.

Rich

 

[SixPapaCharlie]

It's a hosted site.

 

[RJM62]

It's a hosted site.

So's every other site in the world. Or do you mean it's self-hosted?

Rich

 

[SixPapaCharlie]

Sorry. I am NOT hosting it on my server it is a shared server through a service

 

[DFH65]

There have been several vulnerabilities announced recently in Linux/Unix Bash. Also check out the Poodle vulnerability using SSL 3.0.

Question is rarely if, it is when. You service may be blissfully unaware.

 

[RJM62]

Sorry. I am NOT hosting it on my server it is a shared server through a service

Okay, then someone at your hosting company can change the password(s) for the account to restore your access. They really need to be informed of the compromise in any case.

Rich

 

[brian]]

This is the reason we phase out servers here. Gets easier when you use VM ware and you are phasing out a server, moving just data - no executables - to the new server. Ok, we move executables, but not scripts - long story.

Anyway, I assume a server is hacked pretty much once it goes on the internet. It is my process to prove otherwise..

There is a reason I have an airplane with as few computers in it as possible...

 

[wsuffa]

If you want some idea of the persistent threat, a "mail only" Linux server gets at least 250 attempts a day, not including the ones stopped by tight settings on fail2ban (basically, every time F2B kicks off, the hacker tries again). Typically F2B is keeping an address banned close to 24 hours. And that's with root access disabled from remote addresses.

 

[brian]]

At least with a unix box you can tail - f the messages log file and watch the root attacks. Thousands per day.. Wish these a holes had a real job...

 

[Jim Logajan]

There have been several vulnerabilities announced recently in Linux/Unix Bash.

The shellshock vulnerability is one such recent dangerous exploit:
http://en.wikipedia.org/wiki/Shellshock_(software_bug)

 

[wsuffa]

At least with a unix box you can tail - f the messages log file and watch the root attacks. Thousands per day.. Wish these a holes had a real job...

They do have a job. It happens to be criminal.

 

[denverpilot]

If you want some idea of the persistent threat, a "mail only" Linux server gets at least 250 attempts a day, not including the ones stopped by tight settings on fail2ban (basically, every time F2B kicks off, the hacker tries again). Typically F2B is keeping an address banned close to 24 hours. And that's with root access disabled from remote addresses.

Only 250? Lucky bastard. Is ssh on the standard port number? There's really no point in even using port 22 anymore. There's also no point in even allowing password auth anymore. Something you know (port number), something you have (key), something you are (biometrics to even get on the internal network that can reach the server). Can drop the third, but the first two are basically mandatory now. Passwords for server auth should be dead dead dead. If you need single sign on, you need hard keys to get there, and at least encrypted LDAP auth, or Kerberos tickets, or SSH keys.

The only box that answers SSH to the outside world in our environment is an SFTP drop for customers tied to an RDBMS for user/pass, and I'd prefer it used keys. But most of the folks dropping stuff there, aren't bright enough to generate SSH keys nor want the perceived hassle. All users are chrooted into their own chroot jail. I'd use SELinux on that box but it doesn't play nicely with chrooted SSH jails at all.

For standalone or "fun" machines of my own, I ideally want a virtual host that has key based access to get me to a console. Only a few do that. If I can avoid putting SSH on a public IP altogether, I will.

At least with a unix box you can tail - f the messages log file and watch the root attacks. Thousands per day.. Wish these a holes had a real job...

There's event and application logs on Windows servers too, not that I'm a Windows guy, but the assertion that logs don't exist to watch bad Internet behavior on anything except Linux, is not true.

Plus, a really good root script will subtly modify logs to hide in plain sight, once it's in. A bad one will just delete all the logs. The normal method of protecting against this is centralized logging. /var/log/messages is generally untrustworthy on a compromised machine, and if the assumption is all machines are compromised until proven innocent... Well, you see where that goes.

(By the way. It has security challenges of its own, but properly implemented central logging and a tool like Splunk or similar, is really the modern state of the art. Makes hunting for patterns, easy... And really pays back fast for the minimal amount of human time necessary to deploy it right.)

There were some stats from honeypot operators back in the early 2000s about how long it took for the average unpatched machine of various OS flavors with no firewall protection to be broken into or broken. It was measured in single digit minutes for some OSs back then. It hasn't gotten better. You put most OSs as delivered out of the box on a public IP, you're often looking at seconds until it's owned, now.

Weekly patching isn't fast enough for security related patches marked Critical from OS vendors, those are immediate patches, knowing that the second the OS manufacturer releases the patch, thousands of people are immediately reverse engineering why the patch was released and writing code to exploit it.

Various tools currently available for free have hundreds of thousands of pre-written exploits already available for download for them, and will even automatically set up persistent reverse connections to a machine if your choosing once they automatically break into something, so even rebooting it won't keep the bad guys out. Most will also set up hidden Layer 2 network VPN type access, so anything on the same LAN/VLAN as the compromised host is now fully accessible to be attacked from "within", if you're crazy enough to allow full access between bastion hosts.

Automatic exploit software really hit its stride around 2003 and has done nothing but accelerate since. This is stuff young teens learn easily with a little coaching, and even a "hobbiest" level hacker can find thousands of exploitable machines to mess with.

Even the stalwarts of Unix security, the folks at OpenBSD, are regularly getting hit now. Theo de Raat was TICKED at the OpenSSL team for not writing in hooks for Theo's team to audit their memory handling FOR THEM, three years before the recent SSL exploits were found. If they had, it would have never been an exploitable problem. They'd have found it.

Most Linux distributions have at least one Major or Critical security patch per week now, even on the slowest weeks, and Microsoft's "Patch Tuesday" isn't nearly often enough, as shown by their number of patches every month.

The only real fix is to stop plugging things into a worldwide untrustworthy computer network that haven't been specifically written with security in mind. And security coding isn't paid for unless there's a monetary reason to do so that's higher than it costs to pay for security coders.

The really fun part...?

Most of the attacks on my production environment are coming from virtual machines on the Amazon and Microsoft clouds now. That number flipped this year, for our stuff. Used to be APNIC addresses, were on top.

 

[eman1200]

At least with a unix box you can tail - f the messages log file and watch the root attacks. Thousands per day.. Wish these a holes had a real job...

FYI, you can tail logs on windows, I do it all day.

 

[RJM62]

Most of the attacks on my production environment are coming from virtual machines on the Amazon and Microsoft clouds now. That number flipped this year, for our stuff. Used to be APNIC addresses, were on top.

I've been getting quite a few that use Google Docs lately, too.

Rich

 

[B17Rex]

Agree with the above. If you feel the server was compromised, /var/log/messages will most likely be of no use. Any hacker (hell even automated tools used by script kiddies) will clean up their tracks in those log files. Forwarding those logs in real time to a central syslog server is the best way to preserve them.

Re-reading your initial post, you didn't specify the platform in use. Is it a Windows system, Linux system, Unix system?

Unfortunately with a lot of the exploits these days, the best way to recover from a compromised system is not to clean it with AV. Blow it out and start over. It is just not safe to assume your server has been properly cleaned. The recent vulnerabilities of Heartbleed, Shellshock, POODLE, there is some scary **** out there these days.

My best advice would be to use a backup of the data on that server (from before the issues started), fire up a new hosted server, restore the data and go from there.

Cheers

-Brian