Using a Bastion Backup Server to Protect Your Backups

The term "bastion server" is usually used to describe an intermediate network security device.  The concept is the secured network (usually an internal network) is not directly connected to the unsecured network (think Internet).  The bastion server sits in the middle and marshalls the desired traffic between the two networks.   Sounds kindof like a firewall, but it is not the same thing.  If someone penetrates your perimeter defenses, they would be in the bastion server or bastion network and not have direct access to the secured network.  

How does this relate to backups?  A comment on a previous post, "Pull Rather than Push Your Offsite Backups", suggested that an intermediate server can be used to add additional protection to your backups.  The intermediate server is something I have used for many years to ensure that backups are protected from ransomware.   In this post, I am going to elaborate on the value of the Bastion backup server as an additional security measure to thwart hackers and sophisticated ransomware which may target your backups.

Keep in mind that one of the primary success factors for ransomware is the ability to destroy backups.  Otherwise, the victim can easily recover without paying the ransom.  Hackers and ransomware developers are well aware.  Their first objective is to get the ransomware onto machines with high-value data or systems; their next objective is to deny the victims the opportunity to restore from backups.  With that said, it is imperative that your backups be stored so that they cannot be destroyed or modified by anything that may occur on the primary system.  There are many ways to accomplish this, but fundamentally, the more disconnected the backups are from the primary system, the better the protection from ransomware or other threats that may occur on the primary system.  

Simply having a backup in the cloud doesn't necessarily mean it has a high degree of disconnectivity from the primary.  For example, consider a system in New York that is backing up to a remote server in Texas.  Sounds pretty good from a geographic perspective, but if the New York system can read and write backups to the Texas system, then ransomware on the New York system probably has the means to destroy your backups.   In this case, I would say the Texas system is not disconnected from the New York system in a meaningful way.

This is where a bastion server adds tremendous value.  Let's say the production system has no connectivity to the backup location at any time.  Then how will the backup files ever get to the backup location?  One very good answer is to set up a bastion backup server that has read-only access to the production system.  The bastion backup server then pulls the data over its read-only connection to local storage.  Next, your offsite backup system pulls the backups from the bastion backup server over a read-only connection.

Not only does this configuration prevent ransomware on your production server from affecting your backups, but it also prevents the reverse.  Sometimes ransomware will infect the backup system first, and then use the connection to the production system to install itself there.  This is why those read-only connections are very important.   In this case, any event that occurs on the production server, the bastion backup server, or the offsite backup server, is unlikely to affect any of the other systems.

Secure your bastion backup server.  The term "bastion server" by definition usually means a "secure" server.   So if you are implementing a bastion backup server, place a priority on the security of that server.  Use basic security principles, like run only the necessary services to reduce the attack footprint, allow minimal network connectivity, inbound and outbound.  Especially limit or block internet access to and from the bastion backup server.  It is also advised to install minimal software, no email programs, no browsers, ...   Most of these security approaches can be applied to your offsite backup server as well, but the bastion backup server, can be almost completely locked down.

Should you leave a copy of your backup files on the bastion backup server?  The answer depends on the nature of the data and your application and needs.  There is a tradeoff to be considered.  Ransomware is not the only concern.  You've seen many high-profile cases where data breaches of sensitive information have caused expensive and embarrassing consequences for large companies.  An argument can be made that having an extra copy of those sensitive files on your bastion backup server is one more place where a breach could occur and expose sensitive data.  In fact, many of the big breaches involved hackers or malware accessing backup files.   In my personal opinion, the bastion server, if properly secured is the last place where hackers or malware are going to get access to your data.  Your production system is probably much more vulnerable because it must be accessible using certain services, otherwise the data is useless to everyone including you, your company, and your customers.  

One last thought.  It is extremely important to monitor your backups to make sure they are working.  The Bastion backup server is an interesting place to setup monitoring.  But be careful not to create additional exposure for the Bastion backup server by installing untrusted software and giving it access to your backups and your network.  You may want to consider only monitoring the final destination for your backups.   

Pull Rather than Push Your Offsite Backups

 Keeping backup copies of your data in a location separate from the primary data storage is highly advised.  I prefer to store one backup copy locally and a second one in a different location.   A few decades ago, most data backups were stored on magnetic tapes.  At the time, tapes were relatively inexpensive.  But the problem with tapes is they must be physically moved from one location to another.  

I once worked in a large data processing center; numerous backup tapes were created every night after processing.  The tapes were immediately stored in a vault.  The most current backup tapes were pulled, placed in portable carrying cases, and picked up by an offsite vaulting company every morning.  We agreed with the offsite company to keep a fixed number of backups offsite and return the older tapes to the data center.   This very effective process protects the data from multiple losses, including fires and events that might destroy the primary location. 

The problem with rotating backup tapes, or any kind of media offsite, by physically moving the media is the element of human intervention that can break down.  For example, suppose the driver for the offsite vaulting service had an accident and could not deliver the media.  The unreliability factor for small businesses that don't use a professional service is through the roof.   People forget, people get busy, and people procrastinate;... I do not know if those offsite vaulting services that pick up and move media still exist because there are much better ways to get data to a different location now.

The modern way to move data offsite is to copy the data over a network to a secondary location.  This protects the data from fires and other events that may physically destroy the data at the primary location.   However, a few decades forward, the biggest threats are mostly cybersecurity-related.  

In particular, ransomware is a much bigger threat than any kind of physical loss of media.  Ransomware has ramped up its sophistication to where it not only encrypts your data but will also encrypt or destroy your backups.  The motivation is to deny you the opportunity to restore your data and make it your only option to pay the ransom to get it back.

Moving your data electronically from location to location is common, and ransomware developers know it.  Suppose your secondary backup location is accessible by a network from your primary location. In that case, the ransomware may use the same connection to delete, encrypt, or destroy the data at your secondary site.  

Solution:  Pull, Don't Push!

Let's say location A is where your primary data is located.  And you run a backup every day, then copy the backup from location A to location B.  Configure your systems so that neither location has any write or delete access to the other.  This way, any ransomware infection in one location cannot destroy data at the other location.  When data needs to be moved from location A to location B, use a process on location B with read-only access to read and PULL the data from A to B.   Under no circumstances should you allow one server to write, or PUSH, a backup to the other.   

If you follow this simple principle, you can thwart many ways ransomware will destroy your backup data.  Hopefully, your other security measures prevent the ransomware attack before it happens.  If you are hit by a ransomware attack, your primary recovery will be restoring the data and denying the bad actors their reward.

Can Backup's Protect Against Ransomware?

Over the last couple of decades, two technologies have enabled bad actors to obtain vast amounts of ill-gotten money.  Make no mistake, this profit motivation has spurred an entire industry around hacking and malware.  It's not just for fun or notoriety; it's a highly profitable venture.  Ransomware enables bad actors to cause people and organizations to pay to recover their own data. The cryptocurrency has provided a way for them to get paid in a largely untraceable way.  

The rise of ransomware attacks has emerged as one of the most significant cybersecurity threats in recent years. These malicious software programs are designed to infiltrate computer systems, encrypt valuable data, and demand a ransom from victims to regain access to their files.  This has been somewhat of a wake-up call for organizations, and they have increasingly turned to backup solutions as a precaution against data loss.  Data backups are an effective defense against ransomware.  So the ransomware has evolved to target these very backups, leaving victims with limited options to recover their data without succumbing to the attackers' demands.

Understanding Ransomware

Ransomware is malware that gains unauthorized access to a system and encrypts the victim's data using advanced encryption algorithms. The attackers then demand a ransom, usually in cryptocurrency, to provide the decryption key necessary to unlock the files. Once the files are encrypted, the data becomes inaccessible and unusable until the decryption process is completed successfully.

Ransomware Targets Backup Data

Many organizations have invested in comprehensive backup solutions in response to the growing awareness of ransomware threats. These backups are crucial for data recovery in case of data loss due to hardware failures, human errors, or other unexpected incidents. However, attackers have recognized the significance of these backup files and adapted their strategies to target them directly.

Direct Attacks on Backup Repositories

Some sophisticated ransomware variants are programmed to identify and encrypt backup repositories directly. By infiltrating the backup systems, attackers can render the organization's data recovery strategy useless, significantly increasing the pressure on victims to pay the ransom.

Compromising Backup Credentials

Attackers may employ various techniques, such as phishing or brute-force attacks, to access backup credentials. Once obtained, they can manipulate the backup software to encrypt the stored data like the primary system.

Deleting or Corrupting Backup Data

In some cases, ransomware may outrightly delete or corrupt the backup files rather than encrypt backup data. This deprives victims of any hope of recovering their data without paying the ransom, as their backup copies are now compromised.

Preventive Measures

Off-Site and Air-Gapped Backups

Maintaining off-site backups and using air-gapped storage systems can provide an added layer of protection against ransomware. Air-gapped backups are physically isolated from the network, making it significantly harder for attackers to access them remotely.

Encryption and Multi-Factor Authentication

Employ robust encryption for backup data and ensure that multi-factor authentication is in place for all backup solutions. This can help mitigate the risk of unauthorized access to backup repositories.

Regular Testing and Monitoring

Conduct routine testing of backup and recovery procedures to ensure that backups are functional and can indeed be restored. Monitor backup systems for any signs of unauthorized access or suspicious activity.

Conclusion

The ransomware threat landscape continues to evolve, posing significant challenges to organizations seeking to protect their valuable data. As ransomware attacks become increasingly sophisticated, businesses and individuals alike must adopt proactive measures to safeguard their data. Maintaining secure backups, implementing robust cybersecurity practices, and staying informed about the latest threats and prevention strategies are essential steps toward mitigating the impact of ransomware attacks and preserving our digital assets.

Why Backing Up Cloud Storage is Important

Cloud storage has become integral to modern digital life, enabling individuals and businesses to conveniently store vast amounts of data and access it from anywhere. The allure of cloud storage lies in its ease of use, scalability, and cost-effectiveness.  Dozens of cloud storage solutions exist as of the writing of this article.  Microsoft's OneDrive, Google Drive, and DropBox are the most popular ones.

However, as we increasingly rely on the cloud, there is a growing need to recognize the importance of backup solutions.  Since the data is already in the cloud and we often also have a copy on our local computers, why is it important to back it up?

Data Loss Can Happen Even in the Cloud

Many users mistakenly believe that once they store their data in the cloud, it becomes immune to loss or corruption. However, cloud storage is not infallible. While reputable cloud service providers invest heavily in redundancy and robust data centers, unforeseen events like server failures, natural disasters, cyber-attacks, or human errors can still lead to data loss. Without a proper backup strategy, these incidents could leave you helpless and unable to recover critical information.

Protection Against Cyber Threats

Cyber threats like ransomware attacks are on the rise, and they can devastate cloud-stored data just as they can on local devices. Ransomware encrypts your data and demands a ransom for decryption, putting your cloud-stored information at great risk. With a reliable backup solution, you can ensure you have clean copies of your data before the attack, allowing you to restore your files and avoid paying the ransom.

Human Error is Unavoidable

Accidental data deletion is a common occurrence and is often irreversible if no backup exists. Users can inadvertently delete important files, folders, or even entire cloud accounts, whether it's a misplaced click or a wrong keystroke. A backup system acts as a safety net, providing the means to recover lost data and prevent productivity and financial losses.

Cloud Service Outages

Despite the best efforts of cloud service providers, occasional outages may still happen. These interruptions can temporarily leave you unable to access your data when you need it. By having a backup of your cloud-stored files, you can continue working seamlessly, even during service interruptions.

Compliance and Legal Obligations

Various industries and businesses must comply with strict data retention and security regulations. Failure to adhere to these requirements can result in significant penalties, legal troubles, or loss of reputation. A robust backup strategy ensures you can meet these regulatory demands and demonstrate your commitment to data protection.

Cost-Effectiveness and Efficiency

Contrary to the misconception that backups are cumbersome and expensive, modern backup solutions have become more efficient and cost-effective. Many cloud providers offer native backup solutions, and third-party backup services often provide affordable plans tailored to different storage needs. Investing in backups today can save you from expensive data recovery services and potential downtime in the future.

Peace of Mind

A comprehensive backup strategy offers peace of mind to individuals and businesses alike. Knowing that your valuable data is secure and recoverable in case of any mishaps can alleviate stress and allow you to focus on your core activities without constant fear of data loss.

While cloud storage offers unparalleled convenience and accessibility, it is not immune to data loss, cyber threats, or human errors. The need to back up cloud storage cannot be overstated. A robust backup solution is the key to ensuring data resilience, security, and compliance. With the vast array of backup options available today, there is no excuse to neglect this critical aspect of data management.  Data backups are like insurance polices for your data.  So, take the proactive step and safeguard your cloud-stored data today!

Honeypot for spammers

Hey spammers!

If you want to be blacklisted by all of my mail servers, then please harvest these email addresses and send some of your spam here:

info@rhinoback.com
spammer@rhinoback.com
spammer@dataprobackup.com

If you are not a spammer, then please ignore those email addresses.

OMG Spammers are out in full force

We have seen a significant increase in spammy comments on this blog.  Some of them are so blatant and irrelevant to data backup and recovery that it is laughable.  From now on, all comments are moderated.  If the comment isn't interesting and somewhat relevant to data backup and recovery, then it will never be seen by anyone except a moderator.  

Feel free to post comments related to IT, data, computers, backup, disaster recovery, hardware, software,  or whatever.   But, make it something worth posting.   Short comments like "nice post" with a link to your website is not going to make it.   Links in your comments are accepted as long as the comment is good and relevant and the link is not a blatant attempt to sell something not related to backup and recovery.

Thanks for reading

Why You Need to Backup RAID Arrays

I never trust a single hard drive with anything.  All of my important business systems use RAID, mirroring, and sometimes even replication to protect against common hard drive failures.  Every hard drive is going to fail.  Even solid state drives are not immune to failure.  In fact, I am not yet convinced that SSDs are any more reliable than traditional spinning hard drives.  None of this keeps me awake at night.  Any hard drive can fail and I don't lose a bit of data.  Even with this protection, I still backup all important data because there are other failure scenarios that RAID, and mirroring, and replication don't help with.

We had a database server with high-end HP RAID system attached.  There were two physical arrays, one for the database, and one for the logs.  Each array was RAID AVG (AKA RAID 6), which allows up to two hard drives to fail simultaneously without losing any data.  In addition the arrays had hot spare drives that are activated if any drive fails.  We were also using the highest quality enterprise SCSI drives.  Sounds like we had it covered, right?

That database server went down when the RAID controller failed.  We had no access to any of the data, even though every drive was still healthy.  We couldn't afford to let the server stay offline for a day while a new controller was shipped in. In fact, no controller was going to arrive for a few days because of a snow storm.  Fortunately, we had the data backed up and it was easily restored to an alternate database server.

The loss of a RAID controller is an obvious problem that can occur and common RAID redundancy is going to be of little help.  There are many other scenarios where a good back is the only solution.  RAID, mirroring and replication definitely reduces the stress associated with a disk failure in a critical system, but it is by no means a substitute for effective and reliable backups..

Carbonite sues Promise after failures result in lost data

Apparently Online Backup provider Carbonite has suffered serious problems with Promise Array technology over the last couple of years. There are dozens of articles on the internet over the last month about Carbonite suing Promise. For example: Carbonite sues Promise Technology for unreliable storage. You can also find information directly on Carbonite's website: Further clarification on our lawsuit against Promise Technologies. And of course Promise has a response on their website: http://www.promise.com/support/Announcements.asp, however, they are trying their best not to draw any more additional attention to this case and don't even mention Carbonite.

While Carbonite is squarely placing the blame on Promise and at the same time trying to minimize the issue by saying that few customers were impacted, the publicity from this lawsuit may be damaging to both companies. I can clearly see how Carbonite and their customers have suffered pain. And I can also understand why Carbonite wants to hold Promise accountable for their flaws. I have no doubt that Promise Storage systems have serious flaws and reliability problems. However, Carbonite seems to have exposed some flaws of their own.

I have already seen several postings by IT Professionals, one on Carbonite's own blog, pointing out that Carbonite should share some of the responsibility for not implementing a better storage architecture. I don't think these comments relieve Promise for once ounce of their problems, but Carbonite could have done more to avoid them. I am not familiar with Carbonite's process and standards for selecting suppliers, but there may be issues with how the decision to select Promise was made. It's hard to make a case in this area because Promise does seem to have the redundancy, and hotswap replaceable components that you would expect from a high availablity enterprise class solution. I can easily say that regardless of how that decision was made, placing the Promise hardware into a critical area that could result in loss of critical customer data was a major mistake. I am sure Carbonite would agree with that statement now. They obviously had not gained enough experience with these Promise arrays to place them in such systems. But what's more troubling is the fact that they would allow any customer data to reside in a single place dependent on any single piece of technology, regardless of who made it, or how reliable they thought it was.

Those of us who have been in the storage and system management business for many years know that even the best technology can fail. I have decades of experience with enterprise systems and I know that most competent IT organizations have protections in place to prevent a total loss of data in almost any situation. It is ironic that Online Backup is one of the best ways to protect your data against virtually any disaster or event in your own environment.

I am very familiar with the operation of Rhinoback Professional Online Data Backup. I know that they replicate all data to mulitple independent systems just in case of such a failure as what was experienced at Carbonite. Rhinoback even replicates data to a second data center in the event that the entire data center is destroyed or inaccessible. A couple of years ago they also experienced failures with Promise RAID systems, but since they had replicated copies of the data, none of their customers were impacted. This is not just a pitch for Rhinoback, I am sure other online data backup service providers use similar approaches, they are the one I am most familiar with.

Another point I would like to bring to the surface about Carbonite's explanation that the 7500 customers who's data was lost on the Promise equipment that failed. Even though most of them didn't suffer a severe problem because they were able to get another complete backup done before they lost their local data, the fact remains that 7500 customers were exposed with no backup data for a period of time. This is not a good situation for people who think their data is safely stored in professionally managed environment. And even though they were able to backup their data again before they had a disaster, depending on the amount of data they had, it may have taken days or weeks and tons of bandwidth to achieve fully backed up data once again.

In Carbonite's defense; their business model seems to be based on providing the lowest possible cost of service to the largest possible customer base. This is great for the type of customer who considers price to be the most important factor, and there are millions of people who want this type of solution. Carbonite and apparently Mozy too, have done a wonderful job of making online backup available for a very small price to anyone who wants it. They must control costs or this model will eventually fail and then those of you who want to backup your data online for a very small price will not have a solution. I don't know that any of us should have the expectation that they will use the most expensive equipment and keep multiple copies of your data in multiple locations, and defend against every conceivable problem. Maybe we should all be happy that they do what they do. And, by the way I think they provide a valuable service and do a fine job of it.

If you prefer an online data backup service provider that is focused on business and professional users who need more high-availability and redundancy, then take a look at Rhinoback Professional Online Data Backup, or one of the many other fine online data backup services business and office use, but don't expect the prices to be as low as Carbonite or Mozy.