Five Essential Lessons to be Learned When It Goes Very, Very Wrong

in Columns/Technology
Joe Balsarotti

By JOE BALSAROTTI

As I write this column, my staff is in the fourth week of a nightmare scenario for a technology provider. One of our long-time vendors, who provided email and web hosting services, gave 30-day notice that the company was closing down. This followed a series of incidents in short succession of one client’s email being hacked, his tech making a grave error in an attempt to recover data and as a result, the destruction of a number of other clients’ data.

Luckily all but one client did get their data back, but significant downtime resulted for these businesses. Many of those affected by the shuttering of the vendor are our clients, but a number of them were not. Nevertheless, we offered to assist anyone who wanted help during the transition.

Through our industry associations, I’ve always kept a list of trusted colleagues who provide products and services that are complementary to ours. I’ve also kept a list of trusted colleagues who could act as a backup if we needed help. That planning paid off. We were able to work with a company who provided similar services and had the resources to be able to onboard those who were stranded in the digital sea before the looming deadline.

The process of switching all these companies over has been daunting. Each client’s setup is a little different. Some just use email, others depend upon their websites and many utilize both. This was about as close to a nightmare scenario as one can get, except that there was at least a month’s notice.

Working through all of this brought to light many important points.

First off, as we’ve covered before, do you own your data? A handful of the clients had their website created by that vendor and were never given access to it, nor were they given a copy of their data. Unfortunately, that means they will have to pay to have a new website designed for them. It’s not something they were expecting or budgeting for. And until the new site is completed, they don’t have a presence on the web.

Secondly, moving email providers is about the most labor-intensive project one can have. Someone has to touch not only every computer, but also every email and user account on each computer. Local data needs to be backed up, changes made, connections tested and data restored. With the volume of email some users keep, this can be a two- to three-hour process just for one account. Now multiply that by about 800 and you comprehend the scope of the task belonging not only to my team, but to the new datacenter and its staff as well.

We did our best to prioritize those who needed 24/7 access to their email, those who could do with checking through webmail during a transition and a select few who welcomed a day or two of digital peace and quiet.

“Always in motion, this project is,” paraphrasing Yoda. Like any major project your business takes on, making sure you’ve accounted for how much time is spent on each client is essential. With so many separate clients and each user needing attention, it’s easy to lose track as techs get sucked into a rabbit hole resolving one problem or another. Organization is key. Even if it starts to fail under the stress, at least there is a framework to keep the project moving forward.

The third point that reared its ugly head was documentation, or the complete lack of it at many clients. The vendor dealt with these clients directly. And although we often acted as a consultant or even a translator between them, we were not privy to the transactions between them. Many clients were sent scrambling to find account numbers, passwords and to track down who controlled (or even owned) domain names. Getting in contact with the website designers was nearly impossible, as many sites were old and their designers had long since moved on.

The fourth point brought to light was the lack of attention to websites. With only a handful of exceptions, all these clients’ sites were running drastically out-of-date software, security was nonexistent and the web designers who (as we’ve discussed in this column before) should have been doing ongoing maintenance on the sites, never even bothered to discuss it with the clients, let alone sell them on it. One of the affected sites was infected with a virus. As such, we could not download the client’s data because our security systems prevented it, as they should have. If we would have had another month to go through the process, we would have been able to save it, but the cost of cleaning and updating data just to be able to transfer it probably exceeds what designing from scratch would cost.

Lastly, misplaced trust in the cloud was amazing to me. As many times as we preach to clients that they must make backups, the number of businesses who made the faulty assumption that the vendor would have everything available forever – and they didn’t have to take any precautions themselves – was astonishing. The initial problem this vendor had was exasperated by a technician’s error, which then revealed that their assurances of data security were not as they had been portrayed.

The moral of this story – which continues to play out – is that with organization and a little planning, your business can survive a major disruption such as losing a key vendor. There will be a price to pay, however. Lost productivity, unforeseen costs and the potential of lost business are givens. How impactful this can be to your business comes down to how much time and effort you’ve put into being prepared.

Joe Balsarotti is president of Software To Go and is a 38-year veteran of the computer industry, reaching back to the days of the Apple II. He served three terms as chairman of the National Federation of Independent Business’ Missouri Leadership Council, as chairman of the Clayton, Missouri Merchant Association for a dozen years, chaired Region VII of the Federal Small Business Regulatory Fairness Board and currently serves on the Dealer Advisory Panel of the ASCII Group, an organization of nearly 1,200 independent computer and technology solution providers in North America. He can be reached at joe.balsarotti@software-to-go.com.

Leave a Reply

Your email address will not be published.

*