When their rural Minnesota telco was hit by a ransomware attack, staff sprung into action to both combat the attack and communicate with customers and coworkers about its effects.
Transcripts have been lightly edited for clarity and readability.
Andy Johns: What can you do if your telco gets hit with a ransomware attack? That’s what we’ll be talking about on this episode of StoryConnect: The Podcast. My name is Andy Johns with WordSouth, and I’m joined on this episode by Kevin Beyer, who is the CEO and General Manager of Federated Telephone in Chicago, Minnesota, and Farmers Mutual Telephone in Bellingham, Minnesota. Kevin, thanks for joining me.
Kevin Beyer: You’re welcome.
Andy Johns: So, Kevin, I appreciate you being on. This is kind of a topic we hadn’t really gotten into on this podcast before, but it’s something that, unfortunately, you guys have a little bit of experience in. I know that we had talked earlier, you guys experienced a ransomware attack. So if you don’t mind, just kind of give us the high level. What was that, and what did it do to the telco there?
Kevin Beyer: So, yeah, back in June, it was a Sunday, I received a call from my IT supervisor that we were in the midst of what looked like an attack on our network. And he was wondering if I had outsourced someone to probe into our network and simulate an attack to test our systems, which he was sad to hear that I had not done that. We found out that there was a ransomware attack going on. In other words, they were encrypting our networks so that we would not be able to access any of the data or files inside of our networks. And he immediately after that conversation called all of the staff that he could get a hold of to go in and start turning down machines and unplugging machines if they couldn’t access them to turn them down, to limit the amount of exposure we would have to the attack.
Andy Johns: I’m sure that was not a call that you wanted to get or that he wanted to place. So when these have gone on and there’s been some of the headlines when ransomware attack will hit a city government and sometimes that has gone on for weeks in some of those cases. How long did this have you guys down for?
Kevin Beyer: Well, so we found out about it on Sunday and like I said, we went in and and shut down everything, equipment unplugged, other stuff. Continue to try to figure out where they have infiltrated. And luckily, we had recently renewed our cyberware insurance that we had. And the thought occurred that we should call the insurance company to let them know. Immediately upon calling the insurance company, they have a total expert group that they then turn you over to, and depending on what type of attack is happening, will determine the experts you have. But they immediately give you a forensics company to start figuring out what’s going on, who immediately gets into your network and starts figuring out to what extent you’ve been hit. Then they have a cyber attorney to figure out what kind of communications, if any, you should be doing and who you should contact as far as authorities, if any. And then they also, in this case, give you what’s known as a negotiating team if you determine that you want to negotiate for the code to unlock what they have locked up. And so that was a great call because that forensics company, as soon as we could get them in, was able to help our team substantially and figure out what was going on, to what extent, and was able to find additional places to shut down so that they could not continue to do stuff. The other thing they figured out, though, in short order, which was unfortunate, is they had actually gotten in two days earlier and had found where we had been doing our backups. That’s typically what they do, is they try to figure out what they can access and then they figure out where you’re doing your backups so they can totally delete and destroy all your backups so that you’re unable to restore to previous days so that you are truly shut down until you can figure out how to unlock or actually scrub the computers and start over.
Andy Johns: Wow, so yeah, they were making sure you wouldn’t just lose a day, you were going to lose everything you had backed up. So what happened next then? I mean, with the ransomware, was there some kind of demand that they made? Or you said the negotiating team was there. What did you learn about the folks who were doing this, and did they make any kind of demands?
Kevin Beyer: So, yes. So one of the things that happened shortly after, I guess actually prior to him making the call to me on Sunday, was that one of our employees had noticed that there was stuff spitting out on our fax machines, our printers, that were part of the network and were saying something about, “you know, go to this website if you want your system restored and find out what your demand is.” So they were like, what is this? And so that was actually the website you needed to go to to be in contact with the folks that had initiated this attack and would tell you what it is you would need to pay them to get the code to unlock it. And the ransomware team immediately figures out who are we dealing with. What does this code look like? Have we dealt with these people before? Are they likely to provide the code or not? And then they go through a process of pretending to be us, the telephone companies, and not a ransomware negotiating company, so that they can understand who we’re dealing with, to what extent we have a negotiating leverage power, what they know about our financial situation as to what we can afford to pay. So that was interesting to listen to them tell us that, “well, you’re lucky in one extent that, hey, these are new guys. You’re unlucky on another extent to know whether or not we can trust them.” So the next step they do is actually ask you to pride them what you think are a couple of files that would likely be pictures to send to them and ask them to send their code to or take a code and unlock those files on a site that both of you can see and then figure out whether they actually will decrypt that file so you can then see the photo. And so that was the next step we took, and it worked.
Andy Johns: Ok, so things were things are promising then. But while this is going on, so I guess this starts on a Sunday, but that Monday you had employees coming into work to do their jobs. So I guess one of the first concerns, I imagine, will be checking and seeing how the customers, how the members, are impacted. But how did you go about communicating a) with the staff and then, b) with any customers that were impacted by this?
Kevin Beyer: Yeah. So this started to be more chronologically accurate for whoever’s listening. It started at 6:00 p.m. on a Sunday. And by the time we got to midnight on Sunday, we were starting to understand what had all been affected and what all had to be shut down. And by Monday at 8:00 a.m., we had everything down and the forensics company was in looking at what they could look at and determining if there’s any additional malicious software that’s going on or any additional attack holes that could be exploited. So your staff walks in the door and you tell them all they can’t touch anything. They cannot use a single computer. They can’t access any of the data we have anywhere. And their first question is, OK, so when the phone starts ringing, how do we take care of problems? How do we turn up new customers? How do we solve any billing question? Anything. And the answer is, you can’t. You have no information. Zero. And so what we did is we actually took our phones and put them on our after hours answering service, told the operating service that we would not be able to open the day due to unforeseen circumstances, and they should just take calls and forward them to us. So that’s what we did as all staff got all of their computers and started to work with the IT department. And as the forensic company to basically go through each computer to figure out what’s been affected on an individual computer basis.
Andy Johns: And I know you guys are spread out through a couple of offices. So this was all of the different offices. It wasn’t just one where you could move folks around. It was everything?
Kevin Beyer: It was everything because they hit the main server that we would call our network server. So it’s the server that has access to get into everything. So if you want to send a print job out, if you want to access the phone system, if you want to look at any of our billing system, accounting system, get into the switch. Get into what we use a Calix equipment. You couldn’t because you couldn’t access the main server.
Andy Johns: Wow. Yeah. So pretty much pretty much everything. So the good news, I guess, if there was any good news, is aside from the folks who are calling in or trying to change something on their account during that time, there were not widespread network outages or anything like that. At least you had that going for you.
Kevin Beyer: Correct. It had not affected any service to any customer that was not hosting anything on our network. And we did have two customers that were hosting their phone system with us that were affected. But other than that, no. No services, no Internet, no cable TV, no telephone services were affected. We just couldn’t see it. We couldn’t access it. But the fact that no customers calling in with any complaint and, of course, we reached out to some friendly people to see if their service was working. And, of course, we had our own employees look at their services to see if they were working. And everything was working, so that, you know, there was the lucky part. This was truly a ransomware attack. And they just wanted to lock us up and make us pay them to unlock it is what this was about.
Andy Johns: So did the forensics team find anything out? I mean, do we know if this was domestic or abroad or do they know anything about who these folks were? You said they were kind of new guys to the business — I guess it’s the business — the criminal enterprise, whatever it is. Did they find anything more about who these people were?
Kevin Beyer: Yeah, so they did. They were able to tell us that it was one of the Baltic — I can’t remember exactly. I said, but basically, stuff that broke away from the Russian Federation and Baltic kind of state situation. And that they had accessed our network through one of our employees. However, they had done a good job of erasing their tracks, so they could not tell specifically which employee. All they could tell is it initiated through email, and that somebody either click on something embedded in an email or went to a site from an email and clicked on something that let them in.
Andy Johns: That simple?
Kevin Beyer: Yeah, it was interesting. All the training you seem to do to your employees, somebody made a mistake. And there’s just no way, in my opinion, to stop it from happening. Because, I mean, you’re looking at 30 employees getting 50-60 emails a day. Eventually, someone is going to click on something the way it looks. The good part is, of course, is you learn afterwards, what you can segregate, so that they cannot get to certain things. Now, we do have completely new software that we use internally to monitor. We have new software that every employee is required to use. We have different authentication service that we do. So we had thought we had put good things in place, but, evidently, not good enough.
Andy Johns: Got it. I guess it’s like a lot of things. We’ve had folks on after we record a podcast with with folks after they’ve gone through a hurricane or an ice storm or different things like that. And that’s when they work on their disaster plan is, it is after after it happens, they’ll have a good plan for the next time. So what kind of things — you mentioned a few of them there — but in terms of a plan for anything like this. I guess, number one, did you have very much of a plan in place or a playbook as to what to do if this happens? And the number two is that is that anything you guys have been developing since then to be ready in case it ever happens again?
Kevin Beyer: So, yeah, we had plans in place, disaster recovery plans. We had plans in place on if certain services went down, or if certain things were attacked, you know, as far as services. But in this case, since it was just our server locking basically us out of accessing any of that and our customers, so we really didn’t have a good plan for that. And we actually went to day two telling our after hours service to do exactly the same thing. And on day three, when we started to take a few calls because we got a couple of clean laptops from our telephone association and their disaster recovery vehicle, which had a couple of clean laptops that never been on our network and got those booted up and started to be able to access our billing system and our accounting system, we start taking calls. But there was rumors going around town about all the things that could have happened to Federated because it was very unusual. And one of the more entertaining to me was that Kevin Beyer had died, and they didn’t know what to do. So they shut everything down and the offices until they could figure out what to do. So on day three, I went uptown and got meals for everybody, so everybody in one of our towns at least could see that I was alive and washed that one.
Andy Johns: We are happy to break news on this podcast that Kevin Beyer is alive and well, so we’re glad glad to hear that.
Kevin Beyer: Yeah, there was I mean, there was OSHA things that people were talking about. There was all kinds of weird little things going around in different towns. So communication is an important thing that we hadn’t thought about. Is how do you necessarily communicate to the public properly? Unforeseen circumstances doesn’t work long enough.
Andy Johns: So sure, with a lack of information, people are going to make up their own information for sure. So and that brings me kind of the last thing I was going to ask you about. Maybe talk more about that communications plan, but what advice do you have for folks either, I guess we could talk about folks who haven’t gone through a ransomware attack, but particularly if this ever does happen to folks who are out there, you know, other independent telcos like yourself. What advice do you have for folks who are going through something like that? Anything that you may have learned through the process?
Kevin Beyer: Yeah, so two things. I think more along the internal workings with your employees is — day one, day two, I noticed immediately that there was a, I would call it kind of a chaotic, frantic atmosphere within the different offices that was going on. And everybody was trying to talk to the IT folks, and IT folks were I can’t answer all your questions. I’m trying to do work here. I’m trying to figure out what’s going on and get these computers cleaned and figure out which ones are affected. And I don’t have time to answer. So the long, short there is you need somebody to be the person in the office that’s not part of the team that’s restoring or dealing with the computers to answer the questions for the staff and communicate with, what I would call the IT department. So that became me. Everybody that’s not IT comes to me. I go to IT. You don’t have everybody asking them questions and trying to deal with all the different folks there. So you need that piece.
Andy Johns: That’s good insight.
Kevin Beyer: So you have that centralized person, and then figure out your message to the public. Because one thing you don’t want to tell the public is that we’ve been hit with ransomware. We’ve been hit with a cyber attack because then they all freak out about their information, what was compromised. And in this case, nothing was. So it’s a good thing we never said it. We never actually had to make any notifications to anyone because nobody’s information was taken. Nobody was copied or downloaded. They were able to tell that. They simply locked everything up. So we didn’t need to freak people out about their personal information. And that’s another thing your employees are wondering about, all their personal information. None of that was compromised. So those are things we learned and now we use an authentication software additionally. We chose a company called Thycotic, there are other ones out there. And we also determined as we were going through that that we needed to let all employees do their personal stuff on Thycotic as well as the company stuff, because otherwise they have two different systems or two different pieces of software they’re using to store their passwords. And you don’t want them storing their company information company passwords on their personal. So that’s why you allow them to store their personal on the company.
Andy Johns: That makes sense. I’m sure there were a lot of lessons learned. It’s interesting. You mentioned that response vehicle. Was that the MTA, the IRV vehicle, they had there that you said was able to come help out with laptop?
Kevin Beyer: Yes. And I’m happy to report that after our incident, they now have, I believe, six additional clean computers, that if anyone else gets hit, they are able to bring clean computers, never been on your network. You can get up and go much quicker because those two were huge, but I wish I would have had a dozen.
Andy Johns: Nice. Well, that brings it kind of full circle. We had Brent from the association on, it’s been a couple of years ago, but we had him on talking about IRV when they rolled it out there. Now we have heard — obviously you wish you didn’t have to use it, but it’s good to have a resource like that. That’s fun. The podcast comes full circle there, I guess. That’s great.
Kevin Beyer: Yeah. And we actually used IRV, the vehicle itself, as one of our offices because we tried to get everybody into one of our three offices so that we could communicate properly and have access to everybody’s computers, laptops. We had everything brought to that one location. And so we needed extra space. So it actually worked as office space as well.
Andy Johns: Got it. Well, Kevin, I appreciate the insight there. I think it’s something, hopefully, that the listeners out there won’t have to deal with, but if they ever do, I think they’ll be better prepared, having heard that story from you guys. So thanks for taking the time.
Kevin Beyer: You’re welcome. Take care Andy.
Andy Johns: He is Kevin Beyer. He is the CEO and General Manager at Federated and Farmers Mutual Telephone Cooperatives up in Minnesota. And I’m your host, Andy Johns, with WordSouth. Until we talk again, keep telling your story.