Expedient: The Podcast

00:00:06:18 - 00:00:28:18
Bob Arnold, President of DRJ
Hello and welcome everyone to DRJ's webinar series. I am Bob Arnold, president here to this cover Journal, and I'm glad everyone could join us here today for today's webinar. Our webinar today is sponsored by expedient. And we'll be discussing modernizing disaster recovery lessons in resilience from UTS brands. Before we dive in here, just a couple of housekeeping notes.

00:00:28:18 - 00:00:48:04
Bob Arnold, President of DRJ
Today's session is being recorded, and everyone will receive a link to this recording tomorrow via an automated email. Attendees are also in a listen only mode. If you do have any questions, please use the question panel you locate within the Go to Webinar tool. And we're going to address as many of these questions at the end of the session.

00:00:48:04 - 00:00:56:19
Bob Arnold, President of DRJ
And now back to today's topic on modernizing disaster recovery. And I'd like to now welcome turn it over to AJ Kuftic with the Expedient.

00:00:56:21 - 00:01:08:18
AJ Kuftic, Field CTO at Expedient
Hello everyone. My name is AJ Kuftic I I'm field CTO of Expedient. I'm very glad to be on the the DOJ panel here today. And with me is Tim Swingler of Utz brands. Tim, how are you today?

00:01:08:19 - 00:01:10:08
Tim Swingler, VP of Infrastructure & Security at Utz Brands
I'm great.

00:01:10:10 - 00:01:20:14
AJ Kuftic, Field CTO at Expedient
So for those of you who aren't, for those of you who aren't familiar with Utz Brands, Tim, can you kind of explain what UTS brands does? I hope most people know.

00:01:20:16 - 00:01:47:00
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Well, if I go to this site or not. Little Utz girl behind us is, is our logo. So most people know seeing her, it's like, oh, that's that's the little Lutz girl, right? But we are one of the largest companies, manufacturer, salty snacks. Frito lay is our arch nemesis. So, but, yeah, so aside from the arts name, though, we are a brands.

00:01:47:00 - 00:02:06:20
Tim Swingler, VP of Infrastructure & Security at Utz Brands
There's more than one brand than us. So we we make, for instance, on the border, chips, tortilla chips. We make pretzels. SAPs is another Boulder canyon. So there's, there's a portfolio of brands that, we also we manufacture and ship out for, for consumption.

00:02:06:21 - 00:02:29:21
AJ Kuftic, Field CTO at Expedient
And, and just this is a pure, client testimonial. All their snacks are good. And I'm not saying that because they're a customer of ours. I'm saying it because my my belly will tell you. So, Tim, when it comes to the Utz brands, you're making snacks, right? How does it play into that? And in, in terms of disaster recovery, how does this play into the business?

00:02:30:01 - 00:02:55:06
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Well, I'll tell you, when I joined Utz back in 2016, its role is was pretty minimal, right? We could do ship ins and everything out, you know, without minimal involvement. But, today, its role is, is very large in the production process and getting that out there. So for us to make move ship a pallet of chips requires a lot of data entry.

00:02:55:06 - 00:03:20:17
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right? There's there's checks and balances all across the way, you know, from leaving a warehouse to sending that out to, a distribution center to ultimately land it to a store, for you to, you know, to, to purchase. So, you know, today's world, we can still make product, but our, from if we had a downtime event at that, warehouse would fill up very quickly.

00:03:20:19 - 00:03:42:11
Tim Swingler, VP of Infrastructure & Security at Utz Brands
It's more so a perspective of not being able to move the product without, its involvement. And lastly is that. I'm sorry, I, you know, I think one of the things that we have used, is, is is trying to use a best of breed approach with a lot of our systems. We generally try to use, cloud providers for a lot of systems.

00:03:42:11 - 00:04:02:18
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So it's not like we have one massive ERP system that if that was to go down, it would be, critical. We do use, a number of, partner vendors to kind of help us out. So from a resiliency perspective, we're kind of covered in multiple fronts when it comes to disaster recovery.

00:04:02:20 - 00:04:27:18
AJ Kuftic, Field CTO at Expedient
So you're talking when you're talking about like the shipment of things and data entry, there's obviously a big impact that comes into the business. How are you doing. Disaster recovery. You know, through this history of, you know, increased it. Utilization is part of the production process. Now, the business probably sees it. Disaster recovery as a is kind of an imperative, right?

00:04:27:19 - 00:04:50:18
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah, I would say more. So the biggest impact is looking at our Disaster Recovery forum from an analytics perspective. You know, where where how much did we make yesterday like how how productive were our minds. You know, where are our gaps in terms of from a sales perspective? Where do we do we need to make more of this chip or less than that chip?

00:04:50:20 - 00:05:02:04
Tim Swingler, VP of Infrastructure & Security at Utz Brands
And if we don't have that in our, in our, for our upstream to be able to make those daily decisions, it does have a dramatic impact on our business.

00:05:02:06 - 00:05:14:19
AJ Kuftic, Field CTO at Expedient
So can you could it be like a brief history of how us kind of got into Docker? Because obviously everybody's Docker story is different, but what's kind of been the history of Docker so far at UDS?

00:05:14:21 - 00:05:37:18
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah, it's definitely evolved over the years. Like, like a lot of things, with technology, when I first came on board, we had our, all of our virtual servers, we're using, Delphi's in which basically we could leverage what we call a stretched vSAN, where our primary data center would, you know, basically be replicated.

00:05:37:20 - 00:05:52:14
Tim Swingler, VP of Infrastructure & Security at Utz Brands
And here into that technology to our, our Dr. site, both of which were in our corporate headquarters in our campus, which presented a lot of challenges. So, so.

00:05:52:16 - 00:05:57:08
AJ Kuftic, Field CTO at Expedient
We have to see your Dr. site from the production side is a bad thing. Is that what you're saying here?

00:05:57:13 - 00:06:31:09
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah. Like, you know, it's not redundant. You know, we literally get, you know, walk to the doctor site. So that's that was our starting point for for disaster recovery. And, you know, it honestly took until I just went public as a organisé, you know, publicly traded on Wall Street for us to, for our executive team to say, hey, you know, for compliance reasons, we need to invest into, a true Dr., system, you know, that that had all of that.

00:06:31:09 - 00:07:08:01
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So, you know, it took a little while for us to do that. And when we did do that, we we still had our on premise data center, but all we were really doing was replicating that data, to to an offsite facility in, in Dallas. And, we leverage that for, for a period of years. And then ultimately, we made a decision, to move our production data center, to expedient, where, you know, we, you know, had that full, you know, full, redundancy that we were looking for for not just Dr..

00:07:08:01 - 00:07:36:10
Tim Swingler, VP of Infrastructure & Security at Utz Brands
But also for our production data center. Now, at the same time, we moved to expedient we we continued to use the same, Dr.. Provider that we had because we were on a, you know, multi year contract. And, so it took us a year later for us to move our tr, over to expedient so that basically we had our production data center as well as drives up both under one, one vendor.

00:07:36:12 - 00:07:38:03
Tim Swingler, VP of Infrastructure & Security at Utz Brands
And that's where we are.

00:07:38:05 - 00:07:54:16
AJ Kuftic, Field CTO at Expedient
When you start talking about drives and Dr.. As a service is how has that been seen inside of inside of OTS and like what does that bring to the table. Because you mentioned you had Dr.. That was that you were already using but what is Dr.. As a service really bringing to the table this different there.

00:07:54:18 - 00:08:37:05
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So what we really wanted to have with that solution is just have that ability to failover without hesitation. And I think that's a lot of the challenges, I think that most people with with a Docker environment have today is, if, if, if you had to make the decision, and would you pull the trigger and, and do a Docker failover event because I will you know, I've, I've talked to a number of my colleagues in the industry and it's more often that situation where, you know, if something's down, but it's not like the complete data center got completely whacked, right where everything is down.

00:08:37:06 - 00:09:04:01
Tim Swingler, VP of Infrastructure & Security at Utz Brands
There's all kinds of those scenarios. And, you know, I will say that having our production data center and our TR under one provider, is so much more seamless. I mean, so we are now in that situation where we just know that someone has our back and that it's now all on us to, to kind of if we did have to do that, we could just pull that trigger if needed.

00:09:04:02 - 00:09:31:08
AJ Kuftic, Field CTO at Expedient
And I think that's really where at least we as expedience see ourselves in the Dres, and even in the production space of being the experts on that technology and helping get people to where they need to be. But that does include a level of trust. And one of the things that Tim, you and I have talked about in the past is that there is a level of like, you are really putting yourself into the hands of someone else, and how does that level of trust come about?

00:09:31:08 - 00:09:37:07
AJ Kuftic, Field CTO at Expedient
Like, how do you how do you build that trust? And when finding a Dr. as provider like that?

00:09:37:09 - 00:09:56:17
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah, I get you know, there's a lot of I mean, a lot of IT professionals get calls from vendors all the time called calls, emails. Hey, we've got this, we've got that. And honestly, we don't really have I don't have the time and a lot of people time to go and explore all of those various options there.

00:09:56:19 - 00:10:21:18
Tim Swingler, VP of Infrastructure & Security at Utz Brands
What we what we did from a, from a trust perspective is partner with, a company called AAG. Their, their goal is to provide clarity. For, for it in particular where, you know, they are on the pulse of what who's, who's in the market, who's there. And really the key that they provided for us was to say, understand what our business needs are, right?

00:10:21:18 - 00:10:50:00
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Understanding, coming out and just looking at our our production floor, looking at our warehouse, like, understand the true business need of what's going on there. And, you know, and sorting out all of those, Dr.. Device options that are available. And, you know, it came down to really, in the selection process was, you know, selecting expedient as is the clear winner for us.

00:10:50:02 - 00:11:05:12
AJ Kuftic, Field CTO at Expedient
And when you start to look at this, how does flexibility play into that as well? Because obviously you're you're trying to build out not just for today and solve the problem for now. But how does this play into the future now having prod and drives in the same place?

00:11:05:14 - 00:11:34:15
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Well, I think I kind of put put this in the perspective of like infrastructure in the world of it. It's kind of like if you think of a business as a human body, infrastructure is like oxygen, right? If we don't have the infrastructure, you just can't breathe right? It's it's it's the lifeblood really of from and you know, it is the lifeblood when it comes to infrastructure for for for organization.

00:11:34:15 - 00:11:35:09
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So yeah.

00:11:35:13 - 00:11:46:20
AJ Kuftic, Field CTO at Expedient
Every time I see an exchange or an office 365 outage, I'm like, oh, it's not important till it's incredibly important. Now everybody needs their email and nobody's happy. And now they're gulping for air.

00:11:47:00 - 00:12:11:12
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Exactly right. So you know with the so same thing is with our data center partners. When we say go, it's just got to work, right? If we have to fail over or if we we, we have a bump in the road, we just need to have that, you know, 99.99 uptime, right? To know that that trust is going to be there, that at the end of the day, things are just going to you just expect it to work.

00:12:11:12 - 00:12:15:07
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Just like you need oxygen to breathe.

00:12:15:09 - 00:12:39:01
AJ Kuftic, Field CTO at Expedient
And when we start kind of digging into this from a driver's perspective, obviously you were building out things before. How did the business look at the capital angle of potentially doing this yourself? Because I'm sure that came up at one point of why don't we just buy some hardware? We don't need to pay somebody to do this. How did that look to you in the business in terms of a, you know, CapEx versus and opex model?

00:12:39:03 - 00:13:00:18
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Well, I will say that, you know, every organization's a little different. We actually prefer CapEx just in our business model. It works well. And there's been a lot of, you know, discussions with our CFO on terms of, hey, can we capitalize versus an operational spend? Other companies prefer a true operational spend. They want to have a consistent spend month over month.

00:13:00:20 - 00:13:28:16
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So we kind of we kind of delved in the middle of that, you know, we could have gone the traditional path of, you know, looking at, you know, redundancy using our native, you know, for instance, our we use pure storage, right? For our, for, for our solution and pure has it's own, you know, if you have an array that's in a different location, you could use that as a failover.

00:13:28:16 - 00:14:00:09
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right. And so there's there's multitude of type of, you know, failover solutions that we could have looked at. And we did go through that, as part of that process as to, you know, what would work best based on, you know, what solution that you're looking at in terms of, do you have Nutanix or do you have this vendor, that vendor Cisco, you know, so what it came down to for us is really, is using a solution just as Dr..

00:14:00:09 - 00:14:31:06
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Regardless of what, you know, what solution compute and storage that you have. You know, we we leverage sort of and regardless of, you know, what product we have, that's that's the solution. So that in it that alone is is really cost effective for us. Obviously with Dr.. We're not really interested in, you know, sourcing CapEx for secondary hardware for, for our location.

00:14:31:07 - 00:14:59:07
Tim Swingler, VP of Infrastructure & Security at Utz Brands
But in some instances, you know, for instance, with, with our expedient, we basically have a private a private cloud and the the hardware is, is is basically, procured by expedient, managed by expedient. But in the model that we used, it was that it's, it's, it's hardware. So at the end of the contract, if we did want to take it, take it somewhere else, by all means, we have the ability to do so.

00:14:59:08 - 00:15:24:05
AJ Kuftic, Field CTO at Expedient
So and I think that's where some of this comes into play of not just thinking about today, but thinking about tomorrow. And I know one of the big things that always comes up in IT projects is, well, we could just go buy this hardware, we could buy this licensing. But from a staff standpoint, how does this affected your ability to take on new capabilities and handle that with the staff that you already have?

00:15:24:05 - 00:15:40:17
AJ Kuftic, Field CTO at Expedient
Because I'm guessing much like everybody else, nobody's getting extra hands, right? Nobody's getting extra bodies to come in and do things. But the oxygen still has to flow. So how are you and your organization utilizing this as a way to to handle that time and that effort?

00:15:40:19 - 00:16:01:06
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah, I will say that, you know, my team is very small in size. You know, we my team does not have dedicated people who look at this system and that system that say, we have many guys that do a jack of all trades type of thing. So we're very resource challenge. And our goal is to focus on on our business needs as our priority.

00:16:01:07 - 00:16:27:11
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So from a doctor perspective, the thought is let's have the experts do what they do best. Now that doesn't take the onus off us completely. Let's say, okay, well expedient. You just manage that. You know, we trust you implicitly. So to handle, you know, monitoring, managing of both our TR and our production data center, I would say that metrics are a key in this area.

00:16:27:13 - 00:17:03:12
Tim Swingler, VP of Infrastructure & Security at Utz Brands
In, in particular, you know, are we measuring in, using the portal effectively to do weekly checks or is all of our, our all our virtual servers being replicated? So in the event of a failover, we know that everything's current with, with the server replication, for instance. And same thing for our backups. You know, doing periodic, restores of that data, or for those VMs, excuse me, that, you know, filling that back, and, and making sure that they would be operational as well.

00:17:03:14 - 00:17:25:08
AJ Kuftic, Field CTO at Expedient
Yeah. It's it that's something that comes up a lot is how do you handle those things that are outside of our control? When we when we look at things, we don't know exactly what's being built. You could build every server that you want to, but I don't know. What's your new production server. That's the most critical thing in your environment now versus what was previously the most.

00:17:25:10 - 00:17:47:07
AJ Kuftic, Field CTO at Expedient
So I think that's where from a as providers we're helping to provide that service. But it's also working in tandem with you and your team at OTS and any of our other customers who are worth it with us as Dr.. Provide as a Docker client. Excuse me. When we start to look at this, the same problem that has befallen many a person before doing it themselves can befall them again.

00:17:47:07 - 00:18:06:16
AJ Kuftic, Field CTO at Expedient
As a provider of hey, we built these like new ten servers, but no one put them in the protection group so they didn't get replicated. And so when we went to go do a test, they just weren't there. So being able to consider that as part of the overall service, how does that played into your, you know, process?

00:18:06:16 - 00:18:19:19
AJ Kuftic, Field CTO at Expedient
As you mentioned, you're checking to see if things are regularly replicated. But how does that fit into hey, we deployed this new thing. What's your process there to ensure that it gets added in the replication? Or how do you tag those things? What does that look like for you?

00:18:19:21 - 00:18:52:09
Tim Swingler, VP of Infrastructure & Security at Utz Brands
I would say, you know, we we've kind of learned just through, just going through the motions of that. I'd say that is there really a, a defined, method? Now, I think what we've done is just basically, you know, make, maintain, having a, a cmdb or configuration management database so that when you build, a VM from, from scratch that, you know, have you done all of these checks making sure is it in this, you know, is is it being, is expedient aware of it.

00:18:52:09 - 00:19:14:18
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Is this being replicated? Is, is and, you know, and fortunately we have a portal that gives us that visibility. It's not just, opening a ticket with expedient and saying, hey, just want to make sure you guys are replicating this. We, you know, that the nice thing is we have that visibility. The portal to actually see, you know, what's going on.

00:19:14:20 - 00:19:16:12
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So. Yeah.

00:19:16:14 - 00:19:40:10
AJ Kuftic, Field CTO at Expedient
Yeah, that's that's what we call our co managed services. I've worked with other providers where if a piece of dust lands on the, on a server in the data center, you have to put in a ticket to get them to dust the server off. It's very, for somebody who goes from an on prem environment to a non on prem environment or into a provider, it can be very annoying when you go into a space like that where you're kind of you feel like your hands are tied.

00:19:40:12 - 00:19:57:20
AJ Kuftic, Field CTO at Expedient
So being able to say, like you said, go into a portal and say is what is this? What does this protection group are all my VMs in here? If they are in here, what's the replication status and not have to jump through nine hoops to get there? Or ask the weekly tickets? Hey, what's my replication status? I think that's key.

00:19:58:00 - 00:20:12:21
AJ Kuftic, Field CTO at Expedient
And so what does that allows you to focus on from a core business side? Right. Because you're now, you know, kind of offloading some of this to the, you know, the management of the secondary site and hardware, etc.. What does that allows you to focus on from a core business activity side?

00:20:13:01 - 00:20:41:13
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Well, I think the biggest thing that for us is just the other, you know, value adds that we have is that making sure that, you know, we can, you know, focus in on other areas because there's, there's always enough work to go around. Right? But it's always that for us was, you know, letting letting up a vendor partner handle this was just it open the opportunities for us to focus in on getting more business centric problems.

00:20:41:13 - 00:21:10:08
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right. You know, whether it's, you know, opening a new plant location or focusing on, on other things, I think it just makes it easier for us to do the proper due diligence when we do some of those other projects. Right, that we don't have to effectively deal with the fact that, you know, or just go, shortcutting some other things, because we have to deal with all of these things because we are resource challenged.

00:21:10:13 - 00:21:15:16
Tim Swingler, VP of Infrastructure & Security at Utz Brands
If you follow.

00:21:15:18 - 00:21:33:11
AJ Kuftic, Field CTO at Expedient
Sorry, I could not find the mute button there. I mean, and that's I think the I think that's the big thing is that your staff is not big. And for a lot of organizations and even larger organizations that we work with, they still don't have huge staffs. So how do you put them in the right place and being able to focus on that story?

00:21:33:14 - 00:21:51:03
AJ Kuftic, Field CTO at Expedient
What's funny is that story that you just said about, you know, opening a new plant and all the things that go into doing something like that without having to give up on all the stuff that you do to keep the lights on. We've heard that story time and time again, especially for manufacturing companies. Because the you guys make snacks, you don't make it.

00:21:51:05 - 00:22:09:12
AJ Kuftic, Field CTO at Expedient
And when by the way, the the making sure that the snacks are good is why I come to work every day. So when we look at these sorts of, you know, challenges and things, what do you see? Because you mentioned you didn't want to stay in the secondary data center and you moved production out. What was the driver behind that?

00:22:09:12 - 00:22:19:16
AJ Kuftic, Field CTO at Expedient
Because that seems like another sort of, you know, kind of tied into this, not spending your time on things that are just keeping the lights on.

00:22:19:18 - 00:22:46:10
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah. So we, I think our, our goal was to actually, again, going back in time where we did drives first and then move that the production data center second. I think there was a bit of reluctance just because of, you know, I guess, you know, from a an executive level of, you know, a making that honestly, though, it was a bit more financial.

00:22:46:12 - 00:23:12:11
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Challenge than, than, than anything. I went through three different times where I presented, you know, hey, let's let's get out of our on prem data center and move to, a co-location facility. And every time it was, you know, shut down only because it was that at a time that we were looking to to, refresh in our compute storage.

00:23:12:12 - 00:23:39:17
Tim Swingler, VP of Infrastructure & Security at Utz Brands
And those costs alone were just not enough to show an ROI for us to say we got to do it. Everyone agreed. Tim, I think that's the right path. We got to do it. But just the the the cost was not there. And, the data center that we have on prem, it came to we started looking at the, the age of the equipment such as that needs to support that.

00:23:39:17 - 00:24:07:07
Tim Swingler, VP of Infrastructure & Security at Utz Brands
The generators, the UPS, fire control systems and all of those things which were, you know, back in 2015 were brand new. But then in you know, 2023, 2024 had their age. Right? So it wasn't until we started factoring in the replacement of all of that infrastructure to go with the the, the refresh of compute and storage is when the ROI made sense for us.

00:24:07:07 - 00:24:13:20
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So that was yeah, the cost was, you know, a driving factor for sure.

00:24:14:00 - 00:24:49:13
AJ Kuftic, Field CTO at Expedient
And I've definitely been in that position where, hey, the the everything about the room seems fine, but that AC unit is not long for this world, and no one wants to be there when it dies. And I've even been in positions where the room that we were hosting all of our compute in was as part of you mentioned, you know, going from being, you know, a publicly traded company or taking on some new compliance regime, now means that you have to have a biometric reader on the store or a mantrap on this door, and adding that was like $200,000 on a room that nobody really wanted to be in.

00:24:49:15 - 00:25:06:00
AJ Kuftic, Field CTO at Expedient
And so is it. Does it make sense to spend a quarter million dollars on a room and nobody really wants to be in, that's where that ROI aligns to say, okay, we're going to do this move or we're going to do this migration. So when you start to look at that, how have we helped on that testing side?

00:25:06:00 - 00:25:22:05
AJ Kuftic, Field CTO at Expedient
Because you mentioned you don't have a concern about failing over. So what does that testing set look like? Because I know for a lot of folks that are probably on this call listening, there's a bigger story to do testing of, you know, the business continuity side. How do we communicate, how do we ensure that those things are happening?

00:25:22:05 - 00:25:31:17
AJ Kuftic, Field CTO at Expedient
But from a technology side, there's a different angle to it. So how does that testing work for you. And and how do you guys keep track of all that?

00:25:31:19 - 00:26:06:16
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah, I think the biggest thing is we know we can deliver consistently when we need to, you know, performing, you know, bubble tests or complete failure, at, the system level or the entire data center. You know, we can we feel confident, now, I wouldn't say we felt comfortable when we had our on prem data center, but now having both under production and or under one, you know, managed solution is that we we trust that that process can work.

00:26:06:17 - 00:26:14:04
AJ Kuftic, Field CTO at Expedient
So have you done a full like, production level failover or is it only been bubble testing.

00:26:14:06 - 00:26:29:19
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So we've only been was it two months now since we fully fully done the that. So we've only done a, a bubble test so far, but we do have one scheduled here and a full failover, test. In the coming months.

00:26:29:20 - 00:26:49:15
AJ Kuftic, Field CTO at Expedient
And to all my Dr. professionals who are on this, I can tell you with great certainty that it is terrifying to a lot of people to know, like we're failing the entire thing over. We're doing effectively a full disaster recovery, failover with the business still running, and a lot of folks don't want to do that because they're afraid of what will happen on the other side.

00:26:49:17 - 00:27:10:09
AJ Kuftic, Field CTO at Expedient
This is why us can do this. The folks on our side, at least I know they did, was a 200 doctor test last year across our customer base of sitting down, going through, validating that everything was there, doing the failover, failing it back. And I think we had one that did not fully pass. And it was because it was an application issue.

00:27:10:11 - 00:27:29:04
AJ Kuftic, Field CTO at Expedient
So we have a ton of confidence in our ability to help do failover. And I think Tim is another one of those products. He's not even part of that 200 he'd have in this year, my last year. But this is the sort of thing where I think from a building that confidence in knowing you can go do that is, I think, huge.

00:27:29:06 - 00:27:31:19
AJ Kuftic, Field CTO at Expedient
So not everything is perfect, though.

00:27:31:21 - 00:27:58:13
Tim Swingler, VP of Infrastructure & Security at Utz Brands
That's a piece that I want to add to that. Oh, go forward in there. In the failover. It's not the perspective of is sort of working or is it not working? We know that that works, right. The challenge or the uncertainty is more so on the networking fail or the networking side of it. Right. Because you know, you have to you know, this this IP subnet is now visible here versus there.

00:27:58:15 - 00:28:23:04
Tim Swingler, VP of Infrastructure & Security at Utz Brands
And in the in our provider we have before we still manage that network side of things. So you know we just had Dr.. With our on prem that's where the main challenge was for us is that we just did not feel we didn't feel comfortable ourselves, that we can make can we can execute that we know we could probably say, you know, you know, route all to this destination and send everything.

00:28:23:06 - 00:28:44:05
Tim Swingler, VP of Infrastructure & Security at Utz Brands
But we never felt like we had it so that we could say, okay, only this subset of systems. Right. And that's the, the luxury. Now with expedient managing not just the server side, but also the network side, that we feel comfortable that they're both on the expedience network. They're both the same side. So that's what makes it a lot more.

00:28:44:07 - 00:28:51:10
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Again, it takes that, that, that trust document. We now have that trust because it's completely owned and that.

00:28:51:12 - 00:29:08:08
AJ Kuftic, Field CTO at Expedient
So I'm going to ask, an audience question here because I think it ties in to where we're talking about with testing. How often do you test and at what scale is the Scout is the schedule based on calendar or infrastructure changes or both? So I feel like this is a good time to ask it just because it's in context.

00:29:08:08 - 00:29:18:02
AJ Kuftic, Field CTO at Expedient
And we've just been talking about testing a whole bunch.

00:29:18:04 - 00:29:25:20
AJ Kuftic, Field CTO at Expedient
So Tim, how often do you guys test or how often are you planning to test?

00:29:25:21 - 00:29:31:08
Tim Swingler, VP of Infrastructure & Security at Utz Brands
If our CIO would like us to test probably every quarter. I, you.

00:29:31:08 - 00:29:34:01
AJ Kuftic, Field CTO at Expedient
Know, I'm like, do a full failover every quarter.

00:29:34:03 - 00:29:52:15
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Now. Well, I think we want to do two failures a year and we'll test, do a bubble test, the other two times. You know, I always have a conversation with him, like, well, yeah, we're a resource challenge. We have other priorities. We've got this right. Or so. You know, we I'm still at odds with that.

00:29:52:19 - 00:29:56:05
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right? In terms of, trying to come up with that cadence.

00:29:56:07 - 00:30:07:04
AJ Kuftic, Field CTO at Expedient
So. Got it. And so would that schedule be based on just a generic calendar, or would you be focusing on, like when an infrastructure level thing changes? I'm guessing it's just more based on just calendar.

00:30:07:06 - 00:30:34:02
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah. It's just that that check and balances thing. Now we do do you know, from a backup perspective, we do test quarterly backups. So, you know, we, we take a percentage of our VMs and, you know, making sure that that, is recoverable. And that's more so the perspective from a, you know, cybersecurity incident. Right. Being able to have assurance that we that we can recover and that that ransomware, for instance.

00:30:34:04 - 00:30:54:17
AJ Kuftic, Field CTO at Expedient
Yeah, I kind of want to dive a little bit into that, because I think a lot of people, when they think of disaster recovery, it's just I have stuff running here, and now I run it over here and in my view, in the view of expedient and and also kind of what you just said, there was the, the disasters that we see most often are not the natural kind.

00:30:54:18 - 00:31:20:08
AJ Kuftic, Field CTO at Expedient
Right. You're talking about your main data center was in the middle of Pennsylvania, not exactly high propensity for major natural disasters. Knock on wood, let's keep it that way. But when we start thinking about disasters like ransomware that just thinking of, I run stuff in production and I fail it over to disaster to my disaster recovery site may not be sufficient because you're just replicating over the ransomware.

00:31:20:10 - 00:31:29:01
AJ Kuftic, Field CTO at Expedient
So how do you how do you fit backup into that overall strategy in terms of longer term versus shorter term recovery?

00:31:29:02 - 00:31:53:20
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah. So I think it comes down to, you know, what your, you know, server backup policy is right. And in our case, you know, we we have this need to retain data for for more than a year. Right. Because ransomware can lay dormant for months at a time. You don't really know when, when that, malware was presented into your environment.

00:31:53:21 - 00:32:18:02
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right. So the so and obviously all of that's going to be replicated from your production side to your drive. So you can't really leverage that, when you're in a, when you're doing real time replication. So, that's why we, you know, we spend a lot more money on storage, for backups. But it's, it's it's just needed.

00:32:18:04 - 00:32:41:12
AJ Kuftic, Field CTO at Expedient
And, and what's interesting about that is we've had customers who have gone through that, and, the ransomware got in at the beginning of August but didn't actually activate until the beginning of October. So it took about two months. So even in all of their Docker replication that they had, it was in there the whole time. So we were able to utilize a combination of their longer term backups and their Dr..

00:32:41:12 - 00:33:01:01
AJ Kuftic, Field CTO at Expedient
To be able to get them back up and running in a very, very short period of time. But that's the big one that we see a lot. And I've started talking about Dr.. As being more than just replication. It needs to be a backup story too. And I'm glad to hear that you also consider the the backup side of that and doing regular recoveries and restores.

00:33:01:02 - 00:33:32:00
AJ Kuftic, Field CTO at Expedient
Everybody should be testing their their restore process. Most people don't because it's a big pain in the butt, and they don't find out if they can do a restore until, well, it's time to actually do a real life restore. And then they find out whether or not it's actually been working this whole time. So doing a regular recovery of or restore of a backup should be considered just as much as your Dr. testing and your Dr. failover strategies, because it's the same level and it's going to become part of that process whether you'd like it to or not.

00:33:32:02 - 00:33:54:11
AJ Kuftic, Field CTO at Expedient
You've mentioned a number of times here that you moved production and doctrine together, and you've mentioned a few times what that means, but when you're starting to look at having both sides together, what does that really done in terms of the you mentioned the networking pieces, but just in terms of communication and operations, of having somebody who understands what's going on on both sides.

00:33:54:13 - 00:34:18:06
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah. I mean, before when we when we only had Dr.. Again, it was an even that period where our production was with expedient, but our Dr. was with another provider, you know, ots it was in the middle of that. Right. There's, you know, even is going down and saying, okay, well, you know, we need to do a stereo upgrade.

00:34:18:08 - 00:34:45:14
Tim Swingler, VP of Infrastructure & Security at Utz Brands
And the Dr. provider needed to do their end, and then we would have to get open and take it with expedience saying, hey, are we on this right person to, you know, coordinate on all of that? And so, you know, that that was, you know, that and that that's a challenge. And then even again, going back to the networking side of things, are we pointing, establishing, you know, VPN connectivity from between really two competitors?

00:34:45:14 - 00:35:09:12
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right. You know, and fortunately, we were it wasn't something I thought about early on, but I was like, well, geez, are these guys to vendors going to play nice in the sandbox with each other? And amazingly, I was I was surprised. I mean, it was it's like a it was very cordial. It was very open and it was it was much easier than I thought it was going to be on both parts.

00:35:09:12 - 00:35:25:07
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right. So again, with the goal is that at the end of the day, one vendor giving us the assurance that it's managed fully and fully in the end is was, is is really, taking a load off of us.

00:35:25:09 - 00:35:46:17
AJ Kuftic, Field CTO at Expedient
Yeah. I think that's a, that's something that we lean into in terms of, you know, we work together. I'll be honest, the provider space that we are in is we all know each other, right. So being a jerk to one another doesn't do anybody any good. So we all actually work really, really well together. And, it's happened in the opposite direction too, so I'll just be honest there.

00:35:46:19 - 00:36:07:09
AJ Kuftic, Field CTO at Expedient
We try our best to work well, because we know that maybe that customer comes back because they're not happy on the other side or, we are trying to get somebody from, you know, we're talking to a customer who's on their side and trying to play in the sandbox. It requires everybody to kind of be kind. And also the end of the day, you're the customer.

00:36:07:11 - 00:36:29:12
AJ Kuftic, Field CTO at Expedient
Regardless of what happens. You're still the customer. It's your data. It's your applications, your workloads. So we can't be jerks about it because now we look like we're messing with you and they look like they're messing with you and nobody's happy. So, it was very nice that everybody played nice in the sandbox, and hopefully we, you know, continue to do that when we start talking about this overall dress concept and having these additional capabilities.

00:36:29:12 - 00:36:47:16
AJ Kuftic, Field CTO at Expedient
How does that help with strategic planning? Because I know the Dr. is not just, you know, a one time thing. You have to constantly plan for it. And constantly consider, you know, what sort of business continuity strategies you have because the business is constantly changing. So how does that help in terms of being able to provide those services for future needs?

00:36:47:16 - 00:36:52:05
AJ Kuftic, Field CTO at Expedient
As new plants open, as new applications come on line?

00:36:52:07 - 00:37:17:10
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah. I mean, I'd say scalability is key here, right? We know that, we have a good failover solution. And then we can we can add grow more as, as we need. Right. And it's, it's a change. It's not. I don't have to go back to management and say, you know, give me another $200,000 to add, you know, terabytes of storage or things like that.

00:37:17:12 - 00:37:39:06
Tim Swingler, VP of Infrastructure & Security at Utz Brands
But I think the other thing is strategic. When I look at the word strategic planning is also keeping in mind that to be cost conscious when it comes to Dr., you know, during this process, you know, we we kind of learned that, you know, we don't necessarily need to replicate every single VM, you know, from, from one side to the other.

00:37:39:07 - 00:37:58:17
Tim Swingler, VP of Infrastructure & Security at Utz Brands
You know, a lot of systems, a lot of people have, you know, your, your dev environment, your test environment, their QA and their production. When you have all environments that are basically copies of each other, do you need to really think about, replicating every single environment over from site, from your production side to your Dr. location?

00:37:58:17 - 00:38:22:14
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So I think we kind of looked at that and we went back to our, our business partners and are really are it developer team, an analytics team, and said, what do you really need to have in the case of, you know, because, you know, this is worst case, right. We don't necessarily need everything over there. But to say if a disaster happened, we could go back in some time and rebuild the QA environment.

00:38:22:14 - 00:38:44:20
Tim Swingler, VP of Infrastructure & Security at Utz Brands
But what do you need is a bare minimum. And the answer came back was, well, I need a dev environment, I need a production environment. There's other things I don't really I you still have a backup of it, meaning that it's not going to be available. It may not be available, you know, for a period of time, but getting it down to just the essentials that you need is, is very helpful.

00:38:45:00 - 00:39:03:11
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Especially for someone who's trying to plan, you know, in a cost wise, if you're coming up short, it's like, okay, what do we need to have? 100% of our VMs migrated over? Perhaps I can cut that cost, you know, modestly by only taking a portion of what's needed.

00:39:03:13 - 00:39:43:20
AJ Kuftic, Field CTO at Expedient
Yeah, we see that a lot in terms of tearing the in terms of tearing off those VMs of my Devin like, testing. QA is in backups, and those backups are replicated. I have them, but dev and prod are the things that I must have in order to keep things going. So we we definitely see a rightsizing as we as we have customers on board, especially because there is a cost for resources, a cost for licensing, etc. so being able to put in what you need, that also allows you to really focus on what's important and rightsize things down and sometimes even see a cost savings out of it because you realized, oh, hey, we've

00:39:43:20 - 00:40:03:03
AJ Kuftic, Field CTO at Expedient
been replicating these like 50 VMs and we didn't really need to, when you can really dial that down, that's where some, some cost savings can come in. And I guess to kind of wrap this up and kind of lead into some into some customer questions. What impact has transitioning two drivers had on your organizations overall to its strategy?

00:40:03:03 - 00:40:11:17
AJ Kuftic, Field CTO at Expedient
Right now you have this capability. You have the product side too. What is that ledger it to really your IT strategy to lean into.

00:40:11:18 - 00:40:39:04
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah. So I think the common theme that I kind of said here is the fact that, you know, getting more aligning work in other areas, right? So we we've actually shifted, our infrastructure engineers to become less sysadmins and more focused in on, and Azure and cloud governance. So, you know, our resources now, you know, we've, we never had it in Azure or cloud governance team.

00:40:39:04 - 00:41:04:10
Tim Swingler, VP of Infrastructure & Security at Utz Brands
It just we didn't, you know, it was it was not an area that was fully mature an organization. So we've seen in my team in particular, a shift from traditional sysadmins to more of the cloud. You know, and we because we are, you know, have a, all our critical systems are all dependent to tie you know, past data between each other.

00:41:04:12 - 00:41:33:19
Tim Swingler, VP of Infrastructure & Security at Utz Brands
We're generally using that as and it's becoming now more of, a priority focus with, than it is to making sure that those integrations are resilient. And so that's really been the shift, I would say, over the last 12 months for us is that, you know, being away from, you know, maintaining VMs to, you know, making sure our, our cloud environments are secure and are fully resilient.

00:41:33:21 - 00:41:56:15
AJ Kuftic, Field CTO at Expedient
So with that, I'd like to open it up for audience questions. We already answered, the one that came in, if you have any questions, please go ahead and put them into the, into the Q&A panel. Have you been able to fail over to cover single or combined application outages that impact critical business processes without failing over the whole thing?

00:41:56:17 - 00:42:02:20
AJ Kuftic, Field CTO at Expedient
And you touched on it a couple times of being able to failover a system or subset of systems. So can you talk more a little bit about that?

00:42:03:00 - 00:42:25:08
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah. You know, that could be a cluster of virtual servers that we can fail over to or just a single one. Right. So yeah. So absolutely. It's it's and again, I think that in the past we just never broached that because the networking component side of things. But you know, absolutely.

00:42:25:10 - 00:42:43:06
AJ Kuftic, Field CTO at Expedient
There was a second question that came in right with that of that mentioned the backup retention policy and definitions. And how do you define your backup policies? Is that based on, you know, compliance regime and retention or how often you schedule those things? How do those work?

00:42:43:08 - 00:43:08:21
Tim Swingler, VP of Infrastructure & Security at Utz Brands
I, you know, it is there are best practices out there. There may be, you know, it's just it's basically we developed it based on what our, you know, what our business needs are, right? There's, depending on the organization, it could vary. But, you know, we basically do a daily, the daily becomes, turns into a weekly backup and then after four weeks, that weekly becomes a monthly.

00:43:08:21 - 00:43:20:06
Tim Swingler, VP of Infrastructure & Security at Utz Brands
And then after the months, 12 months, it becomes a year. So, that's just the, the process that we've can, come up with for, for backups.

00:43:20:07 - 00:43:43:21
AJ Kuftic, Field CTO at Expedient
And I, I heard you mentioned, you know, moving some of your engineers from being, you know, sysadmins into more, you know, cloud admins and cloud security people. One of the questions here is, you know, now that there's more reliance on cloud and hybrid infrastructures, what are you seeing as the big cybersecurity threats? Obviously, from a manufacturing standpoint, you're worried about what is going to impact my production.

00:43:43:21 - 00:43:50:03
AJ Kuftic, Field CTO at Expedient
But what do you see as these cybersecurity threats and what are your what is your team really focused on right now?

00:43:50:05 - 00:44:13:09
Tim Swingler, VP of Infrastructure & Security at Utz Brands
I think in the perspective of cybersecurity, I think the the other onus is the fact that, you know, we don't really talk about is that maintaining servers, making sure that they're patched appropriately, right. And, you know, experience helping us with that, you know, making sure that it's part of the management and monitoring that those are getting applied appropriately.

00:44:13:09 - 00:44:36:01
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So, you know, there's I would say it's not like we've completely gone away from sysadmin work. It's just changed, to some degree, a lot more work on certificates and other things that are still, I call it in the sysadmin space that you, you think, oh, well, I don't have to worry about those things. And, you know, that they're just they're always going to be there.

00:44:36:06 - 00:45:02:04
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right. But yeah, the other thing I think is the fact that, for us is that we, we again, when looking at from a cloud perspective in cybersecurity, I just assume that Microsoft has all of the security controls in place, you know, in our environment. And, about 18 months ago, it was a we did an assessment and found out, whoops.

00:45:02:04 - 00:45:27:09
Tim Swingler, VP of Infrastructure & Security at Utz Brands
We need we need some more work there. Right. So that's, you know, it. It's just that you think Microsoft Natalie, is everything secure in Azure? Well, that's not really the case. So so there's that. Yeah. There's just that added you know need there to, to look at that. And you know, we, we are continuing to do assessments, right?

00:45:27:12 - 00:46:01:10
Tim Swingler, VP of Infrastructure & Security at Utz Brands
I, you know, the last three months we've done three different assessments, one for our insurance provider, one for just our regular pen test. And then, you know, we have vulnerability scanning, and it seems like we're always remediating something. Right. And it's, you know, it just churns and you just get a little bit better. And then we're also reaching out to other organizations and saying that have fell victim to an attack and meeting out with those, those organizations and saying, you know, are there lessons learned that you had that you could share with us?

00:46:01:10 - 00:46:08:15
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right. And we learned a lot along the way, by just learning from what other people have gone through.

00:46:08:17 - 00:46:23:12
AJ Kuftic, Field CTO at Expedient
So that, there was a question here because you mentioned zero. And you mentioned having pure storage. So are you utilizing both or utilizing one or the other? How is how are you handling that replication?

00:46:23:14 - 00:46:44:03
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Yeah. For pure we, we, we talked about it early on when in the early days of Dr.. About replicating that because our the our prior provider to expedient was storing it on pure as well. So it was like well can we can we take advantage of that versus serve. No. And you know, so there was a lot of debate about it.

00:46:44:05 - 00:47:03:20
Tim Swingler, VP of Infrastructure & Security at Utz Brands
I think if, if you have, a tier provider that has the same, you know, storage platform, there may be some things that you could take advantage of there, but we never did. Yeah. So we, we do use the snapshot, you know, the, capabilities within pure. It really helps with our we have some very large SQL databases.

00:47:03:20 - 00:47:13:02
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So being able to do SQL based snapshots is, is very helpful with with the pure platform. There's a question.

00:47:13:02 - 00:47:33:04
AJ Kuftic, Field CTO at Expedient
In here about data syncing and validation and validating that the data from one side got to the other. And you've mentioned, you know, kind of looking at the monitoring. But how are you validating that that all is correct. Is that just happening during the testing process, or is there some other, function that's happening there?

00:47:33:06 - 00:48:01:08
Tim Swingler, VP of Infrastructure & Security at Utz Brands
So I just a on a monthly cadence, we, we go in there and we, we check on things, and the, the metric is basically, you know, or is anything less than, is that anything sinking that's less than, a day, a day old, you know, basically not keeping up, in our environment, we have very large VMs that change databases that are reloaded daily.

00:48:01:08 - 00:48:29:05
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right. So there's some things that take longer to replicate and some that don't. So we're kind of all over the place. But those ones in particular, we keep an eye out on so and just having that and then I think the bigger thing is, like you said, are we replicating 100%? If we spun up a VM last week, you know, did someone actually go in and say, to to remember to to add that to our, to replication with expedient.

00:48:29:05 - 00:48:35:03
Tim Swingler, VP of Infrastructure & Security at Utz Brands
Right. So there's, that's the other half of it.

00:48:35:04 - 00:48:51:13
AJ Kuftic, Field CTO at Expedient
I think I'll wrap up here with one, last question. Are there any final lessons that you've learned through this whole process that kind of really stand out to you? And, and you'd really want to make sure that everybody kind of gets from all this?

00:48:51:15 - 00:49:12:01
Tim Swingler, VP of Infrastructure & Security at Utz Brands
I'd say, you know, I think one of the things that we're still going through is making sure you have a solid plan, not just the technology, but making sure that you have your ducks in or out in a row from a policy perspective, because, you know, there's there's a lot of emphasis we do in terms of like, an incident response plan for cybersecurity.

00:49:12:01 - 00:49:30:21
Tim Swingler, VP of Infrastructure & Security at Utz Brands
But, you know, your Dr. Runbook is almost just as important and making sure that you're not just, you know, taking what expedient gives you from Dr. Runbook perspective, but adding your own needs to it, right, that are pertinent to your own environment.

00:49:31:00 - 00:49:49:02
AJ Kuftic, Field CTO at Expedient
And with that, I would like to thank Tim Angler from Utz Brands for joining me on this wonderful DOJ. Call. If you have any questions, you can feel free to reach out to Bob, to get those questions over to to me or to Tim. And if we happen to see you at the DOJ conference, we'd love to meet you.

00:49:49:03 - 00:49:51:01
AJ Kuftic, Field CTO at Expedient
Thanks for coming.

00:49:51:03 - 00:50:15:07
Bob Arnold, President of DRJ
All right. Wonderful. Thank thanks, everyone, for your attendance. Here today at today's webinar, hope you gained some valuable insights from the session today. Special thanks to Tim and AJ for their time and expertise today, as well as expedient for sponsoring the event. We will be sending a link to the webinar recording, tomorrow, which you can visit as often as you like or share it with your colleagues.

00:50:15:09 - 00:50:22:00
Bob Arnold, President of DRJ
If you have any additional questions, please contact is covered. Journal. Thanks again. And this now concludes today's webinar. Thanks everyone.