00:08
JD Wallace: Hey everybody and thank you for joining this pure accelerate digital 2021 session. Today, we are going to show you how to set up a Veeam immutable repository built on top of a couple of exciting new features. The first is SafeMode on flasharray C. And the second is the hardened repository feature
00:29
built into VMs. Brand new v 11. I am JD Wallace. I'm a principal strategist here at Pure Storage. And to help me have this conversation today I am joined by my very good friend, Zane Allen from beam Zane going to introduce yourself. Zane Allyn: Yes, JD said I'm Zane Allen, senior systems
00:46
engineer with beam software. Here in the northwest alongside JD, as you mentioned, a really good friend here remembers from our time at Veeam, as well as your time here at Pier where we talk about our solutions quite a bit right especially even around this topic. So really excited here to, to present for all of
01:05
you today. JD Wallace: Yeah, in as you mentioned, we talked about this, unfortunately, a lot. This is a this is a topic that's very top of mind for everyone. It's actually as we record this, details of the colonial pipeline attack are coming out it is
01:20
dominating the national news. And so this is just something that we see absolutely continuing to pick up speed in our industry. Unfortunately, it doesn't show any signs of declining. Some stats, I have here to say that we can expect a new organization to fall victim every 11 seconds, and that the
01:40
total cost is going to exceed $20 billion this year. And so while it's unfortunate that we keep having these conversations, it is fortunate that we've been working together as two companies to deliver some solutions that can actually help. And so that's what we have come here together to talk about
01:58
a little bit today. So when we talk about this, there's really three things that we tend to hone in on. The first is the simplicity of the solution that gets designed, it's got to be incredibly easy to set up. That's going to breed confidence in knowing that it's going to be there when you need it. It also
02:17
has to be built on top of immutable backups so that when the worst happens, you know, you can get those backups restored. But immutability is kind of an overloaded term. And so we actually say immutability, plus immutability, plus resiliency from deletion is a key part of that. And then finally, speed.
02:35
Once you have those immutable backups, and you're in a position where you can restore and come back online, being able to do that incredibly quickly is key to a successful recovery here, right. In Zane, I think you're gonna get a started on some of the actual solutions that we built together that dig
02:51
into those three pillars, right? Zane Allyn: Absolutely. So yeah, JD said the three three pillars here. simplicity, right? That's first first, first and foremost, I think being in pure as long as I've been a beam, and I've been working with fewer, very similar cultures in that designing
03:09
products that are simple to implement, simple to use, simple to manage, right. And what I mean is don't have to really manage just kind of a set it and you know, it's there for you. So having that intuitive design, right, have these two platforms together as a solution is paramount. On top of that, from
03:26
the software layer, Veeam and our Veeam ONE platform, adding some alerting and some visibility into that environment, so visibility into the core environment, and visibility, even more importantly, into the backup environment. It lets you know, things are there when you need
03:40
it. And when things might be happening in your data center, right, some some alerting to suspicious activity. So you can know you have your backups when you need them for restore. And you can know as soon as possible that you know, some something might be up in your environment. So you can get on top of that,
03:56
and mitigate that, that you know that risk. And then what do we do to ensure help ensure that that data is there and that's the immutability plus, with safe mode. JT mentioned and that's, again, from the hardware and software layer here. We're talking about flash arrays with Safe Mode, having the these this
04:17
data in a mutable state, something that even as you'll see through a demo, somebody with admin access can't change. Right? once it's set, it's it's there and prevalent the environment and no matter what happens, you always have a way to get back from that. But building that again, on top of
04:33
this beam software layer in this solution is coupled with the new feature from beam 11, or version 11, our hardened repo and as you'll see through our demo that's truly a repository with limited access, again, even from the admin of the Veeam software, with true image mutability for the files that are written to
04:56
it. So again, now we're really talking about defense in depth here. immutable repository immutable files limited, you know, least privileged access with that underlying snapshot, with Safe Mode that's keeping your data safe from any being altered, being changed being deleted. So no matter what
05:16
happens in the front end, you know, when we get alerted, you know, your data is there and ready to be restored. And then when we get to restore, it's nothing if we can't bring it back quickly. So ransomware does its job, if it even just slows down a business for a bit of time, that's that's where this
05:30
cost comes from. Right? As long as the longer a business is down, the more that cost that business. So it's all about unmatched speed and fast restores, being offering those instant recovery features, databases, full NAS file, filer restores, instantly VMs, you know, whatever data is
05:48
compromised, you're able to bring it back directly from your backup, right on top of the of the pure hardware, giving that that speed and matching the speed so that you can get that environment back to work quickly. So this is kind of the the intro to the solution. And as we go here, we'll we'll show
06:06
you actually how this works. JD Wallace: And what I love is that, you know, we've really kind of come together for a comprehensive strategy that, you know, doesn't just focus on any one of these pillars, it really focuses on all of them. But today, we're really gonna focus
06:21
for our demo on the second one, this immutability, plus resiliency and how we actually implement that in the environment. And I'm really excited to say that that's actually the end of the slides, we're gonna jump right into demo now, aren't we? That's and I will point out, so I love a good
06:37
live demo. This is recorded for accelerate. And so we we do appreciate you bearing with us as we've got this recorded demo, if we were to do this live and take five days, and I don't think you want a five day session. So we will be adding our commentary on top of the recorded demos that we have
06:54
here. Zane Allyn: That's right. Yeah, real comprehensive demo, right, actually taking you through a whole lifecycle here. So yeah. JD Wallace: So to kick things off, let's get started with building a Windows repository with that safe mode feature from
07:09
from pure, created on top of that. So what we're going to do is we're actually going to start out by creating a brand new flash array volume. Start Here at the hardware. Now, Zane knows how creative I am when it comes to naming convention. So you think I'm gonna give it a cool name, like Jabberwocky, or no,
07:32
I'm just gonna call it windows repo. But that's okay, it's easier to find easier to find right. Now, I've already pre created a host here. And what I'm gonna do is I'm going to go ahead and connect my new volume to that host. I'm using I scuzzy in this demo environment.
07:51
Zane Allyn: Yeah, the important part can use this however, however, your environments set up. JD Wallace: And I'm actually already on the host that I'm going to use as a repository. So I'll bring up the I scuzzy initiator, I'll go ahead and connect to that volume, turn on
08:06
multi pathing, of course. And sure enough, there's that new volume. What's once I have that volume, I need to go ahead and format it and make it available for Veeam. To use. Now, here's the first place where you're gonna see something a little different. If you might have done this historically, when I
08:29
format this volume, I'm actually going to format it with the re Fs file system. And this is pretty exciting, because Veeam has some additional technology that they've built in for re Fs that can actually make it incredibly efficient, particularly when you are building those synthetic full
08:49
backup files. And so we're going to take advantage of that on the flash raid platform here by using ZFS Zane Allyn: is going to use the allocation of 64k. So not quite the default. If you do miss this step Veeam will actually remind you to change it. So not a big deal. Just a little nuance
09:05
there. JD Wallace: That's one of my actually, those little features like that, that make it so easy to use some of my favorite, you just get a little pop up that says, hey, you need to know these exact settings you need to use.
09:16
Zane Allyn: And now that it's all set up, we're gonna go in here and create a repository for us to Veeam it's really the normal way. Under a backup infrastructure, we're going to add a backup repository. Since it's direct attached storage, that's what we're going to choose. This is Windows. So we
09:31
chose windows. We're going to give it another highly creative name here. This time we're going to add flash array because that's what it's on. But it's still that Windows repo. Let me hit next. Since this is attached to the Veeam server, we don't have to add a server. We're just going to populate it and find
09:48
that volume that JT created previously. On the drive. Hit next. It's going to create a folder named there for us. We'll take that we are going to go Under advanced and set the use per machine backup files, we're not going to do a scale out of use to those that would be automated there. But since it
10:07
isn't, we're going to do this just adds to the performance. Take the defaults, hit review and apply. And what this is going to do is take those defaults kind of push out the correct services to that repository. And now we have a Windows repository set up on flasharray. We're gonna go ahead
10:27
and change the configuration backup, it's sitting on the C drive now. So we're Veeam, as, as it does prompts you to, Hey, would you like to best practice, move this to your new repository, we're going to take that. And now we're going to go change the jobs that were set up to point to that new Lee created
10:43
windows repository, defaults there, that's the C drive one, there's the new flash array. So it makes it easy when you name it like this to find it. And if you were going to schedule it, you would hear we're just going to run these manually. hit finish and follow up with the VM backup job.
11:15
Very similar, kind of choose flasharray windows repo, hit next. And run when finish. So that's it created the volume made it told beam where that repository sits, move the backup config file there. One last step, since we did move that backup config file, we're gonna go up here into the what we call
11:38
the hamburger menu to configuration backup, and just go ahead and run. Yep, definitely check that it's on the right repository, and do a backup now. So now all of the changes in Veeam are up to date, it's on that config file. If we were to lose Veeam. Right, now, we could quickly get it back up
11:53
to this state. So backup of the backup system. So let's say we got the new volume created in Veeam, put the backup config on it and just pointed our jobs to that new repository. JD Wallace: So now we're going to head back over to the flash RAID configuration. This is where I'm going to get ready to
12:21
enable safe mode. And so what I'm going to do is first I'm going to create a protection group. And the protection group is really just a policy that allows me to specify how often I want flasharray. To create snapshots, I will once again use my super creative naming convention. This is Veeam safe
12:39
mode for the protection group that's going to let me know the purpose that I created that for. And now I just need to add some members into this protection group, I'm going to go ahead and grab that Windows repository that we've been working with. We'll add that in there. Now that I have that member added, I
12:56
need to specify the schedule for how often I want these snapshots to be created, I'm going to go ahead and create a daily snapshot. And I'll keep those for 14 days for a couple of weeks. I don't need to keep any extra copies beyond that. So I'll set those values to zero. And then of course, don't forget
13:14
to actually turn that schedule on. Zane Allyn: Makes sense. JD Wallace: So I've created that protection group added my my repository volume and set up a schedule. Now I am ready to actually enable safe mode. And this is the first place where
13:30
you'll notice, we don't actually have a demo because this is something that we actually manage on the support side. And so what I'll want to do at this point is I'll log into my my pure one website, I'll go and I'll find this flash array listed in pure one. And you'll notice that there's a new
13:51
section that we just added recently, under assessment, and then safe mode that shows you the status of safe mode on an entire on your entire fleet of systems. So I'll go down and I'll find the system that I've been working with. And I'll notice that its status is currently disabled. Now it's
14:10
running a new one a firmware version that it's eligible to have safe mode turned on. If it weren't weren't. Pure one would actually let me know that they would tell me that, hey, you need to upgrade, provide an upgrade first so that we can make sure that features available and then you can turn
14:24
it off. But since I'm ready to go with the proper firmware, there is an option that I can hover over right there called request Safe Mode, I go ahead and click that button and a scheduler will pop up and it'll go ahead and allow me to schedule with pure support when I can go ahead and turn that
14:41
feature on. Now. Why do we take this out of your hands? Why do we actually do this through support? Well, it's because we want to make sure that if an attacker gets access to those admin credentials, that they can't go and make changes or turn it off. And so that's part of the security of the system is
14:57
they're actually taking the ability to do Change and disable Safe Mode away from the user, even if that user has administrative credentials. So now I've done that. Yeah, go ahead, say, Jeff, something. That's brilliant, right, Zane Allyn: taking it out of the hands on prem. So it's having
15:15
that partnership, you make sure that that's, that's there for you when you need it. JD Wallace: And I think that, again, that's a key part, right, because so many vendors out there, they talk about immutability, but you know, they don't get into details about
15:25
what actually is protected, what actually is immutable. And so understanding that we're, we're essentially doing deletion prevention, you know, through this particular method is really what kind of add to the value of this update, disabled brings to the snapshots. So we're going to do a little time traveling Zayn,
15:46
five, and later, five days later, a couple of things have happened in the in the in the past five days, we have reached out to support we've enabled safe mode on our flash racy, we've had enough time that has passed that we've actually created five beam recovery points, restore points, we have
16:04
some backups that we've created in that repository. And our flash array has created five snapshots through that protection group that we created in those snapshots are protected through that safe mode policy. But hold on, I'm sorry. I'm getting emails from me. I know I'm in the middle of something
16:26
kind of important here. Why? Oh, this is actually from being one. Maybe you can tell me a little bit about what this is. Zane Allyn: Why it looks like some suspicious activity on the data store. Okay, maybe sorry to do this. But maybe we bring up Pete one and take a look at where it came from.
16:45
JD Wallace: Well, let's be quick. We're trying to do a demo here. Flip over to v1 for a second. Zane Allyn: Okay, didn't look too. Too bad. There. You got some memory pressure. Maybe it's just JD Wallace: I just saw something change. What's that possible is
16:59
possible ransomware? activity Zane Allyn: Activity? Okay, let's check out the Yeah, look at the alarm, right wait ads showing the CPU usage and write rate is plus that network transmit rate? It's kind of just suspicious, right? This isn't something normal on your
17:16
network. So we might have to look into this. Really sorry. I know we're in the middle of demo here. But you want this to work, right? JD Wallace: High memory usage too. So I've got multiple, multiple alerts
17:31
Zane Allyn: Multiple indications here. So I know maybe we can take a look at the backups right? Let's go make sure we have that on the on the on the report JD Wallace: You're starting to make me a little nervous, right this is let's just
17:44
Zane Allyn: make sure that we get here so we can get out of stuff. Right? So let's go let's go make sure we have that. up Don't get nervous. JD Hey, go see we got the backup there. It's files. We can do a restore. JD Wallace: Otherwise, yeah, it's trying to restore my files
18:00
matter what happens Zane Allyn: will be okay, right. Oh, that's no good. That's okay, man. Do you cannot JD Wallace: find all requested? It sounds like you can't get to my backup.
18:12
Zane Allyn: I don't know what that is. Yeah, there you go. Let's go look at the files themselves. backups. Right there in the folder that JD Wallace: does not look like Zane Allyn: oh, GP G. Yeah, yeah. I
18:31
JD Wallace: I think all my files are encrypted. Zane Allyn: That's garbage. Yeah, those are the blobs. That's the file. Yeah, I think we got on here. But you're telling me just told everybody that we have snapshots we might be okay. Let's Let's go. Let's
18:53
go to the Yeah, there you go rescan the repository. Let's see if that data doesn't pull back up. I like it. It's synchronizing for probably okay. Oh, removed. Okay. It didn't like it either. JD Wallace: I think my backups are gone.
19:16
Zane Allyn: Yeah, I think you're right, at JD. So what are we going to do? JD Wallace: Well, let's, let's go see if we still have the old maybe half safe mode. Yeah, let's see, first snapshots are still here.
19:27
Zane Allyn: Let's show everybody how that how that works. JD Wallace: So I am going to go into that protection group that we set up five days ago. There is that volume that I've made as a member. There's no snapshots. I should have snapshots here but they appear to be gone by if I look a little bit closer,
19:46
they're actually not gone. They're actually in a destroyed state pending eradication. And you can see I'm actually trying to delete it. And you can see what happened is it actually tells me eradication is disabled. I Unable even with my administrative credentials to come in and delete these, and so
20:05
I'm guessing maybe when we got attacked, somebody tried to get in and delete all my snapshots and weren't able to. Zane Allyn: That's what I was gonna mention. But you set a schedule, right? So what if we change it? JD Wallace: We're protected,
20:19
Zane Allyn: either, okay, so even if you leave a password in clear text somewhere they get into your array, Safe Mode is truly still going to be there, because of that process that you went through with support. JD Wallace: So let's go ahead and take this volume completely
20:33
offline on our windows repository server. We want to make sure there's no IO happening. And once we do that, we can go find this snapshot that, again, is protected with safe mode, so it wasn't able to be deleted, we're going to recover that. So now it's just a regular snapshot again, okay.
20:49
And then once we recover it, we're going to restore that in place, and that's essentially going to completely overwrite that corrupt repository volume. With this point in time copy that the snapshot represents, from a time before the attack occurred,
21:06
Zane Allyn: put it back online, I'm just gonna go right back to looking where it did before. JD Wallace: Now when we browse those files, hey, look at that, that looks like a VM file. Zane Allyn: It's got a VM name in it, it is a VM file, look at
21:20
that, like it never disappeared. That is really amazing. JD Wallace: Double check those data files to same thing looks like we got those back. Great. And because those are just snapshots, that was incredibly fast, right. But let's say I restored one before, that was too early, I could go and
21:36
actually iterate through multiple and try that a couple different times. Zane Allyn: All we have to do here is do a rescan like we did before but she'll be more successful this time since those files are truly Veeam files. Yeah, so quick and easy
21:50
snapshot, returned, rescan to the repository don't have to set up anything new adds to those, those those two backups, and we are back in business. JD Wallace: Alright, here's the true test, true test files to restore.
22:10
Zane Allyn: Fingers crossed. Look at that those files. So this isn't a marketing presentation. But we're going to restore the pure logo here. JD Wallace: Just going to restore to the desktop about that.
22:26
Zane Allyn: Yeah, yeah. So here, here's where they could bring a whole share up online, if you needed to, we could restore retire share, you could just roll it back in time, right, just what was changed if it wasn't fully encrypted. So just showing you how one file so we can prove it. But hopefully, you
22:40
can see the ease of use there. I mean, all the infrastructure is still in place. All the setup was there, all we had to do was bring a snapshot back to Veeam. To look at it again. And everything went back to the way it was. Look at that. JD Wallace: Awesome, saved by safe mode in the simplicity of
23:02
Veeam. And being able to recover that and restore from those files very quickly, I want to highlight a couple things before we move on. And so one of the reasons why we were able to do that is because Veeam has this really clever design in your backup files, these backup files that are created are completely
23:19
self describing, I have everything I need to be able to cover in those files. Now, in the demo that we just showed everyone. This was this was an attack that was fairly limited in scope, it attacked my you know, my production file server, and then my backups, and then also the snapshots on flash
23:38
array of those backups. But it didn't actually go after my Veeam server or my database server. What if the scale had been a little bit more broad? What would we do if they had actually gone after? The more of the Veeam? infrastructure? Zane Allyn: Yeah, that's a great question. Um, as you say, very
23:57
portable file format. It's what I really love back beam as a customer. And what I continue to love about being one of the one of the main things, if it have gone a little further and we lost even even the database or just the database of beam, remember the beginning, we took that config file, we wrote it to
24:12
that array, so that is also sitting on that repository that is also in that snapshot. So we could have quickly deployed a new VM server loaded that config file, all of the settings would have come back because the files are self describing we would have just done the same process for our probably would have
24:30
already had the snapshot presented, scanned it with the new repository and then gone to restoring so a few more steps to get the server up and running. But really, that's it right it's an install and a file load. So JD Wallace: let's take this to the worst case scenario though.
24:44
What if What if I forgot to change that location for my config file and I don't have that would have literally all I have left? Is this snapshot of my repository that safe mode is protected. And all I have are my backups. What do I then? Zane Allyn: It's great question, even without the configuration
25:03
file, and no trying stuff on here, but even without the configuration file, because again, those backups are self describing Veeam would just be in place, you would lose all the history, you would lose all of the the configuration of the repository. But, you know, we saw it in the beginning of the
25:19
demo was pretty easy to set up repository, you could quickly do the same steps pointed at that same snapshot, that same volume, and restore everything as it was anything critical the password, you just need to know that. So those portable file formats and, you know, I'm glad you asked, because I can't stress it enough
25:39
for just our key, right, it's all about keeping the data on something quick and secure. Keeping the Veeam server you know, away from it, and then being able to tie those back, right, if you don't have the config file, you can always get your data back and restore your data center.
25:55
JD Wallace: And this goes back to those things that we talked about. We talked about the first three pillars, they'll be in very simple and even resilient, right? So even if I just have those backup files, the the really simple nature of this architecture means that, you know, even if I forgot to backup
26:10
that config file, even if I lost part of my infrastructure, we're going to do as much as possible to be able to protect all this right. And then the last thing I'll point out is, again, we're not really focusing on speed here we're doing, you know, we're just doing a quick demo. But you know, you talked about
26:28
being able to do instant NAS recovery, right. So if this was an if I wasn't trying to just restore a single file, but actually was trying to restore an entire file share, because of the performance I've got on flasharray. with direct flash and Q LC, I'm going to be able to take advantage of some of
26:45
those advanced features, like instant VM instant as instant database recovery, right? Yeah, you Zane Allyn: could run multiple VMs, pull them right off of that flash array, restore file, share at the same time, and then have VM restore all those while it's running. So absolutely. It's all
27:01
about getting back to work as fast as possible. Right. So identifying the threat early, like we said, having the infrastructure and the solution in place to make sure that my backups are protected and available to me, no matter what the scenario is we just said, and then getting those back in
27:14
the environment quickly and VMs instant recovery on top of pure flash array. Does that very well. Absolutely. My favorite solution. JD Wallace: So hey, let's mix things up a little bit. So we talked a lot about flash, racy, and safe mode. And we showed how
27:32
simple that was to implement. But actually, you've got another new feature that was released in Veeam v 11. Now, wood, is this something that we're gonna do instead of safe mode? No, Zane Allyn: no, it absolutely is, we're talking about a solution here. So we are talking about defense in depth. And this
27:51
is just one more layer that we can add to the same solution. So ultimately, we're going to be doing the same. From a pure perspective, it was tools will show here, we're just going to change the LS for the repository and add some new features that Veeam v 11. supports.
28:06
JD Wallace: Yeah, I you know, I love this. Because, you know, unfortunately, the attackers are very smart. And they're getting smarter all the time. They know to come after our backups and our storage solutions, they understand the tools that we're using in our data center environments. And so having this
28:22
layered approach where it's not this or that, but actually having these two together, I think provides provides kind of an extra layer for protection gives me a lot more confidence in the solutions Zane Allyn: 100% and add some diversity, you know, if you're a
28:35
window shop, a little Linux there, if you have a mix of both, it's just the once they figure out one thing, now they have to get through another door. Right? It's just I think you'd like to say it's the moat and the drawbridge. Right. I think that's your, your analogy. I love that one. So, absolutely.
28:50
So JD Wallace: yeah, let's jump into the demo again. And to get started, it's actually going to be very similar to what we did last time, I'm going to go create another volume, this volume is going to be my repository and once again, super
29:04
creative with my name Linux repo. That way I can distinguish it from the windows repo. Now, again, I already have my Linux hosts set up, I get that in advance. So all I need to do is connect this volume to that Linux server. Nice. Now, last time, I came back later and set up safe mode. But now since I
29:27
already have it up and running, let's just go ahead and add this volume to that protection group that I've already created. So save loads already enabled, I don't have to go turn it off. To do this, I'm just gonna go add a new member to that protected group. Pick this new repository, add it into now this schedule is
29:46
protecting both of those, right? So really some Zane Allyn: add to your already existing setup. That's that's, that's great. Adding to simplicity, the solution. JD Wallace: So, once I have that set up, now I need to go To the host and do some configuration. Now we we've got the video for
30:06
you to watch. But you know, it's not super interesting watching command line go by. So we actually make a little bit easier. We put some slides in here to show you the different commands that we're going to be running. But then we're actually going to kind of talk through them as they go through. So
30:19
we'll start it off by SSH into our Linux repository server that we created. We're using Ubuntu In this scenario, you can use a variety of supported operating systems, we'll go ahead and discover those target I scuzzy portals. In our password, there's those two portals. And then I will log into the
30:40
flasharray, I scuzzy target portals on both controllers. And then once I do that, there's some commands that I'm going to run to make sure that this is automatically connected each time I reboot. And when, of course, I want my repository to can you do work, even if I have to reboot it for any reason?
31:02
It'll ask for my password a few more times as a restart. So services. Zane Allyn: Yeah, they get good, important part there too. Again, simplicity, right? I reboot it, it all comes back. I don't have to go through a big procedure just to restart the machine.
31:17
Yep. JD Wallace: But if it didn't come back, there's ravine one steps in, it's gonna let me know, Zane Allyn: right? That's right. That's right, I got problem here. I can't connect anything go quickly rectify it.
31:28
visibility. Unknown: Alright, so now I have that connection between my Linux server and that volume that I have created, but it's still just raw volume. Let's go ahead and format that. And similar to when we were on Windows, and we use re Fs, we're going to use x
31:48
Fs with Linux. And we're gonna do that for the exact same reason Veeam has built some fast clone technology on top of these file systems, that allows them to be very efficient from a capacity and performance perspective when creating the synthetic full backups. And we absolutely want to be able to
32:08
take advantage of that. And we do support that on flash right now. So we're going to configure that with ZFS. Yep. And that command line is right there from our Help Center. That's the format that will show you here. Otherwise, you just use an LVM to create the volumes.
32:22
Perfect. So let's confirm that we the volume has multiple paths, it does indeed. So let's go ahead and initialize the physical volume, we do that with PV create. Then we will create a volume group from this volume called flasharray. That's VG create. And then we'll create a new logical volume called repo.
32:47
That is Lv create. And then once we do that, we go ahead and run the command to format it with x Fs, and we are ready for masculine. Zane Allyn: Yep, yeah. So it's got the right block size, the RFS with the reps link enabled.
33:05
JD Wallace: Okay, so now I got that volume ready to go, let's go ahead and mount that volume to our file system. So to do that, I'll create a directory to mount it to, I am going to, it's going to call it flash tree repo. And we'll go ahead and mount it. Let's double check that it was
33:29
actually mounted. Sure enough, there it is. Nice. Now I need to get the EU ID. And I'm going to use that to create a new entry in the file system table. Zane Allyn: Again, to make sure that that mount point comes back
33:50
up on a reboot. Yep. JD Wallace: And making this all as resilient as possible. Now, again, I'm using I scuzzy in my environment. So I use the underscore net dev parameter there to make sure that it waits on the network before it tries to establish that connection.
34:09
Tip. All right, and so from a Liferay perspective, that's pretty much it that volumes been created, presented, formatted and mounted. And it's pretty much ready to go. But saying there's a couple of extra things that we need to do to this Linux repository, if we want to prepare it for to be a hardened
34:33
repository, right? Zane Allyn: That's correct. Yeah. And these, again, are going to be Linux commands. These are Linux steps here, but it's just going to prepare us so that we can do that first part of the hardened repo that is limiting, least privilege access
34:46
to the service that's ultimately going to run on this Linux repository. So we're going to create a user give the permissions to a directory and prepare it for the next step. So In this case, I'm going to enable route, we're not going to need this past the install, just on a boon to is not enabled, I'm
35:06
going to use it for the install, you can remove that later harden it, we're going to add a new user, we're going to call it beam service, this is going to be the user that we use in Veeam. To create the repository, we're going to give it a password, fill this information out or not, we're just gonna
35:22
blank through it. Yes, and go ahead and check that beam service password, it's got the home folder, group, we're gonna make a directory called backups on that mount point, that's where we're gonna ultimately going to place the backup files. So we check the permissions here we see all the permissions are
35:45
open. So we're going to change the owner to that beam service. And make it so that only the beam service has access to that folder, and then limiting limiting, again, the least privileged. So theme service has least privileged to the Linux repo. Only beam service is really in charge of that folder.
36:05
So again, just really giving it a specific specific duty on this Linux repository. So basically removing group permissions for others. So at that point, Rob prepared, right, you got your volume, we got it mounted, we got directories, we're making sure it's there to reboot, got a new user that has access to a
36:25
directory, the limited the access to it, but it doesn't have that account does not have higher, higher access to the Linux repository. So that in Veeam, we simply may need to make a backup repositories before this case, it's still going to be direct attached, as you'll see. So we'll go into
36:45
backup infrastructure, add backup repository, very similar, direct attached storage. Except this time, you're probably gonna guessed it, we're gonna choose Linux instead of Windows. JD Wallace: And super complicated stuff saying Zane Allyn: super complicated, we're gonna give it that really
37:02
original name, flasharray Linux repo. But again, now now you see why we do this. Now we know the difference. This Linux server doesn't exist. So we're gonna go ahead and add new here, you could have done that ahead of time, but it's just easier to do it in the same wizard. So we're gonna give it its DNS name, hit
37:18
next. And then this is where things change, when we add, we're going to use the new features single use credentials for hardened repository. This is going to be single use, as you'll see, it's not going to be saved, it's only going to be during this setup, we're going to use that user account beam
37:33
service, give it the password we set. And then we're going to do elevate account privileges automatically. But we're also going to use the SU user, if sudo fails, and put in that root password. Yeah, this is where that is needed for install, not needed past this. We're gonna hit OK. And then next, just can
37:59
check that SSH connection. Of course, we trust it, we just set it up. SSH is used to reach out and install that service, that's going to be this transport service here. After that SSH is no longer needed. That's kind of the first part of the hardened repository and that one time use credentials, that service will
38:15
be running under that route itself will other run the other services to make immutability as we'll see. But again, this is just creating that connection, that beam user service to Veeam. And the repository. So now we have the repository server setup. And we can complete the rest of this configuration, we
38:36
have populate now that we have that server in here, just like we did with Windows, and should be able to find our mount point, because he did it right. There it is. Choose that hit next. As we're going to store it, and we're going to check two things here. In this case, you will check for ZFS. We've already
38:55
formatted it. So it's, it's supported. And then the second part is we're going to make our backups immutable. This is a function of Linux at the end of the day, it's going to use the extended attributes. We're going to do the per machine backup files as we did prior. And that's it right we'll set the
39:12
middle building for the desired number of days, we're going to do seven here and x Fs per machine. Take the defaults next and apply. And that's it right it's maybe a few more steps. I got a check box here in there, but really, really easy to set up. Once you have that Linux machine installed. Tell me more
39:31
it is tell where you want to store the backups using ZFS and immutability. We're using that one time use credential, so that after this setup, no longer is SSH, needed no longer is elevated access needed. And to prove that we're going to go into manage credentials here you'll see VM user does not
39:49
exist, that credential is not in VM. So even if I'm a VM admin, I don't have access to a credential to do anything. And that's it really simple to set up right? Linux, Linux never looked so easy. We're really excited about the future. Let's go in here and edit the backup jobs as we did before and point
40:08
them to the new storage. And again, with those names, easy to find flasharray Linux repo right there. Change the NASS job, hit apply, and we'll run when we're finished. We'll do the same for the VM backup job. Flash array Linux repo. Just as usual, V. Next, next finish.
40:39
Next, next finish. Well, next, next run job then finish. That's it. We had a Windows repository. Now we have a Windows and a Linux repository. we're leveraging, again, defense in depth. Now we're layering a hardened Linux repository on top of safe mode. But still very easy, intuitive to set up. And
41:05
JD Wallace: yeah, so it's great that both of these features are working together. And they were both incredibly simple to set up together, too. So there wasn't a lot of extra complexity involved, to be able to get this this layered approach. But now that we've got it set up saying, Let's, let's go test it out a
41:20
little bit. Zane Allyn: Yeah, let's show you what we get right for adding this layer for sure. JD Wallace: So at this point, we've made a couple backups. Zane Allyn: Yep. We're gonna go and try to delete one. Let's
41:40
just try to pull it out there, right. I'm a VM admin, I should have access to this, you would write it in the VM backup job, it hit Delete. Kind of looks like it's gonna work. Why wouldn't it work? I'm logged in as the admin. But because we said immutability, and says right there, those files are
41:58
separate mutability. That's not controlled by veem that's controlled by the repository and VMs gonna report back that you cannot delete these. JD Wallace: I'm gonna call shenanigans, because you actually wrote that software, you control it, right? So want
42:12
Zane Allyn: to go into the directory itself. And let's we have visibility right into the files. This is pulling up the Linux repository because it has that relationship. And I go right into that backups folder. There's all the Veen backups in front of us, we could just strike one right in the middle
42:25
hit the folder bvk. Do you want to delete and you're gonna be met by this right failed to delete file, it's still immutable. Again, beam doesn't have any jurisdiction over the it doesn't make the mutability. That's the repo, JD Wallace: Okay. I'm gonna SSH into the actual repo server,
42:43
though. Let's take Veeam out of the equation, show me directly through logging into the server, I still can't do it. Zane Allyn: Okay, so yeah, using the Veeam servers, right? Maybe I'm the VM admin, I know that VM service password, I get you this log in, I know how to change your directory, I can list those
42:57
backups. I have access to it, right? I can say we made access to that folder. This is that list attribute part we can see those files that are immutable. That's it that I immutability flag is right there. That's a feature. You're gonna sudo chatter? Sure, why not write and enter the password for your beam
43:20
service. And you're going to be stopped by that, right? And we're gonna read your report it better watch out. You better, you better take care of it now, because you're in it. Oh, remove, we'll just do a remove. Same thing, right? Yes, beam service had access to a folder and exclusive access. But it
43:38
doesn't have access to the immutability flag, right, that's managed by the repository. So even a VM admin might might know those accounts, which they don't have to as we showed in the demo, right, I can set that account up and it doesn't save still isn't able to go directly to the machine if you let them
43:53
have direct access and delete these files. So again, another direct in depth, right. And if we still have safe mode underneath that, you still have that layer if something were to happen. But just slowing down and attacker making more blocks, doors locks in the way, slowing down that attack and making sure
44:14
that you have your backups there to restore when you need it very quickly. JD Wallace: And with that, thank you for everyone for joining us for this demo. If you want to dive deeper, there's some additional resources we'll point you to first of all right here
44:30
on accelerate digital 2021. There's another session we encourage you to attend called protecting your data from ransomware attacks with flash array safe mode. We encourage you to go and give that one of you also Zane you've got a website and a blog with additional details.
44:47
Zane Allyn: I do West Coast it hipster that's me. I have a few Veeam blogs and you can find one on the hardened Linux repository where I concentrate on that that aspect in the VM software but you'll notice built on top of the flash Ray, since we built that together, and it ties right into your blog on yours.
45:05
JD Wallace: Yep, absolutely. And I do a little blogging myself over at jdwallace.com, about both pure and Veeam in the intersection of the two. And I, as you pointed out, I have a post specific to the middle of backups with flash, racy, Safe Mode, going through this stuff and a little bit more depth. So
45:23
again, thank you all for joining us, if you'd like to continue the conversation, Zane. And I would love to continue chatting with you. Our emails are right here, you can reach out to us there or we're both on Twitter. I'm at JD Wallace. Zane Allyn: And I'm Alan Z. So just a little back. Alright. So
45:40
love to hear from you and what you think. So thank you all for joining us. JD Wallace: Thank you everybody for joining us. And thank you, again for joining me on this session. It's been a lot of fun. Enjoy the rest of accelerate.
