1
00:00:00,500 --> 00:00:01,670
Hey Cloud Gurus, welcome

2
00:00:01,670 --> 00:00:04,133
to Section 3: Data Storage.

3
00:00:06,520 --> 00:00:10,360
You have just come out of the Data Engineering crash course.

4
00:00:10,360 --> 00:00:11,950
Now that Brian has thoroughly prepared you

5
00:00:11,950 --> 00:00:14,430
for the concepts behind data engineering,

6
00:00:14,430 --> 00:00:16,870
we're going to get started with a foundational concept

7
00:00:16,870 --> 00:00:18,343
of data storage.

8
00:00:20,780 --> 00:00:23,310
In this section, we'll be covering the introduction,

9
00:00:23,310 --> 00:00:27,260
which we're doing right now, using Azure Data Lakes,

10
00:00:27,260 --> 00:00:29,390
where we'll get more familiar with what those are

11
00:00:29,390 --> 00:00:31,900
and how they fit into data engineering,

12
00:00:31,900 --> 00:00:33,740
getting the folder structure right,

13
00:00:33,740 --> 00:00:35,470
talking about the different zones,

14
00:00:35,470 --> 00:00:40,470
how we design our folders, understanding file types,

15
00:00:40,570 --> 00:00:42,740
learning about things like Parquet files

16
00:00:42,740 --> 00:00:45,840
and the different options available to us there,

17
00:00:45,840 --> 00:00:48,130
partitioning data, where we divide up

18
00:00:48,130 --> 00:00:51,390
a single database instance into multiple chunks.

19
00:00:51,390 --> 00:00:53,530
From there, we'll go over some best practices

20
00:00:53,530 --> 00:00:57,820
regarding partitioning and continue that on in part 2.

21
00:00:57,820 --> 00:00:59,380
It's a big topic, and so we want

22
00:00:59,380 --> 00:01:02,210
to make sure we give it the appropriate coverage.

23
00:01:02,210 --> 00:01:05,620
After that, we'll continue on to distributing data,

24
00:01:05,620 --> 00:01:08,050
talking about Azure Synapse Analytics

25
00:01:08,050 --> 00:01:10,400
and how it handles its underlying distribution;

26
00:01:11,430 --> 00:01:15,100
archiving data; pruning data to pull back

27
00:01:15,100 --> 00:01:17,983
only the subset we want and eliminate the rest;

28
00:01:18,880 --> 00:01:21,730
compressing data to reduce the size of our database

29
00:01:21,730 --> 00:01:24,380
and gain performance; sharding data,

30
00:01:24,380 --> 00:01:27,570
where we split it out across multiple computers;

31
00:01:27,570 --> 00:01:29,560
implementing data redundancy;

32
00:01:29,560 --> 00:01:31,760
looking at our options for copying data

33
00:01:31,760 --> 00:01:34,880
within our primary and secondary regions;

34
00:01:34,880 --> 00:01:37,540
and finally, wrapping everything up with a review

35
00:01:37,540 --> 00:01:38,763
in the section recap.

36
00:01:41,550 --> 00:01:43,140
In this section, you'll get a chance

37
00:01:43,140 --> 00:01:46,020
to apply your newfound skills in a practical way

38
00:01:46,020 --> 00:01:47,810
using hands-on labs.

39
00:01:47,810 --> 00:01:51,020
Specifically for this section, you'll be provisioning

40
00:01:51,020 --> 00:01:53,703
and configuring Azure Data Lake storage, Gen2.

41
00:01:56,200 --> 00:01:57,750
As you go throughout this section,

42
00:01:57,750 --> 00:02:00,280
I want you to keep 1 theme in mind.

43
00:02:00,280 --> 00:02:02,540
Properly designed storage lays a foundation

44
00:02:02,540 --> 00:02:04,840
for achieving maximum query performance

45
00:02:04,840 --> 00:02:07,040
and data availability.

46
00:02:07,040 --> 00:02:09,660
Even though we'll get a bit into the weeds here and there

47
00:02:09,660 --> 00:02:12,140
talking about infrastructure and formats,

48
00:02:12,140 --> 00:02:14,470
it all comes back to this ultimate goal.

49
00:02:14,470 --> 00:02:17,453
We're trying to increase performance and availability.

50
00:02:18,320 --> 00:02:20,250
And that's it, I'm excited to get started

51
00:02:20,250 --> 00:02:21,300
on this section with you,

52
00:02:21,300 --> 00:02:24,740
learning how to lay that foundation for making things fast.

53
00:02:24,740 --> 00:02:26,750
When you're ready, let's jump into the first video

54
00:02:26,750 --> 00:02:28,790
on using Azure Data Lakes.

55
00:02:28,790 --> 00:02:30,540
I look forward to seeing you there.