1
00:00:00,590 --> 00:00:03,680
So let's talk about triggering batches.

2
00:00:03,680 --> 00:00:06,640
In this lesson, we are going to jump into Data Factory

3
00:00:06,640 --> 00:00:09,250
and we're going to talk about triggering batches.

4
00:00:09,250 --> 00:00:12,250
Specifically, we're going to look at pipeline triggers

5
00:00:12,250 --> 00:00:13,490
in Data Factory.

6
00:00:13,490 --> 00:00:15,820
We're going to talk about what's possible.

7
00:00:15,820 --> 00:00:17,550
Of course, we're going to jump into the portal

8
00:00:17,550 --> 00:00:20,120
and we're also going to see that in action.

9
00:00:20,120 --> 00:00:21,283
So let's get started.

10
00:00:22,170 --> 00:00:25,910
So first off, configuring a pipeline execution trigger.

11
00:00:25,910 --> 00:00:28,420
There's a couple of different ways that you can do this.

12
00:00:28,420 --> 00:00:31,220
The first way, manual execution.

13
00:00:31,220 --> 00:00:33,520
Literally, just pushing the button.

14
00:00:33,520 --> 00:00:35,570
So this is an on-demand execution.

15
00:00:35,570 --> 00:00:37,010
You go into your pipeline,

16
00:00:37,010 --> 00:00:39,850
you choose the pipeline that you want to trigger,

17
00:00:39,850 --> 00:00:41,190
and you trigger the pipeline.

18
00:00:41,190 --> 00:00:42,650
You can do this a couple of different ways.

19
00:00:42,650 --> 00:00:46,730
You could use .NET or PowerShell or an API or Python.

20
00:00:46,730 --> 00:00:49,690
And of course, you can always use the portal,

21
00:00:49,690 --> 00:00:51,250
which I'll show you how to do that here

22
00:00:51,250 --> 00:00:52,460
in a couple of minutes.

23
00:00:52,460 --> 00:00:55,253
But that's manual execution on demand.

24
00:00:56,240 --> 00:00:59,860
Next up, we can also do trigger execution.

25
00:00:59,860 --> 00:01:02,380
Now, if we trigger a pipeline or set a trigger

26
00:01:02,380 --> 00:01:04,340
for a pipeline, there's a couple of different ways

27
00:01:04,340 --> 00:01:05,860
we can do this.

28
00:01:05,860 --> 00:01:10,110
The most simple way is simply by using a schedule.

29
00:01:10,110 --> 00:01:12,480
Now, if we use a schedule,

30
00:01:12,480 --> 00:01:16,550
it can be recurring or it can be a one-time event

31
00:01:16,550 --> 00:01:19,530
in the future, but it has to be in the future,

32
00:01:19,530 --> 00:01:21,960
no jumping back in time and triggering something

33
00:01:21,960 --> 00:01:23,280
that's already happened.

34
00:01:23,280 --> 00:01:25,073
So just a simple wall clock.

35
00:01:26,270 --> 00:01:28,480
You can also do a tumbling window,

36
00:01:28,480 --> 00:01:32,260
which is quite a bit more complex, and also more useful

37
00:01:32,260 --> 00:01:33,750
in certain cases.

38
00:01:33,750 --> 00:01:35,340
So we've talked about tumbling windows,

39
00:01:35,340 --> 00:01:37,970
and so you should have a good feel for what that is.

40
00:01:37,970 --> 00:01:41,660
But with the tumbling window in Data Factory,

41
00:01:41,660 --> 00:01:43,870
keep in mind that this is fixed.

42
00:01:43,870 --> 00:01:46,400
It's non-overlapping time periods.

43
00:01:46,400 --> 00:01:48,280
It's a contiguous time interval,

44
00:01:48,280 --> 00:01:49,880
which if you remember what that means,

45
00:01:49,880 --> 00:01:53,460
that just means that the windows are adjacent

46
00:01:53,460 --> 00:01:54,820
or next to each other.

47
00:01:54,820 --> 00:01:57,353
So you can see that in the picture on the right.

48
00:01:58,360 --> 00:02:01,610
So you'll often see something like a tumbling window

49
00:02:01,610 --> 00:02:03,110
if we wanted to, for instance,

50
00:02:03,110 --> 00:02:05,930
copy a database into a data lake.

51
00:02:05,930 --> 00:02:08,160
So we could take that database

52
00:02:08,160 --> 00:02:11,110
and we could create a tumbling window trigger

53
00:02:11,110 --> 00:02:15,450
for every 24 hours or every hour if we wanted to.

54
00:02:15,450 --> 00:02:18,720
And the tumbling window could go and it could process

55
00:02:18,720 --> 00:02:20,550
and move all of that data for us.

56
00:02:20,550 --> 00:02:24,230
Using a tumbling window makes that just a little bit easier

57
00:02:24,230 --> 00:02:26,840
when we're defining our starting and end times.

58
00:02:26,840 --> 00:02:28,550
And not only that, we're breaking it up

59
00:02:28,550 --> 00:02:31,160
into those individual slices, which again,

60
00:02:31,160 --> 00:02:33,423
makes it a little bit easier to process.

61
00:02:34,640 --> 00:02:36,473
With tumbling windows,

62
00:02:37,540 --> 00:02:41,200
you can also jump back and do things that happened

63
00:02:41,200 --> 00:02:42,380
in the past.

64
00:02:42,380 --> 00:02:44,740
So if you want to move back in time

65
00:02:44,740 --> 00:02:48,110
and run a tumbling window from last week, moving forward,

66
00:02:48,110 --> 00:02:50,060
you can do that with a tumbling window.

67
00:02:51,420 --> 00:02:54,800
Finally, you can do event-based.

68
00:02:54,800 --> 00:02:58,680
Event-based, shockingly, processes when an event happens.

69
00:02:58,680 --> 00:03:00,450
Hey, what about that?

70
00:03:00,450 --> 00:03:02,730
But with event triggers,

71
00:03:02,730 --> 00:03:06,020
you can only do this right now in blobs.

72
00:03:06,020 --> 00:03:09,170
You can't set triggers outside of a blob.

73
00:03:09,170 --> 00:03:10,910
And so with that being the case,

74
00:03:10,910 --> 00:03:12,820
there's really 3 different types of trigger.

75
00:03:12,820 --> 00:03:15,507
Well, 1, when you create a blob,

76
00:03:15,507 --> 00:03:17,265
2, when you delete a blob,

77
00:03:17,265 --> 00:03:20,520
or 3, when you create or delete a blob.

78
00:03:20,520 --> 00:03:22,390
So kind of like 2 and a half,

79
00:03:22,390 --> 00:03:25,290
but that's how event-based triggers work.

80
00:03:25,290 --> 00:03:28,770
Event-based triggers have a many-to-many relationship,

81
00:03:28,770 --> 00:03:30,260
which means that 1 trigger

82
00:03:30,260 --> 00:03:33,190
can execute a whole bunch of different pipelines,

83
00:03:33,190 --> 00:03:35,300
or 1 pipeline could be executed

84
00:03:35,300 --> 00:03:37,420
by a lot of different triggers.

85
00:03:37,420 --> 00:03:41,880
So those are the different types of triggers with pipelines.

86
00:03:41,880 --> 00:03:44,890
What I want to do now is jump over and show you

87
00:03:44,890 --> 00:03:48,190
in the portal so you can see how this works.

88
00:03:48,190 --> 00:03:53,020
So let's jump over into a Data Factory pipeline.

89
00:03:53,020 --> 00:03:55,760
And I have just created a super simple pipeline

90
00:03:55,760 --> 00:03:57,120
and I've published that.

91
00:03:57,120 --> 00:03:59,210
And I'm going to show you a couple of different ways

92
00:03:59,210 --> 00:04:00,900
to run triggers.

93
00:04:00,900 --> 00:04:04,310
The first is just going up here to Add Trigger.

94
00:04:04,310 --> 00:04:05,373
I can click on it.

95
00:04:06,270 --> 00:04:07,900
Click on Trigger Now,

96
00:04:07,900 --> 00:04:10,220
and it will literally start running my pipeline.

97
00:04:10,220 --> 00:04:11,290
That simple.

98
00:04:11,290 --> 00:04:13,690
That is manual triggers.

99
00:04:13,690 --> 00:04:15,700
Or I can come into New Edit

100
00:04:15,700 --> 00:04:18,130
and I can create a more complex trigger.

101
00:04:18,130 --> 00:04:20,560
And let's just say, I want to do New.

102
00:04:20,560 --> 00:04:23,590
I can name my trigger and describe it, of course.

103
00:04:23,590 --> 00:04:25,440
And then I can choose what I want to do.

104
00:04:25,440 --> 00:04:27,570
Do I want a schedule? Do I want an event?

105
00:04:27,570 --> 00:04:29,410
Do I want a tumbling window?

106
00:04:29,410 --> 00:04:32,440
If I want to schedule, I just simply click on that

107
00:04:32,440 --> 00:04:34,980
and you can see it opens up my little calendar for me

108
00:04:34,980 --> 00:04:37,810
and I can pick one when I want that trigger to run.

109
00:04:37,810 --> 00:04:40,290
And I can create a recurring trigger if I want to as well.

110
00:04:40,290 --> 00:04:41,410
So if I want this to go off,

111
00:04:41,410 --> 00:04:44,780
say, every hour, I could do that,

112
00:04:44,780 --> 00:04:47,050
and it would start my recurring trigger,

113
00:04:47,050 --> 00:04:48,480
and I could of course set an end date

114
00:04:48,480 --> 00:04:49,803
if I wanted to as well.

115
00:04:51,650 --> 00:04:54,270
Tumbling window, very similar.

116
00:04:54,270 --> 00:04:55,520
I set my recurrence.

117
00:04:55,520 --> 00:04:57,400
I can go into my advanced settings here

118
00:04:57,400 --> 00:05:01,210
and I can define my trigger as well.

119
00:05:01,210 --> 00:05:02,270
So I could come in here

120
00:05:02,270 --> 00:05:04,900
and I could define how big my window is, for instance,

121
00:05:04,900 --> 00:05:07,670
or set offsets if I want to.

122
00:05:07,670 --> 00:05:09,290
I can create all of that there. Set up

123
00:05:09,290 --> 00:05:11,020
concurrencies and delays.

124
00:05:11,020 --> 00:05:13,640
And so that's how our tumbling window works.

125
00:05:13,640 --> 00:05:16,650
And then of course, we also have our storage events.

126
00:05:16,650 --> 00:05:19,060
I'm just simply going to define where it is

127
00:05:19,060 --> 00:05:22,360
and I'm going to specify my blob path.

128
00:05:22,360 --> 00:05:25,410
I would create those and my trigger is up

129
00:05:25,410 --> 00:05:26,710
and running at that point.

130
00:05:29,410 --> 00:05:31,900
And so, just to kind of give you what this looks like

131
00:05:31,900 --> 00:05:32,733
on the other side,

132
00:05:32,733 --> 00:05:35,150
let's go ahead and just create a simple schedule.

133
00:05:35,150 --> 00:05:37,480
I'll click on that and we can just leave that alone.

134
00:05:37,480 --> 00:05:39,570
Click on OK.

135
00:05:39,570 --> 00:05:41,610
Click on OK there.

136
00:05:41,610 --> 00:05:45,580
And so now, if I go over into Manage

137
00:05:45,580 --> 00:05:47,250
and I go to Trigger here,

138
00:05:47,250 --> 00:05:49,920
you can see that I have a trigger set up.

139
00:05:49,920 --> 00:05:51,410
And I can click on that.

140
00:05:51,410 --> 00:05:54,440
I can see the code for that trigger if I want to,

141
00:05:54,440 --> 00:05:59,440
or I can stop my trigger or start my trigger. That simple.

142
00:06:00,660 --> 00:06:01,830
And if I click on it, of course,

143
00:06:01,830 --> 00:06:03,210
it's going to pull up the same box.

144
00:06:03,210 --> 00:06:06,710
And so I can do edits to the trigger if I need to do that.

145
00:06:06,710 --> 00:06:10,243
That's pretty much it for triggers in the Azure portal.

146
00:06:11,130 --> 00:06:13,620
Let's go back and just talk briefly about

147
00:06:13,620 --> 00:06:14,840
what we have learned.

148
00:06:14,840 --> 00:06:17,810
First off, manual triggers are triggers

149
00:06:17,810 --> 00:06:19,240
that you create on your own

150
00:06:19,240 --> 00:06:22,740
when you want to run a pipeline.

151
00:06:22,740 --> 00:06:24,750
Autotriggers, 3 different types,

152
00:06:24,750 --> 00:06:26,993
schedule, tumbling, and event.

153
00:06:27,840 --> 00:06:29,250
And for the DP-203,

154
00:06:29,250 --> 00:06:31,640
you really need to just understand what's possible.

155
00:06:31,640 --> 00:06:33,550
So for this, scheduled triggers,

156
00:06:33,550 --> 00:06:35,710
well, you can schedule those in the future.

157
00:06:35,710 --> 00:06:37,740
Tumbling triggers, it's a tumbling window

158
00:06:37,740 --> 00:06:40,300
that you can set from the past, present or future,

159
00:06:40,300 --> 00:06:42,210
and you can create a tumbling window

160
00:06:42,210 --> 00:06:44,780
that's going to process your pipeline

161
00:06:44,780 --> 00:06:47,120
or you can also do an event-based trigger,

162
00:06:47,120 --> 00:06:49,440
which is only blob storage.

163
00:06:49,440 --> 00:06:50,880
If you got those basics down,

164
00:06:50,880 --> 00:06:53,840
you should be pretty good to go for the DP-203.

165
00:06:53,840 --> 00:06:56,360
And with that, hey, we're onto the next lesson.

166
00:06:56,360 --> 00:06:57,310
I'll see you there.