0
1
00:00:01,020 --> 00:00:01,320
All right.
1

2
00:00:01,320 --> 00:00:03,130
So here's the solution to the challenge.
2

3
00:00:03,240 --> 00:00:09,320
So the first thing that we need to do is we need to import some frameworks and we're going to be using CoreML,
3

4
00:00:09,360 --> 00:00:11,630
as well as Vision.
4

5
00:00:11,670 --> 00:00:13,330
Apple's machine learning framework.
5

6
00:00:13,350 --> 00:00:19,980
And the next thing we're going to do is to create a method called detect that takes a single parameter
6

7
00:00:20,070 --> 00:00:21,610
called image.
7

8
00:00:22,020 --> 00:00:25,810
And this is going to be in the format of a CIImage,
8

9
00:00:25,830 --> 00:00:28,070
so a Core Image Image if you remember.
9

10
00:00:28,080 --> 00:00:34,260
So in order to pass our userPickedImage into this method, we're going to have to convert our
10

11
00:00:34,350 --> 00:00:34,780
userPickedImage.
11

12
00:00:34,800 --> 00:00:40,500
So let's create something called ciImage and use it to store the results of our CIImage overload
12

13
00:00:40,590 --> 00:00:43,770
that turns a UIImage into CIImage.
13

14
00:00:43,890 --> 00:00:49,230
And as you can see, Xcode is very unhappy with this because, at the moment, if you have a look at this
14

15
00:00:49,230 --> 00:00:52,880
userPickedImage, its data type is of optional Any?
15

16
00:00:53,100 --> 00:00:59,530
So we have to convert that into the data type that is expected in this method which is a UIImage.
16

17
00:00:59,670 --> 00:01:02,800
When we try to display this image inside the imageView,
17

18
00:01:02,880 --> 00:01:09,530
we downcasted it into a UIImage, so we can actually do this at a higher level by moving this up here.
18

19
00:01:09,690 --> 00:01:14,970
So we're saying that let the userPickedImage be of data type UIImage, and we can wrap this whole thing
19

20
00:01:15,000 --> 00:01:18,180
inside an "if let" just to be ultra safe.
20

21
00:01:18,180 --> 00:01:25,190
All right. So, now we're saying if the user did indeed picked an image and it can be converted into a 
21

22
00:01:25,200 --> 00:01:30,960
UIImage, then we're going to call that userPickedImage, and then we're going to convert it into
22

23
00:01:30,960 --> 00:01:31,840
a CIIMAGE.
23

24
00:01:32,160 --> 00:01:36,690
Now, I'm going to use a guard statement to protect the CIImage.
24

25
00:01:36,690 --> 00:01:41,860
So guard let this statement to try and convert our UIImage into a CIImage.
25

26
00:01:41,970 --> 00:01:45,670
But if it fails, then I'm going to throw a fatalError.
26

27
00:01:46,410 --> 00:01:53,580
And the message is going to say, "Cannot convert to CIImage," so that we'll know what's going on.
27

28
00:01:53,610 --> 00:02:00,360
But if it did succeed, then we should be able to call this method detect passing in this newly created
28

29
00:02:00,360 --> 00:02:01,260
CIImage.
29

30
00:02:01,290 --> 00:02:02,370
So let's do that.
30

31
00:02:06,580 --> 00:02:11,010
And just check to make sure that your casing is the same as this one.
31

32
00:02:11,200 --> 00:02:18,890
Alternatively, you could just call it something like converted CIImage, and that will make it a little
32

33
00:02:18,890 --> 00:02:21,540
bit more explicit when you're selecting it here.
33

34
00:02:21,890 --> 00:02:22,490
All right. Cool.
34

35
00:02:22,490 --> 00:02:25,400
So we've let the user pick an image.
35

36
00:02:25,400 --> 00:02:32,280
We've converted it into a CIImage, and then we are passing it into this method called detect.
36

37
00:02:32,540 --> 00:02:37,550
So at this stage, our image is right here and we're going to do some funky stuff with it in order to
37

38
00:02:37,560 --> 00:02:40,460
detect what is the flower that's in the image.
38

39
00:02:40,460 --> 00:02:46,370
So the first thing that we need to do is to use VN CoreML model to create a vision container for 
39

40
00:02:46,370 --> 00:02:47,890
our mlmodel.
40

41
00:02:47,900 --> 00:02:50,110
So what is our model called?
41

42
00:02:50,130 --> 00:02:55,560
So let's have a look at this class and you can see it's called FlowerClassifier and it has this variable
42

43
00:02:55,590 --> 00:03:00,200
called model, and that's what we're going to tap into and we're going to create a vision container for
43

44
00:03:00,200 --> 00:03:00,590
it.
44

45
00:03:00,590 --> 00:03:11,350
So let the model = VNCoreMLModel for model, which is called
45

46
00:03:11,480 --> 00:03:15,240
FlowerClassifier().model.
46

47
00:03:15,260 --> 00:03:18,600
And this, of course, is something that can throw an error.
47

48
00:03:18,830 --> 00:03:26,900
So let's market with a "try" and use a "guard" statement to protect the cases where it might fail, in which
48

49
00:03:26,900 --> 00:03:32,410
case, we'll throw a fatalError and we'll say, "Cannot import model."
49

50
00:03:32,540 --> 00:03:34,850
So we've created a vision container for our model.
50

51
00:03:34,850 --> 00:03:41,270
The next thing we need to do is to create a request, so we'll call it request, and it's going to be
51

52
00:03:41,270 --> 00:03:44,350
done using VNCoreMLRequest
52

53
00:03:49,010 --> 00:03:57,710
and it is going to take a model which is our newly created model up there, and it's also going to have a
53

54
00:03:57,710 --> 00:04:03,650
completion handler. So we're going to call the request that comes back, request, and the error, we're going
54

55
00:04:03,650 --> 00:04:08,260
to call error. And we're just going to delete that code placeholder for now.
55

56
00:04:08,540 --> 00:04:10,880
Okay, so we've got a model. We've got our request.
56

57
00:04:10,910 --> 00:04:14,290
We just need to create the handler to process that request.
57

58
00:04:14,330 --> 00:04:16,650
So let's call that handler, handler.
58

59
00:04:17,150 --> 00:04:23,140
And we're going to use VNImageRequestHandler to create it.
59

60
00:04:23,270 --> 00:04:31,520
And it's simply going to take a CIImage as the input which is, of course, this image that got passed
60

61
00:04:31,520 --> 00:04:32,280
into the method.
61

62
00:04:32,300 --> 00:04:33,950
So it's just called image.
62

63
00:04:33,960 --> 00:04:34,490
All right.
63

64
00:04:34,700 --> 00:04:43,820
And then we're going to say händler.perform request and the request is in the format of a array
64

65
00:04:44,060 --> 00:04:51,050
of VNCoreMLRequest, and it is going to equal this request that we created up here.
65

66
00:04:51,200 --> 00:04:53,510
So this line of code can also throw an error.
66

67
00:04:53,690 --> 00:05:00,680
So let's market with a try and enclose it in a "do catch" block, so that we can actually log the error
67

68
00:05:00,770 --> 00:05:06,580
if there was one.
68

69
00:05:06,580 --> 00:05:07,440
All right.
69

70
00:05:07,570 --> 00:05:16,780
Final step is once this request has completed, then our completion handler is going to activate and we're
70

71
00:05:16,780 --> 00:05:20,460
going to get back a request or an error if there was a problem.
71

72
00:05:20,620 --> 00:05:26,130
So we're going to do something with that request, namely, we're going to process it and look for where
72

73
00:05:26,130 --> 00:05:28,450
that image was classified as.
73

74
00:05:28,460 --> 00:05:37,360
So I'm going to call it classification and I'm going to set it to equal the request.results?. first,
74

75
00:05:37,780 --> 00:05:45,370
and this is going to be cast as?  VNClassificationObservation.
75

76
00:05:45,400 --> 00:05:50,070
Now, the classification has a property called identifier.
76

77
00:05:50,140 --> 00:05:54,820
And this is a string that describes what the classification was.
77

78
00:05:54,820 --> 00:06:01,480
So if it was a flower, it might be rose, or it might be Barberton daisy, or whatever it may be. But this is
78

79
00:06:01,480 --> 00:06:05,350
what we want to put at the top of our navigation controller.
79

80
00:06:05,590 --> 00:06:08,570
So let's tap into our navigation bar.
80

81
00:06:08,740 --> 00:06:19,570
So self.navigationItem.title and that is going to be set to equal the classification identifier.
81

82
00:06:19,570 --> 00:06:21,750
So let's put that over here.
82

83
00:06:23,340 --> 00:06:29,280
And that's all the code that we need to write in order to use our FlowerClassifer model to detect what
83

84
00:06:29,280 --> 00:06:32,610
flower might be shown on the image that the user took.
84

85
00:06:32,610 --> 00:06:34,020
So let's give it a spin.
85

86
00:06:34,030 --> 00:06:35,110
All right, so here's our app.
86

87
00:06:35,130 --> 00:06:41,850
I'm going to tap the camera button and I'm going to take a photo of our flower and I'm going to hit
87

88
00:06:41,910 --> 00:06:43,110
Use Photo.
88

89
00:06:43,230 --> 00:06:47,100
Now, it's going to process that image and give us the result.
89

90
00:06:47,100 --> 00:06:50,340
So it's telling us that this is a Barberton daisy
90

91
00:06:50,490 --> 00:06:54,000
and, although, I have to say my knowledge of flowers is pretty limited,
91

92
00:06:54,090 --> 00:06:56,020
so this app is really useful for me.
92

93
00:06:56,130 --> 00:07:01,590
But I've Googled what Barberton daisy looks like and I think this is the right one.
93

94
00:07:01,680 --> 00:07:04,620
So there's a couple of UI things that are a little bit niggling.
94

95
00:07:04,620 --> 00:07:08,090
Firstly, why does our image look so odd on screen?
95

96
00:07:08,310 --> 00:07:14,020
Well, remember that we allowed the user to edit the image and that cropped it into a square crop,
96

97
00:07:14,070 --> 00:07:16,440
so a 1x1 scale factor image.
97

98
00:07:16,440 --> 00:07:18,890
Now, our screen is actually a rectangle.
98

99
00:07:18,900 --> 00:07:25,620
So in order for this to look a little bit more decent, you can change the image views content mode to
99

100
00:07:25,680 --> 00:07:28,980
Aspect Fit or Aspect Fill whichever you prefer.
100

101
00:07:29,300 --> 00:07:36,420
And the other thing that is kind of odd is that why is it the Barberton Dezi classification is all in
101

102
00:07:36,450 --> 00:07:37,300
lowercase.
102

103
00:07:37,410 --> 00:07:41,770
Ideally, I would want it to say capital "B" and capital "D" for Barberton Daisy,
103

104
00:07:41,820 --> 00:07:44,110
especially when it's displayed on the nav bar.
104

105
00:07:44,190 --> 00:07:50,070
So the reason is because if you have a look in our flower labels all of these are actually all lowercase.
105

106
00:07:50,220 --> 00:07:52,790
But we have a neat trick in Swift that we can change that.
106

107
00:07:52,830 --> 00:07:58,140
So here where we're setting the navigation title to the classification.identifier string.
107

108
00:07:58,140 --> 00:08:01,050
We can add another dot and we can say "capitalized"
108

109
00:08:01,050 --> 00:08:04,870
and it turns it into a capitalized representation of the string.
109

110
00:08:05,280 --> 00:08:15,630
And now, if you run the app again, you can see that it looks much better.
110

111
00:08:15,650 --> 00:08:15,940
All right.
111

112
00:08:15,950 --> 00:08:17,330
So looks a little bit nicer.
112

113
00:08:17,330 --> 00:08:22,640
And also you can see that the result is now capitallized which looks a lot nicer.
113

114
00:08:22,640 --> 00:08:23,020
All right.
114

115
00:08:23,030 --> 00:08:23,480
Awesome.
115

116
00:08:23,480 --> 00:08:29,210
So we've now managed to make our app use our custom image classification model that was converted from
116

117
00:08:29,300 --> 00:08:34,930
a open-source Caffe model, and we're able to start classifying all the flowers that are around us.
117

118
00:08:34,940 --> 00:08:36,320
So if you haven't coded along,
118

119
00:08:36,330 --> 00:08:39,640
give it a spin and check out this awesome app.
119

120
00:08:39,770 --> 00:08:44,570
But in the next lesson, we're going to make this app even more awesome because we're going to cross-match
120

121
00:08:44,750 --> 00:08:51,020
the result that we got back from our image classification against Wikipedia to bring up a description
121

122
00:08:51,140 --> 00:08:56,690
and a useful snippet of information about the flower, as well as an image of the flower on Wikipedia.
122

123
00:08:56,690 --> 00:08:58,570
So you've got all of that to look forward to.
123

124
00:08:58,640 --> 00:09:00,410
And I'll see you on the next lesson.