All right. 1
So here's the solution to the challenge. 2
So the first thing that we need to do is we need to import some frameworks and we're going to be using CoreML, 3
as well as Vision. 4
Apple's machine learning framework. 5
And the next thing we're going to do is to create a method called detect that takes a single parameter 6
called image. 7
And this is going to be in the format of a CIImage, 8
so a Core Image Image if you remember. 9
So in order to pass our userPickedImage into this method, we're going to have to convert our 10
userPickedImage. 11
So let's create something called ciImage and use it to store the results of our CIImage overload 12
that turns a UIImage into CIImage. 13
And as you can see, Xcode is very unhappy with this because, at the moment, if you have a look at this 14
userPickedImage, its data type is of optional Any? 15
So we have to convert that into the data type that is expected in this method which is a UIImage. 16
When we try to display this image inside the imageView, 17
we downcasted it into a UIImage, so we can actually do this at a higher level by moving this up here. 18
So we're saying that let the userPickedImage be of data type UIImage, and we can wrap this whole thing 19
inside an "if let" just to be ultra safe. 20
All right. So, now we're saying if the user did indeed picked an image and it can be converted into a 21
UIImage, then we're going to call that userPickedImage, and then we're going to convert it into 22
a CIIMAGE. 23
Now, I'm going to use a guard statement to protect the CIImage. 24
So guard let this statement to try and convert our UIImage into a CIImage. 25
But if it fails, then I'm going to throw a fatalError. 26
And the message is going to say, "Cannot convert to CIImage," so that we'll know what's going on. 27
But if it did succeed, then we should be able to call this method detect passing in this newly created 28
CIImage. 29
So let's do that. 30
And just check to make sure that your casing is the same as this one. 31
Alternatively, you could just call it something like converted CIImage, and that will make it a little 32
bit more explicit when you're selecting it here. 33
All right. Cool. 34
So we've let the user pick an image. 35
We've converted it into a CIImage, and then we are passing it into this method called detect. 36
So at this stage, our image is right here and we're going to do some funky stuff with it in order to 37
detect what is the flower that's in the image. 38
So the first thing that we need to do is to use VN CoreML model to create a vision container for 39
our mlmodel. 40
So what is our model called? 41
So let's have a look at this class and you can see it's called FlowerClassifier and it has this variable 42
called model, and that's what we're going to tap into and we're going to create a vision container for 43
it. 44
So let the model = VNCoreMLModel for model, which is called 45
FlowerClassifier().model. 46
And this, of course, is something that can throw an error. 47
So let's market with a "try" and use a "guard" statement to protect the cases where it might fail, in which 48
case, we'll throw a fatalError and we'll say, "Cannot import model." 49
So we've created a vision container for our model. 50
The next thing we need to do is to create a request, so we'll call it request, and it's going to be 51
done using VNCoreMLRequest 52
and it is going to take a model which is our newly created model up there, and it's also going to have a 53
completion handler. So we're going to call the request that comes back, request, and the error, we're going 54
to call error. And we're just going to delete that code placeholder for now. 55
Okay, so we've got a model. We've got our request. 56
We just need to create the handler to process that request. 57
So let's call that handler, handler. 58
And we're going to use VNImageRequestHandler to create it. 59
And it's simply going to take a CIImage as the input which is, of course, this image that got passed 60
into the method. 61
So it's just called image. 62
All right. 63
And then we're going to say händler.perform request and the request is in the format of a array 64
of VNCoreMLRequest, and it is going to equal this request that we created up here. 65
So this line of code can also throw an error. 66
So let's market with a try and enclose it in a "do catch" block, so that we can actually log the error 67
if there was one. 68
All right. 69
Final step is once this request has completed, then our completion handler is going to activate and we're 70
going to get back a request or an error if there was a problem. 71
So we're going to do something with that request, namely, we're going to process it and look for where 72
that image was classified as. 73
So I'm going to call it classification and I'm going to set it to equal the request.results?. first, 74
and this is going to be cast as? VNClassificationObservation. 75
Now, the classification has a property called identifier. 76
And this is a string that describes what the classification was. 77
So if it was a flower, it might be rose, or it might be Barberton daisy, or whatever it may be. But this is 78
what we want to put at the top of our navigation controller. 79
So let's tap into our navigation bar. 80
So self.navigationItem.title and that is going to be set to equal the classification identifier. 81
So let's put that over here. 82
And that's all the code that we need to write in order to use our FlowerClassifer model to detect what 83
flower might be shown on the image that the user took. 84
So let's give it a spin. 85
All right, so here's our app. 86
I'm going to tap the camera button and I'm going to take a photo of our flower and I'm going to hit 87
Use Photo. 88
Now, it's going to process that image and give us the result. 89
So it's telling us that this is a Barberton daisy 90
and, although, I have to say my knowledge of flowers is pretty limited, 91
so this app is really useful for me. 92
But I've Googled what Barberton daisy looks like and I think this is the right one. 93
So there's a couple of UI things that are a little bit niggling. 94
Firstly, why does our image look so odd on screen? 95
Well, remember that we allowed the user to edit the image and that cropped it into a square crop, 96
so a 1x1 scale factor image. 97
Now, our screen is actually a rectangle. 98
So in order for this to look a little bit more decent, you can change the image views content mode to 99
Aspect Fit or Aspect Fill whichever you prefer. 100
And the other thing that is kind of odd is that why is it the Barberton Dezi classification is all in 101
lowercase. 102
Ideally, I would want it to say capital "B" and capital "D" for Barberton Daisy, 103
especially when it's displayed on the nav bar. 104
So the reason is because if you have a look in our flower labels all of these are actually all lowercase. 105
But we have a neat trick in Swift that we can change that. 106
So here where we're setting the navigation title to the classification.identifier string. 107
We can add another dot and we can say "capitalized" 108
and it turns it into a capitalized representation of the string. 109
And now, if you run the app again, you can see that it looks much better. 110
All right. 111
So looks a little bit nicer. 112
And also you can see that the result is now capitallized which looks a lot nicer. 113
All right. 114
Awesome. 115
So we've now managed to make our app use our custom image classification model that was converted from 116
a open-source Caffe model, and we're able to start classifying all the flowers that are around us. 117
So if you haven't coded along, 118
give it a spin and check out this awesome app. 119
But in the next lesson, we're going to make this app even more awesome because we're going to cross-match 120
the result that we got back from our image classification against Wikipedia to bring up a description 121
and a useful snippet of information about the flower, as well as an image of the flower on Wikipedia. 122
So you've got all of that to look forward to. 123
And I'll see you on the next lesson.