RunwayML
Model 1: SPADE-FACE
This model generates realistic images from doodles of faces. I was immediately drawn to it because it seemed like a very difficult task. The model asked for 2 different inputs. That didn’t make much sense to me because I assumed you would only input one doodle. I then thought that maybe the model required an image of a realistic face and a doodle and somehow adapted the styles.
I ran the model but the result was not what I expected at all.
The output looked like something in a modern art museum but no where close to the desired output. It was still really interesting to see as working with faces is always really tricky.
I then tried the same SPADE model but one that is meant for landscapes.
The result was a little bit better, but still not what I expected.
I started researching the SPADE model itself and surprisingly enough in all examples it works amazingly well. I noticed however that the type of input is always very similar.
I then tried again to rerun the model with a similar looking doodle.
As I expected, the model worked a lot better on images similar to the ones it was trained on. I also expect that the same would apply to the faces example.
Model and Data Biography:
According to the project website, the dataset was generated and trained from 40k images scraped from flickr. The model itself is a collaboration between UC Berkeley, NVIDIA, and MIT. I was unsure whether images on flickr are open source or not. I found out that images on flickr require permission from the owner to be used as they are not public domain. The developer of the SPADE model did not provide any further information into the specificities of their data. However, it can be safe to assume they did not ask for 40k permissions. Even though most of the images are landscape images and are pretty harmless. It is still always a worry when a model can get away with something like this.
Experience working with the model:
The faces one was an adaptation to the original model and it wasn’t as easy to work with because I wasn’t totally clear on what to do. The landscaped model was pretty easy. That is all due to runway’s interface however.
Working with RunwayML:
RunwayML makes it really ease to explore different models. There is definitely more options simply because you see your output right away it makes using the model in projects much easier. Since I am taking a CS machine learning class this semester, this is always refreshing because there is very little applications/visualizations in CS classes. It is easy to forget that those numbers have any effect on real life. I feel like ml5.js gave me a bit more flexibility because I had the p5.js editor at my disposal. It was still really fun to explore.
