We all have that a special piece of art, whether it be a print from your most recent trip or a piece of fridge art. We all have something in our home we would love to put on display. But finding pieces that pair well, with an item such as this landscape, can be difficult. You might start by going to any art buying website and using their built-in tools to search by color or style, but you are still left with hundreds of options to choose from and none of them were recommended with your exact item in mind. You could also hire an interior designer but that can get expensive. And so, I developed ‘An Eye For Art.
I started by scrapping images from minted.com, an art buying website. I then hand labeled a subset of the images into 6 groups. These groups were selected for two reasons 1) they represented the bulk of images sold on minted.com 2) each group contains similar features. As art can contain multiple features items were grouped based on the most dominate feature element.
I then used the labeled data to retrain the VGG16 model which is a popular image classification model trained on a large database. I started with this model because the architecture is comparatively simple to break down – it is made up of a series of convolution layers to summarize features, pooling layers to down sample the generated feature maps and at the top of the model are dense fully connected layers that categorize the images. Through google colab and using the keras python package I used transfer learning to retrain the top three layers of the model with my database images. To avoid overtraining the model I added dropout layers which forced the model to learn from sparse input and intern allowed me to have more generalizable model – this was done in an effort to accommodate a potential range of user input items. This resulted in an accuracy of 81%. And a F1 score of 0.80, with 1 being perfect precision and recall. As my over-arching goal was to identify similar features and not necessarily the category of art, I found that the accuracy of this model fit my needs.
With one of the top layers from my trained model I was able to pull out a feature array for each piece of art in my database. These arrays were then used to create a feature space so that the differences between images could be quantified as the distance between image features. The three closest items to the user image were selected to represent items that shared similar features.
For the final product we get a gallery wall built around the features identified in your personal piece of art.