Deep Learning Helps Yelp Identify Cover Photos

Engineers at Yelp, the popular crowd-sourced business review site, developed an algorithm that understands and determines which uploaded image should be a restaurant’s cover photo.
In the past, cover photos were chosen by calculating a function based on votes, likes, upload date and image caption – but the system was highly subject to selection bias. Cover photos are clicked much more than others and as a result, once a photo ends up on the business page, it will most likely remain there even though there are more attractive photos uploaded at a later date. Also, people tend to click on blurry photos to understand what the photo is.
Using GPUs on the Amazon cloud with the cuDNN-accelerated Caffe deep learning framework, the engineers trained their model on EXIF data – they found a good proxy for quality was whether a photo was taken by a digital single-lens reflex camera (DSLR). These cameras give photographers more control and people who use DSLR cameras typically have more experience and skill in capturing high quality images.
“Training our model on such photos allows it (model) to learn important photo features and recognize great photos even when they are not taken by a DSLR camera,” mentioned Yelp software engineer Alex M. in a blog post in reference to an image that was taken by an iPhone, but still given a very high score.

The Photo Understanding team believes the quality of cover photos for restaurants have significantly improved as a result of their new scoring algorithm and admits there is more work to be done on the usefulness and relevance of the photos.
Read more >