Monday, February 04, 2008

New feature: Grouping search results by visual similarity

It's been a busy few weeks for me while I was working on a new feature. Having neglected the blog for way too long I decided that it is time to update you on what I've been working on.

One of the key differences between Behold and other image search engines is Behold's ability to search not only by matching tags, but also by specifying what the results should look like. For this Behold has 28 popular visual filters, e.g. 'looks like picture of a building' or 'picture of a face', and uses computer vision to reorder the image search results accordingly. These filters can be turned on and off by the user during the search.

With a newly introduced feature, Behold goes one step further and automatically suggests these filters after analysing the words in your query. It then shows you what your search results would look like if you applied one of these filters, so you save time on finding the right one. This can be useful when your search results contain the desired subject in a number of different contexts.

For example, when searching for eagle, Behold offers the visual filters animal, face and bird to refine the search further. When such suggestions are available, they are placed in the top right corner of the screen:



How does Behold know which filters to suggest? After carefully studying and visualising the most popular tags and search requests, a set of a few thousand rules was generated that links query words with Behold's visual filters. Extra effort went into ensuring that these rules suggest visual filters that usefully refine most queries.

This makes Behold similar to the Clusty web search engine that automatically groups search results into subcategories based on their summarisations. The key difference, however, is that Behold uses computer vision to group images by visual similarity. This feature is now live and, as always, I appreciate your feedback!

Here is a link to two demo videos of this new feature.