Wednesday, November 29, 2006

Why focus on the web?

Some of my friends and colleagues have asked me why, in this age of photo-sharing websites such as Flickr, the focus Behold's research are images from the web. There is no doubt that Flickr is a very exciting platform for developing image search engines of all kinds, Retrievr being one excellent example. Retieving images from the web, however, can pose two additional challenges. On average, the pictures tend to be of pretty bad quality (for example see some random images in Behold). And, importantly, website owners are not incentivised to tag their photos properly. The first problem forces one to explicitly model junk images likely to be irrelevant for any query, which is an interesting research problem in itself. The second shortcoming can be compensated by looking at the interplay between image content and html metadata (a feature that was recently added into Behold), and an interesting question is how to find the right way to combine the two. That being said, there is no reason why we shouldn't apply Behold to images from Flickr one day.

Saturday, November 25, 2006

New feature!

You can now search in Behold using both the regular metadata a la traditional image search engines like Yahoo and the automatic image annotations. Here's how it works: you enter a text query into the traditional text search: box and then pick one or more keywords from the visual vocabulary to refine your search in refine with image analysis:. A few encouraging examples:

#1. Search for beach using only metadata:

then refine with the visual keyword beach:


#2. Search for bus using only metadata:

refine with the visual keyword car:


This can be useful when you want only a certain type of pictures from a given location, e.g. london buldings or parisian towers. It can also be helpful when metadata for a particular word is poor and you would like to filter your results, e.g. metadata lake vs. combined lake and metadata car vs. combined car.

Essentially this new feature lets you see how automatically assigning annotations to images based on their contents can improve standard text-based image search on the web.

Give it a go!

And while we are at it here's a link to a short report outlining how Behold indexes images and summarising the search engine's current features.

Thursday, November 23, 2006

Automatic image annotation

This Wikipedia aricle is a useful literature primer for anyone interested in automatic image annotation. But did you know that one of the first papers describing an attempt to recognise image content from a statistical perspective dates back to 1974? This was recently pointed out to me by Zdenek Zdrahal during my visit to Open University's Knowledge Media Institute lab. The paper is called A Semantics Based Region Analyzer by Jerome A. Feldman and Yoram Yakimovsky, in Journal of Artificial Intelligence 5(4):349-371 1974. It's interesting what they managed to achieve with the very limited computational resources they had back then.

Monday, November 20, 2006

What can Behold recognise?

After some months being spent on improving the search quality a new version of Behold has been rolled out. The set of keywords that it recognises has somewhat shrunk but hopefully the ones that remain should work better. As well as querying by keywords it is now possible to browse images based on their content, using the algorithm of Daniel Heesch. As this page explains, Behold uses very simple techniques to assign keywords to images so each individual keyword search can give rather vague results. However combining related keywords in your query often helps, e.g. boats vs. boats and water and car vs. car and car_park. Once you have located a close enough image to your target using keywords you can start browsing the database for more similar images. What can you find in Behold by combining keywords and browsing?