Our client is the largest online real estate presence in Australia. They have millions of residential real estate photos. They found an opportunity to extend their brand into home improvement but needed to sift through all of these photos, find the best photos, categorize them by indoor room or outdoor space, and then tag these photos with color, materials, and features.
The workflow starts with 20,000 photos. The overall process involves first selecting the “best” photos, generally regardless of the subject. Ideally, best is relative to the context of the overall objective - to find the best residential photos, either indoor or outdoor, that correspond to bathrooms, bedrooms, gardens, and pools. However, we found that it works to just have a crowd of workers simply play a game to select what they believe is the best photo in general and then filter the photos afterwards.
We developed a simple game to find the best photos of 20,000 photos, ranking them by a score we develop through the game. A player is given two photos and asked to simply vote on which is the better of the two photos. We provide some criteria to what we consider “good” photos but usually the quality of the photo itself - composition, lighting, and content are what people use to select the best photo.
The photos that pass the quality filter are then categorized and tagged. It is at this point that we allow photos that don’t match the space requirements to be thrown out. Those photos that match a space we’re interested in - interiors or exteriors - are then tagged with styles, features, materials, colors, and other attributes.
We provide help for identifying these different attributes with rollovers showing examples for which we are looking. We have benefitted from workers that have experience in interior design but we’ve found that we can generally get good results by providing the help.
We are unique from other approaches in that we specifically review all work from our workers. We found that this has paid off as we don’t require additional steps such as comparing results to one another and then selecting those tags that get the most “votes”.
We are able to remove select tags if we choose. If we see that there are tags that should have been added but weren’t, we can run the photo back through the tagging process. We can even approve the work but decide not to use the photo anyway. Finally, we can send a message to a worker concerning this specific photo, either admonishing them to do better or complementing them on their work and providing them a bonus.
Our client can regularly poll our service for new photographs through our API. A mix of valid and invalid photos are returned with valid photos tagged with relevant attributes.
Finally, reports are provided on the current state of the system along with weekly reports on the total photos tagged.
One of the challenges in crowdsourcing is handling “scammers” - workers that try to game the system by randomly selecting tags in the hope of just getting by by meeting a minimum number of tags that are inapplicable. We have developed a relationship with many of our workers such that some are “trusted” and have their work automatically approved. We’ve had to take action on specific workers that have tried to game the system by admonishing them with email and in extreme cases outright banning them from doing any work for us.
On the positive side, we have workers that specifically want to be advised as to when work will become available. We believe this is a tribute to our approach.
We’ve learned a lot about taking a large number of photos, sorting, categorizing, and tagging them regularly and at a high volume. We’re exploring alternative workflows that allow us to work in greater bulk and in a directed manner. We continue to explore how to use innate human visual intelligence in a highly distributed, parallel computing environment.
For example, we’re exploring a process where we present over a hundred photos at a time to a trained worker. Knowing objectives for quality, relevance, and volume, the worker can quickly select photos for further tagging. This allows us focus the work and more quickly cull the base of candidate photos. Once in the tagging process, we can also use trained workers to focus on particular attributes, responding to changing needs for photos for different needs.
We look forward to continue developing an enterprise level crowdsourced photo tagging service that regularly delivers quality photos accurately tagged at a high level of responsiveness.