strigi

Strigi is a daemon which uses a very fast and efficient crawler that can index data on your harddrive. It indexes within archives like zip files, and calculates a hash for each file, so it can easily detect duplicates.