After the Gatherer retrieves a document, it passes the document through
a subsystem called Essence [14,13] to extract
indexing information. Essence allows the Gatherer to collect indexing
information easily from a wide variety of information, using different
techniques depending on the type of data and the needs of the particular
corpus being indexed. In a nutshell, Essence can determine the type of
data pointed to by a URL (e.g., PostScript vs. HTML) , ``unravel'' presentation nesting formats (such as
compressed ``tar'' files), select which types of data to index (e.g.,
don't index Audio files), and then apply a type-specific extraction
algorithm (called a summarizer) to the data to generate a content
summary. Users can customize each of these aspects, but often this is
not necessary: Harvest is distributed with a ``stock'' set of type
recognizers, presentation unnesters, candidate selectors, and
summarizers that work well for many applications.
Starting with Harvest Version 1.2 we are also integrating support for summarizers based on outside ``component technologies'' of both a free and a commercial nature.
Below we describe the stock summarizer set, the current components distribution, and how users can customize summarizers to change how they operate and add summarizers for new types of data. If you develop a summarizer (or an interface to a commercial system) that is likely to be useful to other users, please notify us via email at harvest-dvl@cs.colorado.edu so we may include it in our components distribution.