Data files of German Decompounder for Apache Lucene / Apache Solr / Elasticsearch

This project was started to offer German decompounding out of box for users
of Apache Lucene, Apache Solr, or Elasticsearch. The problem with the data files is
their license, so be careful when packaging them. Apache Lucene is an Apache v2.0
licensed project, so the data files cannot be shipped together with the distribution.

For decompounding German words, the recommended approach is the following:

• First use a hyphenator to create syllables of the input tokens. Of course this does
  way too much. If we would index syllables the user would match a lot of wrong stuff.
  The hyphenator rules are used in many word processor programs (e.g., Open Office or
  Latex). They are provided here in the format of an XML file for Apache FOPs (Formatting
  Objects Processor, taken from https://sourceforge.net/projects/offo/). Those files
  can be read by Lucene’s HyphenationCompoundWordTokenFilter to do the hyphenation.
  Be sure to use the files of offo-hyphenation v1.2, not newer (2.x) ones (Lucene can’t
  read them)!
• The second step is therefore to take the syllables and form words out of them again.
  The Lucene HyphenationCompoundWordTokenFilter can do this based on a dictionary.
  This project here mainly provides the dictionary to do this (see below). As the
  dictionary solely provides parts of compound words (not the compounds itsself),
  it is important to use the onlyLongestMatch setting of the token filter,
  otherwise you might get wrong decompounding results (especially as the dictionary
  also contains very short words).
• The third step is stemming the full word (the token filter keep the original by
  default) and also its parts. You should use the light German stemmer (not the
  minimal), because the decompounded parts have lots of filler characters that should
  be removed by the stemmer. The minimal stemmer is not able to do this.
  As decompounding is no longer a minimal approach, you may consider to use a separate
  lucene field only using the minimal stemmer but not doing decompounding for
  preferring exact matches in your search.

The dictionary file dictionary-de.txt is developed here and was
created based on the fabulous data by Björn Jacke: https://www.j3e.de/ispell/igerman98/

I used his large and high quality dictionary to make a dictionary file only containing
the parts of German compounds. The dictionary therefore is not large, it contains
about 14,500 tokens, that are commonly used to form compounds. The dictionary does
not contain the compounds, only the parts that are used to create them.
The dictionary was lowercased and the umlauts restored to their UTF-8 representation.

Keep in mind: The files provided here are for new German orthography (since 1998)!

Apache Solr example

Here is a config example for Apache Solr. To use it put the two data files
into the core’s config directory’s lang subfolder. After that you can add the
following definition to your Solr schema:

<!-- German -->
<fieldType name="text_de" class="solr.TextField" positionIncrementGap="100">
  <analyzer> 
    <tokenizer class="solr.StandardTokenizerFactory"></tokenizer>
    <filter class="solr.LowerCaseFilterFactory"></filter>
    <filter class="solr.HyphenationCompoundWordTokenFilterFactory" hyphenator="lang/de_DR.xml"
      dictionary="lang/dictionary-de.txt" onlyLongestMatch="true" minSubwordSize="4"></filter>
    <filter class="solr.GermanNormalizationFilterFactory"></filter>
    <filter class="solr.GermanLightStemFilterFactory"></filter>
  </analyzer>
</fieldType>

Important: Use the analyzer for both indexing and searching!

Elasticsearch example

Here is a config example for Elasticsearch. To use it put the two data files
into the ${ES_HOME}/config/analysis directory of your ES node and add
the following settings to your index. After that you can use the
german_decompound analyzer in your mapping.

"settings": {
  "analysis": {
     "filter": {
        "german_decompounder": {
           "type": "hyphenation_decompounder",
           "word_list_path": "analysis/dictionary-de.txt",
           "hyphenation_patterns_path": "analysis/de_DR.xml",
           "only_longest_match": true,
           "min_subword_size": 4
        },
        "german_stemmer": {
           "type": "stemmer",
           "language": "light_german"
        }
     },
     "analyzer": {
        "german_decompound": {
           "type": "custom",
           "tokenizer": "standard",
           "filter": [
              "lowercase",
              "german_decompounder",
              "german_normalization",
              "german_stemmer"
           ]
        }
     }
  }
}

Important: Use the analyzer for both indexing and searching!

Lucene API example

Custom Analyzer for use with the Apache Lucene API.

Analyzer analyzer = CustomAnalyzer.builder(Paths.get("/path/to/german-decompounder"))
       .withTokenizer(StandardTokenizerFactory.NAME)
       .addTokenFilter(LowerCaseFilterFactory.NAME)
       .addTokenFilter(HyphenationCompoundWordTokenFilterFactory.NAME,
                       "hyphenator", "de_DR.xml",
                       "dictionary", "dictionary-de.txt",
                       "onlyLongestMatch", "true",
                       "minSubwordSize", "4")
       .addTokenFilter(GermanNormalizationFilterFactory.NAME)
       .addTokenFilter(GermanLightStemFilterFactory.NAME)
       .build();

Important: Use the analyzer for both indexing and searching!

Help Out!

If you have suggestions for improving the German dictionary, please send
a pull request, thanks! Be sure to only send “plain words”, no compounds!

License

See NOTICE.txt for more information!