All DKPro Core files are stored using UNIX line endings. If you develop on Windows, you have to set the core.autocrlf configuration setting to input to avoid accidentally submitting Windows line endings to the repository. Using input is a good strategy in most cases, thus you should consider setting this as a global (add --global) or even as a system (--system) setting.

After changing this setting, best do a fresh clone and check-out of DKPro Core.

The name is the first thing to consider when creating a new module.

Although the modules are technically all the same, in the naming scheme, we discern between the following types of modules:

In addition to the four categories, there are a view unique modules which do not fall into any of these categories, e.g. de.tudarmstadt.ukp.dkpro.core.testing-asl or de.tudarmstadt.ukp.dkpro.core.doc-asl.

The naming scheme applies in several occasions:

Once you have decided on a name for the new module, you proceed by creating a new module folder. Module folders are created directly under the root of the DKPro Core source tree.

Although the folder structure of DKPro Core appears as if there would be a flat list of modules, there is actually a shallow hierarchy of modules (i.e. the folder hierarchy does not correspond to the Maven module hierarchy).

The DKPro Core Parent POM is at the root of the module and of the folder hierarchy. Its parent is the DKPro Parent POM which is maintained in a separate source tree and which follows its own release cycle. The DKPro Parent POM defines a set of default settings, profiles, and managed dependencies useful for all DKPro projects. The DKPro Core Parent POM defines settings specific to DKPro Core.

New modules are added either in the <modules> section of the DKPro Core ASL Parent POM or of the DKPro Core GPL Parent POM depending on whether the new module can be licensed under the Apache License or whether it has to be licensed under the GPL due to a GPL dependency. The these two parent POMs configure different sets of license checkers. For ASL modules, the Apache RAT Maven Plugin is used; for the GPL modules, the License Maven Plugin is used.

Note that the <modules> section in these POMs points to the folders which contain the respective modules. Since the folder hierarchy is flat (unlike the module hierarchy), the module names here need to be prefixed with ../.

In addition to adding a new module to the <modules> section of the respective parent POM, it also needs to be added to the <dependencyManagement> section with this POM:

Next, you create a basic POM inside your new module folder. Below is an example of a minimal POM for a new Apache-licensed component module. If you create a GPL-licensed module instead, replace the -asl suffixes with -gpl and copy the license header from another GPLed module.

In order to avoid unpleasant surprises, DKPro Core uses the Maven Dependency Plugin to check if all dependencies used directly within the code of a module are also explicitly declared in the module POM. If this is not the case, the automated builds fail (they run with -DfailOnWarning). This means, you have to declare dependencies for all libraries that you are using directly from your code in the <dependencies> section. If a dependency is only required during testing, it must be marked with <scope>test</scope>. Below, you find a few typical libraries used in many modules. Note that there is no version defined for these dependencies. The versions for many libraries used by multiple modules in DKPro Core are defined in the DKPro Core Parent POM. Only libraries that are specific to a particular module, e.g. the specific NLP library wrapped, should have their versions defined within the module POM.

You may notice the foo.version property in the minimal POM example above. This property should be used to set the version of wrapped NLP library. It should appear in the name of the module as well as in the specific dependency for the wrapped library.

When you package models for your new component, they need a special treatment in the POM. First, although it is a good idea to create unit tests based on the models, most often you do not want to download all models and run all unit tests during a normal developer build (some models are very large any may quickly fill up your hard disk). Second, the Maven Dependency Plugin is unable to detect that your code or tests make use of the models and it needs to be configured in a special way to allow the build to pass even though it considers the model dependencies as unnecessary.

So assuming you have a model for your component, then first add it to the <dependencyManagemen> section of the POM - here you specify the version but not the scope. All models you have get added to this section, irrespective of whether you want to use them for testing or not.

If you also want to use the model for testing, then you add it also to the <dependencies> section of the POM. Here you specify the scope but not the version. Then you also have to configure the Maven Dependency Plugin to accept the presence of the dependency.

As said before, if you have many models for your component, it is a good idea to use only a small set for regular testing. If you want to create tests for additional models or even for all of your models, then it is best to add the dependencies for these under a profile called use-full-resources. This profile is enabled for automated builds or can be enabled on demand by developers who wish to run all tests. In the example below, we add an additional test dependency on a German model if the profile use-full-resources is enabled. Note that the Maven Dependency Plugin is also again configured within the profile and that the combine.children="append" parameter is used to merge the configuration with the one already present for the default build.

To conditionally run a test only if the required model is enabled, you can use the AssumeResource class from the DKPro Core testing module.

Every module must contain a file called LICENSE.txt at its root which contains the license text. Copy this file from another Apache-licensed or GPL-licensed module (again check if you need to use GPLv2 or v3). If this file is not present, the build will fail.

If the module contains code or resources from a third party (e.g. a source or test file which you copied from some other code repository or obtained from some website), then you need to add a file called NOTICE.txt next to the LICENSE.txt file. For every third-party file (or set of files if mutiple files were obtained from the same source under the same conditions), the NOTICE.txt must contain a statement which allows to identify the files, identify from where these files were obtained, and contain a copyright and license statement. Check the license of the original files for whether you have to include the full license text and potentially some specific attribution (possibly from an upstream NOTICE file).

There are a couple of things useful in every unit test:

To avoid repeating a longish setup boilerplate code in every unit test, add the following lines to your unit test class:

Additional benefits you get from this testContext are:

A typical unit test class has consists of two parts

In the following example, mind that the text must be provided with spaces separating the tokens (thus there must be a space before the full stop at the end of the sentence) and with newline characters (\n) separating the sentences:

Test cases for segmenter components should not make use of the TestRunner class, which already performs tokenization and sentence splitting internally.

The AssertAnnotations class offers various static methods to test if a component has properly created annotations of a certain kind. There are methods to test almost every kind of annotation supported by DKPro Core, e.g.:

The ReaderAssert and WriterAssert classes can be used to text I/O components. They allow building AssertJ-style unit tests with DKPro Core reader and writer components.

One of the simplest tests is a round-trip test where an input file is read using a reader for a particular format, then written out again using a writer for the same format.

The reader is set up to reader the test input file. Instead of setting PARAM_SOURCE_LOCATION, it is also possible to set the input location using readingFrom(). The writer automatically makes use of a test output folder provided by a DkproTestContext - therefore a target location does not need to be configured explicitly.

Assuming the writer produces only a single output file, this file can be accessed for assertions using outputAsString(). If multiple output files are created, an argument can be passed to that method, e.g. outputAsString("output.txt"). This will look for a at the target location whose name ends in output.txt. If there is none or more than one matching file, the test will fail.

If the original input file is in a different format or cannot be fully reproduced by the writer, then it is easy to set up a one way test, simply by changing the final comparison. The following example also shows how to specify additional parameters on the reader or writer.

In order to test the ability of readers to read multiple files, the asJCasList() method can be used. While pipelines typically re-use a single CAS which is repeatedly reset and refilled, this method generates a list of separate CAS instances which can be individually validated after the test. To access elements of the list use element(n).

To add a new type, first locate the relevant module. Typically types are added to an API module because types are supposed to be independent of individual analysis tools. In rare circumstances, a type may be added to an I/O or tool module, e.g. because the type is experimental and needs to be tested in the context of that module - or because the type is highly specific to that module.

Typically, there is only a single descriptor file called dkpro-types.xml. Within a module, we keep this descriptor in the folder src/main/resources under the top-level package plus type name of the module. E.g. for the module

the type descriptor would be

If there is no suitable type descriptor file yet, create a new one.

When a new type system descriptor has been added to a module, it needs to be registered with uimaFIT. This happens by creating the file

consisting of a list of type system descriptor locations prefixed with classpath*:, e.g.:

Instead of pre-generating the JCas classes and storing them in the version control, we use the jcasgen-maven-plugin to automatically generate JCas classes at build time. The automatic generation of JCas classes need to be explictily enabled for modules containing types. This is done by placing a file called .activate-run-jcasgen in the module root with the content

However, in some we customized the JCas classes, e.g. we added the method DocumentMetaData.get(JCas). Such classes are excluded from being generated automatically by placing them in a second descriptor file called dkpro-types-customized.xml, e.g.

Often a type comes with a certain policy. For example, root nodes in a dependency relation tree should have the type ROOT and the features governor and dependent should point to the same token. Another example would be that if a constituent is a child of another constituent, then its parent feature should be set accordingly.

To ensure that all components adhere to such policies, it is a good idea to implement checks for them. This can be done simply by placing a new check implementation into the package de.tudarmstadt.ukp.dkpro.core.testing.validation.checks in the testing module. Tests implemented with TestRunner and IOTestRunner use these unit tests automatically. All other checks should invoke AssertAnnotations.assertValid(jcas).

The architecture for resources (e.g. parser models, POS tagger models, etc.) in DKPro is still work in progress. However, there are a couple of corner points that have already been established.

Resources needed by DKPro components (e.g. parser models or POS tagger models) are not packaged with the corresponding analysis components, but as separate JARs, one per language and model variant.

Due to license restrictions, we may not redistribute all of these resources. But, we offer Ant scripts to automatically download the resources and package them as DKPro-compatible JARs. When the license permits, we upload these to our public Maven repository.

If you need a non-redistributable resource (e.g. TreeTagger models) or just want to package the models yourself, here is how you do it.

Whenever one existing model have a new release, it is good to update the build.xml changing the:

After that, run the ant script with the local-maven target, add the jars to your project classpath and check if the existing unit tests work for the up to date model. If they do, then run the script again, this time with the remote-maven target. Then, change the versions from the models in the dependency management section from the project’s pom file, commit those changes and move these new models from staging into model repository on zoidberg.

Typical metadata items for a model.

Almost all models should declare at least one tagset. We currently declare only the tagsets that a model produces, not those that it consumes.

The Dublin Core (DC) metadata items are not (yet) widely used throughout the models. This might change in the future.