Hermes Architecture and User Guide

Document collections and corpora allow for the following access to their collection of documents:

In addition to the methods above, corpora allow for access to individual documents using get(String) method where the string parameter is the document id.

The main method for manipulation of a collection is through the update(SerializableConsumer<Document>) method, which processes each document using the given consumer. For document collections this method acts as a map whereas for corpora the update persists to the underlying storage.

Corpora also allow for individual documents to be udpated via the update(Document) method. Additionally, documents can be added and removed from corpora using the following set of methods:

Hermes provides a simple boolean query language to query documents. The query syntax is as follows:

Multiword phrases are expressed using quotes, e.g. 'United States' would match the entire phrase whereas United AND States only requires the two words to present in the document in any order. The default operator when one is not specified is OR, i.e. United States would be expanded to United OR States.

As shown in the code snippet above, querying a corpus results in a SearchResults which retains the query that generated results and a document collection view of the results.

A common step when analyzing a corpus is to calculate the term and document frequencies of the words in its documents. In Hermes, the frequency of any type of annotation can be calculated across a corpus using the termCount(Extractor) method. The analysis is defined using an Extractor object, which provides a fluent interface for defining annotation type, conversion to string form, filters, and how to calculate the term values (see Text Mining for more information on Extractors). An example is as follows:

By default, the TermExtractor will specify TOKEN annotations which will be converted to a string form using the toString method, all tokens will be kept, and the raw frequency will be calculated.

In a similar manner, document frequencies can be extracted using the documentCount(Extractor) method. An example is as follows:

Both the termCount and documentCount methods take an Extractor, which can include any type of extraction technique (discussed in Text Mining).

While n-grams can be extracted using the termCount and documentCount feature, Hermes provides the nGramCount(NGramExtractor) method for calculating document-based counts of n-grams where the n-gram is represented as Tuple of string. An example of gathering bigram counts from a corpus is as follows:

By default, the NGramExtractor will specify TOKEN annotations which will be converted to a string form using the toString method, all tokens will be kept, and the raw frequency will be calculated.

In addition, Hermes makes it easy to mine "significant bigrams" from a corpus using the significantBigrams(NGramExtractor, int, double) and significantBigrams(NGramExtractor, int, double, ContingencyTableCalculator) methods. Both methods take an NGramExtractor to define how the terms should be extracted (note that the min and max order is ignored), a (int) minimum count required to consider a bigram, and a (double) minimum score for a bigram to be considered significant. Additionally, a ContingencyTableCalculator can be given which is used to calculate the score of a bigram (by default Association.Mikolov is used which is the calculation used within word2vec to determine phrases). Both methods return a Counter<Tuple> containing the bigrams and their score. The following example illustrates finding significant bigrams using Normalized Pointwise Mutual Information (NPMI):

Often times we only want to use a small portion of a corpus to test for analysis in order to test it out. The corpus class provides a means for performing reservoir sampling on the corpus using the following two methods:

Both return a new corpus and take the sample size as the first parameter. The second method takes an additional parameter of type Random which is used to determine inclusion of a document in the sample. Note that for non-distributed corpora the sample size must be able to fit into memory.

The Corpora class provides a groupBy(SerializableFunction<? super Document, K>) method for grouping documents by an arbitrary key. The method returns a Multimap<K, Document> where K is the key type and takes a function that maps a Document to K. The following code example shows where this may of help.

Note that because this method returns a Multimap, the entire corpus must be able to fit in memory.

The $_ (or this) operator represents the current object in focus, which by default is the object passed into one of the LyreExpression’s apply methods. Note that one-argument methods in Lyre (e.g. lower, isUpper, etc.) have an implied $_ argument if none is given.

String Literals: Lyre allows for string literals to be specified using single quotes ('). The backslash character can be use to escape a single quote if it is required in the literal.

Numeric Literals: Lyres accepts numerical literal values in the form of ints and doubles and allows for scientific notation. Additionally, negative and positive infinity can be expressed as -INF and INF respectively and NaN as NaN.

Null: Null values are represented using the keyword null.

Boolean Literals: Boolean values are represented as true and false.

A list of literals or expressions can be defined as follows:

Note when a list is the return type and the returned list would have a single item the single item is returned instead. For example, if a method generated the list [1], the value 1 would be returned instead of the list.

Length: The length of a list is determined using the `llen' method as follows:

where we are returning the length of the list of entities on the object in focus.

List Accessors: Lyre provides three methods for accessing a list of items:

The following code snippet illustrates using these three accessor methods:

List Selectors: Lyre provides two methods for selecting the best item in a list:

The following code snippet illustrates using these two selection methods:

List Transforms: Lyre provides three methods of transforming a list:

Note that Lyre will create a one-item list if the list item passed in is not a collection. The following code snippet illustrates using these three transform methods:

List Predicates: Lyre provides three methods for testing a list based on its items:

Note that Lyre will create a one-item list if the item passed in is not a collection. The following code snippet illustrates using these three predicate methods:

Logical Operators: Lyre provides a set of logical operators for and (&&), or (||), and xor (^) that can be applied to two predicate expressions. Note that if a non-predicate expression is used it will evaluated as a predicate in which case it will return false when the object being tested is null and true when not null with the following checks for specific types of the expression being treated as a predicate:

Negation: Lyre uses ! to denote negation (or not) of a predicate, e.g. !isLower negates the the string predicate testing for all lowercase letters, returning true if the string passed in has any non-lowercase letter.

Relational Operators: Lyre provides the standard set of relational operators, =, <, ⇐, >, >=, and !=. How the left-hand and right-hand sides are compared is dependent on their type. The following table lists the comparison rules.

Pipe Operators: Lyre provides two pipe operators. The first is the And-pipe operator &> which sequentially processes each expression with the output of the previous expression or the input object for the first expression. All expression are evaluated regardless of whether or not a null value is encountered. The second is the Or-pipe operator |> which sequentially processes each expression with the input object, returning the result of the first expression that evaluates to a non-null, non-empty list, or finite numeric value.

Plus: The plus operator, +, can be used to concatenate strings, perform addition on numeric values, or append to a list. Which operation is performed depends on the LHS and RHS type as follows in order:

Membership Operators: Lyre provides to membership operators the in and the has operator. The in operator, LHS in RHS, checks if the left-hand object is "in" the right-hand object, where in means "contains". Lyre is able to handle collections, lexicons, and CharSequence as the right-hand object.

The has operator, LHS has RHS, checks if any annotations on the LHS HString evaluates to true using the right-hand expression.

The code snippet above checks if the current HString in focus has any phrase chunks whose part-of-speech is NP (Noun Phrase).

Slice Operator: Performs a slice on Strings and Collections where a slice is a sub-string or sub-list. Slices are defined using the square brackets, [ and ], with the starting (inclusive) and ending (exclusive) index separated by a colon, e.g. [0:1]. The starting or ending index can be omitted, e.g. [:1] or [3:], where the implied starting index is 0 and the implied ending index is the length of the object. Additionally, the ending index can be given as a relative offset to the end of the item, e.g. [:-2] represents a slice starting at 0 to item length -2. An example of the slice operator is as follows:

Length: The length in characters of the string representation of an object or the number of items in a list can be determined using the len method.

If: The if-then ,if( PREDICATE, TRUE_EXPRESSION, FALSE_EXPRESSION ), method performs a given true or false expression based on a given condition. The following example snippets checks if the object in focus is a digit and when it is returns the string literal '[:digit:] and when it is not returns the item.

When: The when, when( PREDICATE, TRUE_EXPRESSION ), performs the given expression when the given condition is true and returns null when the condition is false. The following example snippets checks if the length of the item in focus is greater than three and when it is returns the concatenation of the string literal 's3=' and the substring/sublist starting at index 3 to the end of the item.

Not Null: The not null,nn( EXPRESSION, DEFAULT_VALUE_EXPRESSION ), returns the result of the given expression when not null and the result of the default value expression when null. The following example snippets checks for the existence of entities on the object in focus and when there are none (the empty list is automatically converted into a null value) will return the string literal 'non-entity'.

Match All: The all predicate, ~, returns true for all input.

Exists: The exists predicate, exists( OBJECT ) checks if the Object exists meaning it is not null and not a blank CharSequence or empty list.

The code snippet above checks if the object in focus has at least one overlapping entity returning true if it does and false otherwise.

Look Behind: The positive, (?< …​), and negative, (?!< …​ ), look behind predicates determine if the previous annotation matches (positive) or does not match (negative) the given expression. Note that positive and negative look behinds should be done on single HStrings or used in a list operator.

Look Ahead: The positive, (?> …​), and negative, (?!> …​ ), look ahead predicates determine if the next annotation matches (positive) or does not match (negative) the given expression. Note that positive and negative look aheads should be done on single HStrings or used in a list operator.

String Matching: Lyre provides the following predicates for matching strings:

Note that for all methods listed above, the OBJECT is optional and will default to $_ if not specified.

Tag Matching: The tag predicate, #TAG_VALUE( OBJECT ), checks if the Tag on the object in focus is of the given Tag value. Note that object in focus must be an annotation and the tag value needs to be convertible to that annotation’s tag type.

String transforms: Lyre provides the following methods for transforming objects into a string representation:

All of the methods listed above can specified by their name only, e.g. string, when taking $_ (this) as the argument.

Apply transform: Lyre uses LHS ~= RHS to denote that the RHS expression is to be applied to the LHS expression. This is especially useful for regular expression matching and substitution.

Context Expansion: The context method, cxt( HSTRING, CONTEXT_SIZE ), returns a contextual (previous or next) token for the given HString at the given position (relative) where a positive value for the context size represents a next contextual token and a negative value a previous contextual token.

String Padding:

Note that in both lpad and rpad if the given PADDING_CHARACTER has a length > 1 only the first character is used.

HString Trimming: The trim, trim(HSTRING, PREDICATE), method removes tokens from the left and right of the input HString if they evaluate to true with the given predicate. The following code snippet trims token from the input HString when they are a stopword or have a length less than three characters.

Annotation Accessors: Lyre provides the following methods for accessing the annotations on an HString:

Note that in call cases if the returned list has a single annotation, the single annotation will be returned instead.

The following code snippet shows examples of extracting annotations in Lyre.

Attributes: Lyre provides the following methods for accessing the attributes on an HString:

*Token Length: To determine the number of tokens in an object in focus the tlen method is available.

Lexicons: The lexicon function, %LEXICON_NAME, Returns the lexicon with the given LEXICON_NAME using Hermes’s LexiconManager. The returned lexicon can then be used to check of the existence of words. The following snippet gives two examples of checking if the object in focus is in a lexicon named person:

As can be seen in the snippet above, the lexicon acts as a predicate checking for existence.

Word Lists: The word list function, wordList( LIST_EXPRESSION ), will create a temporary word list for use. The following code snippet shows an example of constructing a temporary word list made up of the strings "dog", "cat", and "bird" and using the constructed word list as a container to check if the object in focus’s string representation if one of the items in the word list.

Feature Generator: Lyre provides the following methods for easily constructing lists of features:

Count Generator: Lyre provides the following methods for easily creating Counters over lists of items:

Content can be matched in the following manner:

The following word classes are defined:

Token-based regular expressions allow for matching of attribute values including numeric comparisons for number-valued attributes. The first set of attribute-based matching will match non-numeric attributes, such as strings and tags.

Additionally, the full range of numeric comparisons are available:

Tags: Tag attributes can be easily matched using #TAG_VALUE, e.g. #NOUN for tokens would match all tokens whose part-of-speech is a noun.

Categories: Whether or not an HString or one of its sub-spans contains a given category value can be expressed using $CATEGORY ~ 'VALUE' or $CAT ~ 'VALUE' where the VALUE must be predefined BasicCategory.

Matches can include the existence of annotations including optional constraints on the match. Annotations are declared using @ANNOTATION_TYPE and constraints can be specified using parenthesis, i.e. @ANNOTATION_TYPE(…​). The following code snippet gives examples:

The following set of greedy qualifiers are supported:

Hermes’s token-based regular expressions also supports the following logical operators:

The following special constructs are supported:

Look Behind: The positive, (?< …​), and negative, (?!< …​ ), look behind predicates determine if the previous annotation sequences matches (positive) or does not match (negative) the given expression.

The first expression, from the snippet above, matches noun phrases which are preceded by the phrase "manager of". The second example, matches all HStrings whose lemmatized form is "bank" and are not preceded by a determiner or adjective.

Look Ahead: The positive, (?> …​), and negative, (?!> …​ ), look ahead predicates determine if the next annotation sequence matches (positive) or does not match (negative) the given expression.

The first expression, from the snippet above, matches noun phrases which are followed by verb phrases. The second example, matches all HStrings which represent Digits and are not followed by punctuation.

Named Groups: Named groups can specified using (?<NAME> …​). The groups captured HString is accessed using TokenMatcher.group(String) where the String is the group name. The following example illustrates a named group:

Backreference: Backreference to named groups is possible using \GROUP_NAME.

The example snippet above matches all sequences in which a person entity is followed by a distant verb phrase for which the the nsubj is the matched person entity. Note that the backreference will match when there is overlap of between the HString in the named group and the HString being examined for the backreference.