Skip to content

Solr

Attention: The Solr index might experience delays and should not be used as source of truth for critical operations.

Introduction and activation

Whenever a EntryStore entry is created, modified or removed in EntryStore, events are created which are caught by listeners taking care of the procedures in the Solr index. If needed, the Solr index can be recreated during the startup of EntryStore. The Solr-related parameters in the EntryStore configuration are:

entrystore.solr=on
entrystore.solr.reindex-on-startup=off
entrystore.solr.reindex-on-startup.wait=off # if on: blocks the startup until reindexing is complete
entrystore.solr.url=http://localhost:8080/solr/entrystore-core1
entrystore.solr.max-limit=200 (default: 100)
entrystore.solr.auth.username=<username when connecting via HTTP, leave empty if no authentication is necessary>
entrystore.solr.auth.password=<password when connecting via HTTP>
entrystore.solr.request-timeout-seconds=60 (6.x, default: 60)

entrystore.solr.url is the URL of the Solr core, e.g. http://localhost:8983/solr/entrystore.

Setting up Solr

EntryStore 6.x requires an external Solr 10 server that is reachable via HTTP(S). Embedded Solr is no longer supported; a value of entrystore.solr.url that is not an HTTP(S) URL makes startup fail.

The Solr core must be created with the configset of the EntryStore release, i.e. both schema.xml and solrconfig.xml and the accompanying files. It can be found in solr/conf in the source repository and in /srv/entrystore/solr/conf in the Docker image. Use the configset of the release that is deployed and reload the core when EntryStore is upgraded. Further requirements:

  • The schema uses solr.ICUCollationField, so Solr must load the analysis-extras module (e.g. SOLR_MODULES=analysis-extras for the official Solr Docker image).
  • At startup, EntryStore checks via the Schema API that the core's schema contains the fields it writes. If the core does not exist (404) or the Schema API is not accessible to the configured Solr user (401/403), startup is aborted. If Solr is unreachable, EntryStore starts anyway.
  • A custom schema must keep all field definitions of the shipped schema.

The script docker/solr-start.sh in the source repository starts a Solr container with a core that is set up this way, e.g. for development.

EntryStore 5.x

EntryStore 5.x uses Solr 9 and also supports an embedded Solr server: if entrystore.solr.url is a local path, e.g. entrystore.solr.url=/srv/entrystore/solr/, an embedded Solr server (SolrJ) is used. EntryStore then initializes the core with its default configuration (solrconfig.xml_default and schema.xml_default in the source repository). The locations of these files can be overridden with entrystore.solr.schema.url and entrystore.solr.config.url (URLs with http, https, or file schema; redirects are not followed). For an external Solr server, the supplied schema.xml has to be used.

Indexed information

The following pieces of information of an entry are indexed in Solr (see also schema.xml) and can therefore be used as search parameters (they are case-sensitive!):

  • title: All titles (DC and DC terms) in all languages, including FOAF and VCard names.
  • title.lang: To facilitate sorting after titles a dynamic field is created with the pattern title.[two-letter language code according to ISO 639-1]. No multi value support because these fields are intended to be used in the context of sorting. Example: title.en for a title in English, title.nolang for a title without language set.
  • description: All descriptions in all languages.
  • tag.literal: All literal tags in all languages. Covers common tag predicates such as dc:subject and dcterms:subject.
  • tag.literal.lang: If a tag has a language set, a dynamic field is created with the pattern tag.literal.lang, e.g., tag.literal.en for all literal tags in English.
  • tag.uri: All resource tags. Covers common predicates such as dc:subject and dcterms:subject.
  • metadata.predicate.literal.<md5>, metadata.predicate.uri.<md5>: Used to carry out queries for exact predicate-object combinations. Due to restrictions in Solr it was necessary to shorten the predicate URI, which was done using an MD5 hash (hex) of the predicate URI truncated after 8 characters. Collisions are unlikely and in case of occurence non-fatal. The field value is the string value of the object. The field names are different for URI and literal objects due to different indexing strategies in Solr for the respective object type (literals are indexed using "text_ngram" and URIs are indexed using "string").
  • metadata.predicate.literal_s.<md5>: same as metadata.predicate.literal.<md5> (without _s) above, but indexed as "string" instead of "text_ngram".
  • metadata.predicate.literal_t.<md5>: same as metadata.predicate.literal.<md5> (without _t) above, but indexed as "text" instead of "text_ngram".
  • metadata.predicate.date.<md5>: same as metadata.predicate.literal.<md5> above, but indexed as "date" instead of "text", enabling date-specific queries such as range queries etc.
  • metadata.predicate.integer.<md5>: same as metadata.predicate.literal.<md5> above, but special treatment for integer literals. This field is single-valued to allow for sorting and covers all numerical integer datatypes (including e.g. short, byte, long, etc). Integers are indexed using Solr's "slong" type.
  • metadata.predicate.decimal.<md5>: same as metadata.predicate.literal.<md5> above, but special treatment for decimal (fractional) literals. This field is single-valued to allow for sorting and covers the datatypes xsd:decimal, xsd:float and xsd:double. Decimals are indexed using Solr's "sdouble" type, which enables numeric range queries, numeric sorting and numeric faceting. Literals with these datatypes are also indexed as strings in metadata.predicate.literal_s.<md5>, but string comparison is lexicographic and therefore not suitable for numeric queries.
  • related.metadata.predicate.*.<md5>: same as metadata.predicate.*.<md5> above, but based on the related property index as described further down on this page.
  • metadata.object.literal: All values of string literals (the datatype must not be specified), independent of language.
  • metadata.object.uri: All object URIs.
  • lang: The language of the resource, fetched from dc and dcterms:language. Used for searches.
  • all: catch-all Solr field, containing title, description, tags from above. This field is used if no search property is provided in the Solr query.
  • uri: Entry URI.
  • resource: Resource URI.
  • rdfType: The RDF type of the resource, fetched from the entry and the metadata graph.
  • context: Resource URI of the entry's surrounding context.
  • contextname: Name of the context if the indexed entry is a context or system context. This field is not used for regular entries.
  • creator: Creator URI.
  • contributors: URIs of contributors.
  • lists: Resource URIs of referring lists.
  • public: true if the entry metadata is readable by the guest user. If the guest readability of a context changes, the context is reindexed automatically (see Automatic reindexing of contexts), but other ACL changes do not trigger a reindex, and until a reindex is complete the public field may be outdated. Do not use this field if you do not fully understand how it works.
  • created: Creation date.
  • modified: Modification date.
  • graphType: the builtin type, case sensitive (i.e. Context, SystemContext, User, Group, List, ResultList, String, Graph, Pipeline, None).
  • entryType: the location type, case sensitive (i.e. Local, Link, LinkReference, Reference).
  • resourceType: the resource type, case sensitive (i.e. InformationResource, ResolvableInformationResource, NamedResource, Unknown).
  • projectType: EntryScape project type (http://entryscape.com/terms/projectType), case sensitive.
  • acl.admin: URIs of principals with admin rights (explicitly set in entry info).
  • acl.metadata.r: URIs of principals with read rights on metadata (explicitly set in entry info).
  • acl.metadata.rw: URIs of principals with read/write rights on metadata (explicitly set in entry info).
  • acl.resource.r: URIs of principals with read rights on the resource (explicitly set in entry info).
  • acl.resource.rw: URIs of principals with read/write rights on the resource (explicitly set in entry info).
  • profile: URI of the entry's application profile. Originates from the entry graph and consists of the object value of the predicate http://entrystore.org/terms/profile (the deprecated predicate http://entryscape.com/terms/entityType is supported for backwards compatibility).
  • metadata.predicate.literal_l.<md5>, related.metadata.predicate.literal_l.<md5> (6.x): internal fields used for language-specific facets. They are not searchable and cannot be used as facet or sort fields.

The following fields support incremental/partial matching through n-grams (3 to 25 characters) on the index:

  • title
  • title.lang
  • description
  • tag.literal
  • tag.literal.lang
  • metadata.predicate.literal.*
  • related.metadata.predicate.literal.*

In 6.x, values are split into words at whitespace and the n-grams are created per word; words of one or two characters and words of more than 25 characters are also indexed as a whole. Query terms are split at whitespace and lowercased, but not n-grammed. As a consequence, a search for eu* matches the word "EU", a whole long word matches, and a phrase query matches adjacent words of one value.

EntryStore 5.x

In 5.x the n-grams are created across the whole literal, including whitespace. Words of one or two characters are only matched if they are followed by a space, words of more than 25 characters are not matched as a whole, and phrase queries on these fields match practically nothing.

CAVEAT: all colons in URI parameters must be escaped with a back slash, i.e. "\:", otherwise Solr throws an exception.

RDF Format

By default, the Solr search returns all RDF graphs in RDF/JSON. This can be overridden by providing the URL parameter rdfFormat with the alternative value application/ld+json to serialized the RDF graphs using JSON-LD. Optimization takes place to detect the namespaces in use, which are used as JSON-LD contexts. Any other value of rdfFormat results in RDF/JSON. A + sent unencoded (rdfFormat=application/ld+json) is accepted. In 6.0, other values and an unencoded + resulted in empty graphs.

Solr Query Syntax

The default query field concatenator in EntryStore is OR. This can be overridden in every query, see examples below.

See the following links for more detailed information about the query parser syntax:

EntryStore uses Solr's standard query parser with all as default field and OR as default operator. Solr local parameters ({!...}) cannot be used in the sort and facetFields parameters.

The query parameter must be at least three characters long. If Solr is disabled (entrystore.solr=off), the search answers 503.

The following characters that need to be escaped (using \) inside the query parameter because they are part of the Solr query syntax: \+-!():^[]"{}~*?|&;/.

Sorting

The parameter sort can be used for Solr-style sorting, e.g. sort=title.en+asc,modified+desc. The default sorting value is to sort after the score (relevancy) and the modified (last modification date). All text_sort and single value fields can be used for sorting, this basically excludes title, description and keywords, but allows sorting after e.g. title.en. title.* is of type text_ngram, but when used as sorting property it is internally rewritten into title_sort.* which is of type text_sort and ignores case. In 6.x, sort only accepts plain field names followed by asc or desc; function queries and local parameters are answered with 400.

Default language

Mostly of relevance for instances that are being used with a language that is not natively supported by the client application (e.g. client supports lang1 and lang2, but the user mostly provides metadata in lang3). In such cases it is possible to configure a default sorting language that will result in literals that match the default language being indexed as title.default and title_sort.default, respectively.

Configuration:

entrystore.solr.default-sorting-lang=<language code> (default: no default language set)

Boosting

In order to increase the relevance (boost) of certain fields it is possible to use the caret symbol ^, which is regular Solr syntax. To boost the title field with factor 10 it is possible to use the following construct: title:someterm^10. Boosting is specific for terms and not for fields, if provided at query time. Boosting at index time is possible, but requires modifications of the Solr configuration.

By default the fields used for sorting are score and modified, so in order to get a search result fully based on relevance it is needed to override the default sort setting by providing score as only sorting parameter: sort:score+desc.

Example

  • http://base/search?type=solr&query=title:organic^50+OR+title:farming^25+tag.literal:organic^10&sort=score+desc

The ^ needs to be URL encoded like the rest of the query. The example above does not encode special characters and URIs for better readability.

Accuracy of result count

Every search result also contains an estimated result count (e.g. to be used with pagination). This number may not reflect the real amount of results which the requesting user has access to. In such cases the result count is always higher than the number of accessible entries.

The reason for this lies in how ACL management is done. The ACL is contained in the triple store, but for performance reasons it cannot be included in the Solr-index. The search is run against the Solr-index which also returns the result count (this is the same number as returned by the EntryStore search resource in the REST API), but the ACL of the matching entries is not looked up before the entries and their metadata are returned in the partial result set (because of pagination).

Pagination

The response size is limited to 50 by default and can be set with the url parameter limit. If the result count is greater than the limit it is possible to fetch the next result batch using the offset parameter.

The following URL-parameters which are relevant for pagination are:

  • limit: default 50. Values above the maximum limit are reduced to it; the maximum is 100 by default and can be changed with the setting entrystore.solr.max-limit, see above. limit=0 returns only the result count.
  • offset: default 0, determines where the returned result set starts.

Examples

  • http://base/search?type=solr&query=title:organic+AND+graphType:List&sort=score+desc,modified+desc
  • http://base/search?type=solr&query=organic+AND+lang:de&sort=title.de+desc,modified+desc
  • http://base/search?type=solr&query=organic+OR+ecology+lang:en (this would yield the same result also without OR)
  • http://base/search?type=solr&query=organic+lang:en&limit=10&offset=20
  • http://base/search?type=solr&query=title:*&limit=10&rdfFormat=application/ld+json
  • http://base/search?type=solr&query=metadata.predicate.decimal.256bd150:[10+TO+100]&sort=metadata.predicate.decimal.256bd150+asc: numeric range query and numeric sorting on a predicate with decimal, float or double values.

Syndication feed

Returning data in the format of an Atom or RSS feed can be enabled by using the parameter syndication, defining the format you want to use. The following values are supported:

  • atom_1.0
  • atom_0.3
  • rss_2.0
  • rss_1.0

You can also use the lang parameter to define preferred language of entries to return. The value should be a 2 character code following the ISO 639-1 specification ("se", "de", etc.). If omitted, en is used. The number of items in the feed is determined by limit.

The following parameters are not supported while using syndication:

  • offset - Pagination of feeds is not supported

For information on how to configure custom URL patterns for links of feed items please see the page on syndication.

Parameters

  • facetFields: A comma-separated list of fields that should be treated as facets. In 6.x only plain field names are accepted; local parameters such as {!facet.limit=-1} and the catch-all fields all, fulltext and metadata.object.literal are answered with 400.
  • filterQuery: Filter queries in Solr syntax to filter the results of a (faceted) search, e.g. metadata.predicate.uri.256bd150:http\://example.org/crop. Several filter queries are separated by commas; a comma inside a filter query must be URL-encoded twice (%252C).
  • facetMinCount: Result threshold for facets to be included in query result. Must be an Integer value, default is 1. 0 also returns facet values without hits.
  • facetLimit: Maximum amount of facets to be returned. Must be an Integer value, default is 100. Values up to 1000 are allowed. Higher limits can be allowed by setting entrystore.solr.facet-max-limit in entrystore.properties.
  • facetMatches: To only return facet values that match the given text. Only applies together with facetFields. In 6.x the value is literal text of up to 256 characters that must match the whole facet value; the characters \^$.|?*+()[]{} must be escaped with a backslash, e.g. Dr\. Smith.
  • facetMissing: If set to true, a count of all results that match the query, but which have no facet value for the field should be computed and returned in the response. Default is false.
  • facetLang (6.x): Language tag to only return the facet values of literal fields that are in this language or have no language, see below.

EntryStore 5.x

In 5.x, facetMatches is a regular expression, and facetFields and sort are passed to Solr as they are, including local parameters.

Language-specific facets

With facetLang (6.x), the facet values of literal fields (metadata.predicate.literal_s.<md5>, metadata.predicate.literal.<md5> and their related. variants) are restricted to labels that occur in the requested language or without language. The language is matched as a prefix, so en also matches en-GB. facetLang requires facetFields.

The count of a facet value is its count across all languages, taken before access control is applied, not the number of entries in the requested language. The selection of values happens before facetLimit is applied, so a label that is rare in the requested language but frequent overall may be missing from the top values. Labels longer than 256 characters are not returned when facetLang is set.

Fields

Fields that are indexed fully and without modification (i.e., field type string" in Solr) may be used as facets; in particular the following fields are intended to be used as facets:

  • metadata.predicate.date.<md5>
  • metadata.predicate.integer.<md5>
  • metadata.predicate.decimal.<md5>
  • metadata.predicate.literal.<md5> (this field is of type "text" which EntryStore automatically translates into metadata.predicate.literal_s.<md5> for facetted queries)
  • metadata.predicate.literal_s.<md5>
  • metadata.predicate.uri.<md5>
  • All fields containing a URI
  • All fields named with pattern *Type

Examples

  • http://base/search?type=solr&query=organic&facetFields=metadata.predicate.literal.256bd150&filterQuery=metadata.predicate.literal.256bd150:crop: Use dc:subject as facet and value "crop" as filter.

Limits and rate limiting

The following settings (6.x) restrict the input of /search and protect Solr from expensive requests. Requests exceeding a limit are answered with 400. All limits are unlimited (0) by default; a negative value makes startup fail.

entrystore.solr.search.query.max-length=0
entrystore.solr.search.sort.max-length=0
entrystore.solr.search.filter-query.max-length=0
entrystore.solr.search.filter-query.max-count=0
entrystore.solr.search.facet-fields.max-length=0
entrystore.solr.search.facet-fields.max-count=0

The number of search requests per client IP address can be limited (6.x). Requests above the limit are answered with 429. The limit is disabled by default (max=0), applies to authenticated users as well, and is keyed by the client IP only, so clients that share an IP address (e.g. users behind NAT or the EntryScape API server) share one budget. See Reverse proxy for how the client IP is determined.

entrystore.solr.search.rate.limit.max=0
entrystore.solr.search.rate.limit.window=1m

The related property index contains properties from other entries that the entry's metadata links to. All relations that are to be followed must be configured using the following settings:

entrystore.solr.related=on|off (default: off)
entrystore.solr.related.properties.n=<predicate URI>

By default only entries in the same context are indexed. This can be changed per property by appending the "global" parameter to the property configuration. The URI and the parameter need to be separated by a comma:

entrystore.solr.related.properties.n=<predicate URI>,global

Example

entrystore.solr.related=on
entrystore.solr.related.properties.1=http://purl.org/dc/terms/creator,global
entrystore.solr.related.properties.2=http://schema.org/image
entrystore.solr.related.properties.3=http://schema.org/location
entrystore.solr.related.properties.4=http://schema.org/address

Inspecting an entry's index

The owner of an entry can inspect the indexed information of an entry by accessing its index URL which is contructed by appending /index to the end of the entry URI.

Example: https://base/1/entry/1/index

The response body contains a JSON object with all indexed fields and its values.

Rebuilding the index

The index can be rebuilt upon startup (recommended) through a property in the EntryStore settings (see above).

In 6.x, EntryStore also rebuilds the index automatically at startup when the index was created by another EntryStore release or another Solr major/minor version. For this it keeps the marker files SCHEMA_VERSION and SOLR_VERSION in entrystore.data.folder; without a configured data folder, the automatic rebuild does not happen.

Alternatively, admin users (i.e. admin and members of the admin group) and users with administration rights on a context have the possibility to rebuild the Solr index or specific contexts live without restart through the management API using the following request:

POST http://base/management/solr

Content-Type: application/json

{
  "command": "reindex",
  "context": "https://base/_contexts/entry/context-id"
}

If the context parameter is omitted, the whole repository will be reindexed; this requires admin rights. The context parameter must be the entry URI of the context.

In 6.x the request is answered with 202 when the reindex has been started, with 409 if a reindex of the whole repository (or of the same context) is already running, including the reindex at startup, and with 503 if Solr is disabled. The progress can be followed in GET /management/status/extended under solr.indexingContexts.

Upgrading to version 6.1

The metadata.predicate.decimal.<md5> field (see above) is new in EntryStore 6.1. Existing documents in the index do not contain it, because decimal, float and double literals were previously only indexed as strings. To populate the field for already indexed data, the repository has to be reindexed once after the upgrade, either upon startup or through the management API as described above.

Instances coming from 5.x get a fresh index anyway (see Upgrading from 5.x to 6.x). Instances that already run a 6.0 build need the following steps:

  1. Deploy the configset of the 6.1 release to the Solr core (see Setting up Solr). Instances running a custom schema.xml must add all new field definitions, among them metadata.predicate.decimal.*, metadata.predicate.literal_l.* and their related. variants. EntryStore 6.1 refuses to start if the literal_l fields are missing.
  2. Empty the index before starting 6.1, e.g. by deleting the core's data directory: the 6.1 schema stores many fields with docValues, which Solr cannot add to an existing index. Without this step, Solr rejects every document and the index stays empty.
  3. Start EntryStore 6.1. With entrystore.data.folder configured, the index is rebuilt automatically; otherwise start once with entrystore.solr.reindex-on-startup=on.
  4. Check that solr.rejectedDocuments in GET /management/status/extended is 0.

Automatic reindexing of contexts

If the ACL of a context is updated in a way that changes the context's (and its entries') metadata readability (public/non-public via the _guest user), the context is automatically scheduled for reindexing. There is a built-in delay of 10 seconds before reindexing start in case the client/user toggles the _guest readability within a short time frame.