Quickfind

Quickfind is the indexing mechanism for all text information within ExtraView.  It is recommended that this be configured for all ExtraView installations, although querying for information will work without this being turned on.  The major advantage of Quickfind is that the speed of searching and querying is greatly increased.

Quickfind configuration is completed by setting up behavior settings and configuring the Quickfind indexing task.

Note that documents and attachments larger than 16MB are not indexed.

The Quickfind indexes are held externally to the ExtraView database, utilizing an open source set of libraries from the Apache Foundation.  The libraries are known as Lucene.

Quickfind Task

The section on Managing Quickfind contains full information on Quickfind configuration.  This page solely discusses how to perform the initial configuration and subsequent upgrading of the Quickfind indexes.

Within a new installation all the setup and configuration may be accomplished within the Manage Quickfind task, or can be managed with the information following on this page.

This Quickfind task automatically indexes newly added text to all issues entered or updated by users. Once configured and started, this task will keep the indexes up-to-date as users add or update any text within issues.

For new installations, once you have completed all the steps on this page, go to the Managing Quickfind page to configure and start the Quickfind task.

Behavior Settings

Behavior Setting Purpose
QUICKFIND_ACTIVE_INDEX This setting determines which index is currently active for QuickFind. The value may be PRIMARY or ALTERNATE. Quickfind keyword searches are performed using the directory in QUICKFIND_INDEX_LOCATION as the primary or QUICKFIND_INDEX_LOCATION_ALT as the alternate respectively
QUICKFIND_INDEX_ENUMERATED This setting controls whether the titles to UDFs are indexed.  If indexed, keyword searches that contain issues with the values of these titles will be returned within the results
QUICKFIND_INDEX_LOCATION The name of the directory on the file system where Lucene full text indexes will be stored. Before any indexes are built, you must also enable full text searching by setting ENABLE_QUICKFIND. You must also ensure that the path is valid from the application server(s) that are running ExtraView, and that you have all the permissions to read and write to the storage. You must also ensure that you set up a separate backup method for this external storage as backing up your database will not backup the full text indexes
QUICKFIND_INDEX_
LOCATION_ALT		 This is an alternate directory path for the QUICKFIND index files. When using the ALTERNATE QUICKFIND_ACTIVE_INDEX, keyword searches are performed using this directory.
QUICKFIND_INDEX_USERS This setting controls whether Quickfind includes user IDs in the text search for keyword searches. Valid values are YES and NO.
QUICKFIND_MAX_HITS This is the maximum number of search hits in the QuickFind index permitted with any single query. When a greater number of issues are returned from the index, only the first QUICKFIND_MAX_HITS are used. The default is 20,000.

Preparation

Before setting up Quickfind, you should carefully consider where to store the index files that are created.  The following pointers should be taken into account:

  • You should have sufficient space for the index storage.  This is dependent upon the size of database and the amount of text that is contained within it.  Indexes can vary in size from tens of Megabytes of space to several Gigabytes of space in a very large system
  • You must have write permission from the ExtraView WEB-INF folder to this location
  • The path to the index files may be relative to WEB-INF, or be an absolute path.  It is recommended that you choose an absolute path.  The reason is that upon a system upgrade, you will probably rename and replace the old ExtraView installation, and the relative path that contains the indexes will no longer be in the correct location.

FullTextIndexSetup

If you are enabling Quickfind on an existing installation, you index all the existing information by using an external program named FullTextIndexSetup. This should be accomplished with the task named Quickfind Synchronization Task (FULL_TEXT_SYNCHRONIZE) turned off, or the ExtraView application server stopped. Some upgrades may also require the indexes to be rebuilt, using the same FullTextIndexSetup program.  As discussed below, it is possible to keep your ExtraView installation online while you upgrade Quickfind.

The FullTextIndexSetup utility is extremely quick on a new database, but may take some time on a very large existing database. It is difficult to predict exactly how long this will take on a large database, as there are many factors such as processing speed, memory, amount of text, and the number of stored attachments that all have an impact. Your users can continue working during the period of indexing, and the search results will improve as the process continues. Our recommendation is to start the process after the majority of users leave work for the day.

The FullTextIndexSetup utility is found in the directory named WEB-INF/data.

The syntax to run this utility is:

Windows platforms - 

FullTextIndexSetup.bat JAVA_HOME TOMCAT_HOME EV_BASE [app_server_name] [-no_reset] [-alternate altIndexDir]

Linux platforms - 

FullTextIndexSetup.sh JAVA_HOME TOMCAT_HOME EV_BASE [app_server_name] [-no_reset] [-alternate altIndexDir]

  • JAVA_HOME is the path to your Java
  • TOMCAT_HOME is the path to your application server
  • EV_BASE is the path to the ExtraView installation
  • app_server_name is the name of the application server node on which the indexes are to be stored within the path defined by the behavior setting named QUICKFIND_INDEX_LOCATION.   Quickfind only runs on one node within a multi-node environment and this parameter specifies which node is to be used.  If you are running with a single application server then ExtraView will ascertain the name of the application server and this parameter is not needed
  • It is likely that you will have environment variables set for JAVA_HOME and TOMCAT_HOME, and possibly EV_BASE but these can be set in the batch file or shell script file if needed
  • -alternate altIndexDir provides a location for an alternate index to be used, when upgrading the Quickfind software.  The location for the index is taken from the behavior setting named QUICKFIND_INDEX_LOCATION if this is not set
  • -no-reset allows for a restart of the process in the case where processing was interupted for any reason.

Updating Quickfind

From time-to-time ExtraView recommends that the libraries be updated, for bug fixes and performance enhancements.  Unfortunately, the Apache Foundation do not always maintain backwards compatibility with the format of the indexes that are stored.  The re-indexing of the Quickfind indexes within ExtraView may take a long time with large databases, and it can introduce unacceptable downtime if a system is required to be kept offline while the re-indexing occurs.

To mitigate this, ExtraView Corporation has built a methodology that allows the new indexes to be built whilst the old ones are maintained in the live system, and for a switch to the new indexes being made at any point in time, without the loss of any indexing that's completed or in progress.

The process is this, assuming that the active index is set to a value of PRIMARY.  At the outset, your behavior settings will look something like this:

  • QUICKFIND_ACTIVE_INDEX - PRIMARY
  • QUICKFIND_INDEX_LOCATION - /usr/locn/indexes

We want to set up an alternate set of indexes in the new location of /usr/locn2/indexes and switch to using these for the future.

Step 1

Backup the existing indexes stored in the location /usr/locn/indexes.

Step 2 - Process for Large Installations

For smaller databases where you can stop the ExtraView server while you perform the update, see the alternate step 2.

This creates a new set of indexes in the location /usr/locn2/indexes while your ExtraView installation continues with normal operations.  On large sites this may take many hours, or even days.  This is accomplished external to ExtraView with the FullTextIndexSetup utility so that the normal functionality of ExtraView is not disrupted while the re-indexing is underway. 

Run the FullTextIndexSetup as described in the previous section for this step, but use the -alternate altIndexDir parameter to build the new indexes in the path /usr/locn2/indexes.

Step 2 - Alternate Process for Smaller Installations

Stop the application server - usually Tomcat.  Reindex using the FullTextIndexSetup process.  Restart the application server.  Note that users cannot access the installation during this time.  Step 3 is unneeded if you choose this step.

Step 3

Once the FullTextIndexSetup is complete, you will switch ExtraView to point to the newly built indexes.  There is not a problem if there is a delay between building the new indexes and switching from the old ones as the indexing task recognizes the precise point in time at which the new indexes were built, and will simply add to the new indexes any information that needs to be processed in the time gap between the completion of building the new indexes and the current point in time on the live system.

To point to the new indexes, alter the two behavior settings as follows:

  • QUICKFIND_ACTIVE_INDEX - ALTERNATE
  • QUICKFIND_INDEX_LOCATION - /usr/locn2/indexes

The Quickfind task wil immediately point to the new index location and will process any issues to bring the new indexes completely up-to-date.