CJK with Solr for Libraries, part 5 (Edismax Woes, part 2)

This is the fifth of a series of posts about our experiences addressing Chinese, Japanese and Korean (CJK) resource discovery in SearchWorks, the Stanford University Libraries "catalog" built with Blacklight on top of our Solr index, and the second in the sub-series on problems we had switching to edismax from dismax.

You might be interested in this post in particular if you use Solr's edismax query parser or if you want to get more exact matches for user queries.

Edismax Woes, Part 2

In the third part of this series, I mentioned that we had 21 relevancy acceptance tests fail when we used the edismax Solr query parser instead of dismax.  Recall that our test failures were in the following categories:

1. Journal titles
2. Hyphens preceded by a space (but no following space)
3. Boolean NOT
4. Synonyms for musical keys

This blog post will discuss how we addressed relevancy failures in the first category;  I discussed failures in categories 2 and 3 in the fourth part of this series.

Digging Into Relevancy Differences

Here is example output from some failing tests in category 1:

  rspec ./spec/journal_title_spec.rb:22 # journal titles The Nation as everything search

  rspec ./spec/journal_title_spec.rb:32 # journal titles The Nation (National stems to Nation) with format journal

I used manual searches to confirm that users would perceive the edismax results as worse than dismax.  I also tried a number of similar searches (and wrote tests) to better pinpoint this problem.

Here are the first five results for a title search on The press using using dismax and edismax with the same index:

Looking at these results, I immediately have two questions:

1. Why are the first two results of edismax not exact title matches?
2. Why are the scores of the three documents that appear in both sets of results different?

Edismax Formula Difference

To answer these questions, I first looked at the Solr output with debugQuery=true.  The Solr query analysis was shown as exactly the same EXCEPT for this difference noted in SOLR-2058 in the comment from Michael Dodsworth on 25 Sep/12.

I'll try to express it more succinctly (thanks to Tom Burton-West) :

Let
  A = field1:"term1 term2"
  B = field2:"term1 term2"
  C = field3:"term1 term2"

Dismax:    DisjunctionMaxQuery (A|B|C)
  returns the score of whatever field has the highest score.

Edismax:
    DisjunctionMaxQuery (A)
    DisjunctionMaxQuery (B)
    DisjunctionMaxQuery (C)
  returns the sum of the scores of each of any of the above queries that match.  So if your phrase is in all 3 fields, you get the sum of the scores for each matched field.

Unfortunately, SOLR-2058 is marked as fixed, despite this difference.  So we have an initial diagnosis, but not a treatment plan.

Visualizations To The Rescue!

I was stumped as to what to do about the above until I used the visualizations of Solr scoring data made available at explain.solr.pl.  The full visualizations I created for scoring the top 5 results of a search for 'the press' are available here:

edismax:  http://explain.solr.pl/explains/m63o1yhg
dismax:  http://explain.solr.pl/explains/a7bkurhb

Here is the visualization of the score of the first dismax result, id 9162486 with title The press:

The score is overwhelmingly dominated by the phrase match in the unstemmed short title field, title_245a_unstem_search.  In fact, this is true for all the top 5 dismax results:  if the pie charts below weren't different colors and didn't have text labels for the tiny pie pieces, I think you'd be hard pressed to tell them apart:

However, with edismax, we have two basic patterns within the first 5 results, and the phrase match in the unstemmed short title field is not nearly as dominant:

Here is a closer look at the visualization of the score of the first edismax result, id 8192320 with title MEET THE PRESS:

What I learned from these visualizations is that to make the edismax results more like dismax, I needed a way to give more weight to exact matches of the entire string in the title_245a_unstem_search field.

Another visualization example, of the top 5 results of a search for 'the nation':

edismax:  http://explain.solr.pl/explains/6bracmzw
dismax:  http://explain.solr.pl/explains/4med7pae

Tie Parameter

"DisMax" is an abbreviation of "disjunction maximum", which is a partial description of the way user queries are turned into low level Lucene queries.  From http://wiki.apache.org/solr/DisMax, dismax is "designed to process simple user entered phrases (without heavy syntax) and search for the individual words across several fields using different weighting (boosts) based on the significance of each field."  From the same document:

So basically, dismax pays attention only to the highest scoring query match in any of the document's fields.  The tie parameter, documented here, can be used to dial up the influence of other matches:

A tie value of 0.01 was used for both dismax and edismax searches.  That value is very close to zero, so the highest scoring matching clause should already be dominating the total score.  I tried a tie value of 0.99 to see if I could make a difference in the relevancy ranking this way, and while my document scores changed, the result order remained the same.   Below are results for edismax search for 'the press' with different tie values:

tie 0.01:  http://explain.solr.pl/explains/hwi1ma6x
tie 0.99:  http://explain.solr.pl/explains/jfmdli56


I coerced a colleague into trying a similar experiment, and was able to confirm the tie parameter affected his results as expected.  Further experimentation indicated that setting the tie value to something like 0.00001 for my search DID have the effect of making the highest scoring matching clause highly dominant;  this led me to the conclusion that my field boosting values were ridiculous (as high as 200,000 !) and affected my ability to use a reasonable value for the tie parameter..

Re-adjusting Boost Values

Our boost value settings were a big nasty kludge that I partly inherited and partly created. Simplifying them was definitely in order ... but perhaps not at the same time as a change to edismax, nor during a Solr upgrade from 3.5 to Solr 4.  Sure, having about 600 relevancy tests gave me a decent amount of confidence I could revise my boost values without degrading the user experience of relevancy, but upgrading to edismax was causing me plenty of difficulty without revising boost values.

Eventually, I did greatly simplify our boost values and our relevancy tests were indispensable to that process.   Of course I de-coupled this from fixing the problems we had with edismax, and again from the Solr upgrade from 3.5 to Solr 4.  You can see our simplified boost values in our solrconfig.xml file at https://github.com/solrmarc/stanford-solr-marc/blob/master/stanford-sw/solr/conf/solrconfig-slave.xml.   The highest boost value is now 5,000, a great reduction from 200,000.

However, in terms of addressing the relevancy test failures with edismax, adjusting the boost values and the tie parameter wasn't really part of the route taken.

Back to Edismax Woes

The visualizations of Solr scores helped us see that our edismax results need to give more weight to exact matches of the entire string in the short title (title_245a_unstem_search) field.  The tie parameter wasn't really working for us.  What now?

Bill Dueber of the University of Michigan wrote a wonderful blog post on using "fully-anchored" text fields to get an "exactish" match.   This approach seemed well worth a try.  I will discuss our fully-anchored solution in the next post in this series.

CJK with Solr for Libraries, part 4 (Edismax Woes, part 1)

This is the fourth of a series of posts about our experiences addressingChinese, Japanese and Korean (CJK) resource discovery in SearchWorks, the Stanford University Libraries "catalog" built with Blacklight on top of our Solr index.

You might be interested in this post in particular if you use Solr's edismax query parser.

Edismax Woes, Part 1

In the third part of this series, I mentioned that we had 21 relevancy acceptance tests fail when we used the edismax Solr query parser instead of dismax.   If I inserted the "e" in my solrconfig, I would see these test failures;  if I removed the "e", the tests would pass.  I had heard "edismax is better", but here I had proof positive that edismax was WORSE.

Our relevancy problems with edismax were unexpected blockers to improving CJK resource discovery:  recall that edismax is required to fix relevancy when using the CJKBigram filter (SOLR-3589) as mentioned in part two of this series.

I believe the intent is for edismax to be an exact equivalent of dismax with additional features ... but this is not true at this time.  It turns out that edismax has a number of unresolved bugs and unimplemented features (see SOLR-2368).

Recall that our test failures were in the following categories:

1. Journal titles
2. Hyphens preceded by a space (but no following space)
3. Boolean NOT
4. Synonyms for musical keys

I will discuss categories 2 and 3 in this blog post.

Edismax Bug with Boolean Operators

One of the unresolved bugs with edismax (see SOLR-2368 for a comprehensive list) is SOLR-2649, "MM ignored in edismax queries with operators."  This bug essentially means that if a Boolean operator, such as NOT or OR appears in the query string, then all terms in the query are effectively "OR'ed" together.  Note that 'AND' is unaffected;  the four operators affected are:  NOT, OR, - (prohibited), + (required). 

It turns out that all of our failing hyphen tests were queries with a space before but not after the hyphen, such as 'under the sea -wind.'   Solr interprets such a hyphen as a "prohibited" operator, which is the same as NOT, and such hyphens are included in bug SOLR-2649.  Thus, our failure categories 2 and 3 are essentially the same. 

Our default operator is AND and we set lowercaseOperators to false (see http://wiki.apache.org/solr/ExtendedDisMax#lowercaseOperators), so the SOLR-2649 bug means any query with terms of uppercase OR or NOT or a '-' or '+' character preceding a term gives unexpected results, unless the query has either explicit AND or explicit '+' for all other terms.  Our mm setting is 6<-1 6<90%, or high enough that 4 terms should be AND'ed together.

Here are some illustrative example user queries, how they are analyzed by Solr (via debugQuery=true), and the results:

DISMAX:
  q=customer driven academic library
    +(((custom)~0.01 (driven)~0.01 (academ)~0.01 (librari)~0.01)~4) ()
    4 hits
  q=customer NOT driven academic library:
    +(((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01)~3) ()
    96 hits
  q=customer -driven academic library:
    +(((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01)~3) ()
    96 hits
  q=customer academic library:
    +(((custom)~0.01 (academ)~0.01 (librari)~0.01)~3)()
    100 hits

EDISMAX
  q=customer driven academic library:
    +(((custom)~0.01 (driven)~0.01 (academ)~0.01 (librari)~0.01)~4)  
    4 hits
  q=customer NOT driven academic library:
    +((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01)  
    984300 hits
  q=customer -driven academic library:
    +((custom)~0.01 -(driven)~0.01 (academ)~0.01 (librari)~0.01)
    984300 hits
  q=customer OR academic OR library NOT driven:
    +((custom)~0.01 (academ)~0.01 (librari)~0.01 -(driven)~0.01)
    984300 hits
  q=customer academic library:
     +(((custom)~0.01 (academ)~0.01 (librari)~0.01)~3)
    100 hits

You can see that the mm is missing from the middle 3 edismax queries (no ~3 term);  this is the manifestation of the bug.  You can also see how the number of results would be confusing to an end user.

Great - now we have learned there is a known Solr bug, and we understand its scope.  What shall we do?  The first, best option would be for someone else to fix the problem.  Given that the bug report is from July 2011 and heavily commented, that seems unlikely to happen soon.  The next best option is to fix the problem myself.  However, having already looked at the source code for edismax, I pale at the very thought.  

So I took a different tack:  I examined whether we had a significant number of user queries impacted by this bug.

Will Our Users Encounter this Bug?

Prior to the CJK improvements, SearchWorks has only supported boolean operators in its Advanced search.  Our usage data from July 2013 - November 2013, shows "advanced" is the initial search behavior somewhere between 6% and 18% of the time.   For example, here are stats for initial search behavior in July 2013:

And the analogous data for November 2013:

So let's be generous, and say that 20% of the SearchWorks queries use the boolean-enabled advanced searches.  Of those, how many contain boolean?

We looked at 10,000 advanced search queries culled from Google Analytics from July, 2013.  (Note that this could have been done via the Solr logs, but we have a load balanced Solr set up so it would have involved getting logs from 3 machines and their backups ... so we just went with Google Analytics.)  Using grep, we determined the following occurrences of the affected operators in an advanced search where SOLR-2649 might apply:

Thus, the boolean operators affected by SOLR-2649 occur extremely rarely in SearchWorks advanced search queries, which are themselves, at best 20% of our queries.

Armed with this data, we chose to let this remain broken in SearchWorks for the time being, so the failing relevancy tests in categories two and three above have been resolved as "do not fix" until the Solr bug is addressed.

Stay Tuned ...

In my next post, I will tackle the first category of failing relevancy tests, which required a work around to avoid degrading relevancy for our users when we switched to edismax.

CJK with Solr for Libraries, part 3

This is the third of a series of posts about our experiences improving CJK resource discovery for the Stanford University Libraries.

We recently rolled out some significant improvements for Chinese, Japanese and Korean (CJK) resource discovery in SearchWorks, the Stanford library "catalog" built with Blacklight on top of our Solr index. If your collection has a significant number of CJK resources and they are in multiple languages, you might be interested in our recipes. You might also be interested if you have a significant number of resources in other languages with some of the same characteristics.

Relevancy Testing

In the second part of this series, I explained why SearchWorks needed to change from using the Solr dismax query parser to the edismax query parser.   I would not undertake nor recommend a fundamental change to your Solr query processing without a good testing methodology.  Nor would I change an index to accommodate CJK without a way to ensure it didn't break existing functionality.  Basically, it's a bad idea to change anything about query processing without a way to ensure it doesn't degrade existing relevancy.  This implies automated relevancy acceptance testing is needed.

Luckily for us, we have been doing automated relevancy testing for a while, first by using cucumber tests within our SearchWorks Blacklight Rails application, and now much more efficiently by using rspec-solr (http://rubydoc.info/github/sul-dlss/rspec-solr) to interact with Solr directly instead of going through the whole Rails application.  It allows our SearchWorks relevancy testing application, sw_index_tests (available at https://github.com/sul-dlss/sw_index_tests), to parse Solr responses and use rspec syntax to check whatever we want about the Solr documents returned, without going through the entire Rails stack.  I blogged about this a while back.

When we were about to switch to edismax to facilitate CJK discovery, we had around 580 relevancy tests in sw_index_tests, including tests for everything, author, title, subject and series search results, diacritics and punctuation in search terms, and journal titles, among other things.  These test searches (and their expected results) were amassed over a period of 3-4 years:  tests were written every time a tweak was made to address a problem, or when the indexing code changed for some other reason (e.g. providing call number searching).  These tests have never been comprehensive, but they are a lot better than nothing.  We run them against our live production index nightly via our Jenkins continuous integration server.  Sometimes we have to tweak the tests when records are added, removed, or changed in the production index, but that's easy.  The peace of mind knowing we have a way to do relevancy acceptance testing is well worth the trouble.

And in case you're wondering, we pass an additional http argument (testing=sw_index_tests) to Solr so we can easily segregate these test queries from actual user queries in the Solr logs:

Here are a few example tests from sw_index_tests:

Title:

Author: 

Journal Title: 

Diacritics: 

I will talk more about our CJK relevancy tests later;  my main point here is that we have automated tests to help us determine if anything breaks when we make changes to our Solr index or configurations, and you can do it too!   Heck, you can even use ours and change the acceptance conditions to be your ids and expected result numbers.

I would love to hear from anyone else who does automated relevancy testing, as it seems to be a rare thing.

Edismax != Dismax

Technically, to switch from the dismax query parser to the edismax query parser, you need only add the "e" to your Solr request handler defType declaration:

<requestHandler name="/search" class="solr.SearchHandler">
  <lst name="defaults">
    <str name="defType">edismax</str>

When we tried this, we had 21 failures out of approximately 580 tests.  The failures were in four categories:

1.  Journal title failures

Example of a failing test:

  rspec ./spec/journal_title_spec.rb:22 # journal titles The Nation as everything search
  rspec ./spec/journal_title_spec.rb:32 # journal titles The Nation (National stems to Nation) with format journal

2.  Queries having hyphens with a preceding space (but no following space)

Example of a failing test:

  ./spec/punctuation/hyphen_spec.rb:140 # 'under the sea -wind' hyphen in queries with space before but not after are treated as NOT in everything searches 

3.  Boolean NOT operator

Failing tests:

  ./spec/boolean_spec.rb:100 # boolean NOT operator  space exploration NOT nasa has an appropriate number of results
  ./spec/boolean_spec.rb:88 # boolean NOT operator  space exploration NOT nasa  should have fewer results than query without NOT clause

4.  Synonyms for musical keys.

We use Solr synonyms to equate the following (for all musical keys - the full list is here):

  f#, f♯, f-sharp => f sharp
  ab, a♭, a-flat => a flat

Failing tests:

  ./spec/synonym_spec.rb:204 # musical keys sharp keys f# major
  ./spec/synonym_spec.rb:316 # musical flat keys author-title search (which is a phrase search) b♭

Clearly, we needed to address these problems before we could use edismax in production SearchWorks, which means they needed to be fixed as a prerequisite for improving CJK discovery.

How to Analyze Relevancy Problems

Thankfully, there are some excellent tools for debugging relevancy problems.

1.  Solr query debugging parameters 

If you add debugQuery=true to your Solr request, then you will get debugging information in your Solr response.  If you are at Solr release 4.0 or higher, you could use debug=query instead.  Here is an example:
My request:

 http://(solr baseurl)/solr/select?q={! qf=$qf_author}zaring&debug=query

The debug query part of the response (simplified a bit):

We can see exactly which Solr fields and terms are being searched, and their boost factors.  This example shows different terms being searched in the stemmed and unstemmed version of the fields.  (Note: the decision to stem author fields was deliberate to allow users to find, say "Michaels, Amanda" when they query "Amanda Michael", or if they use "Crook" when the name is actually "Crooke".)

See http://wiki.apache.org/solr/CommonQueryParameters#Debugging for more information.

2.  Analysis GUI

Another Solr supplied tool is the Analysis form in the admin GUI.   This tool lets you see how each part of your analysis chain of tokenizer and filters affects the data according to field, field type, or dynamic rule in the Solr schema.

This shows, for Solr field author_1xx_search, at which point in the analysis chain "zaring" becomes "zare".   I entered "zare" as a query value, and the faint purple highlighting of the bottom two lines on the left field value side shows that zare and zaring will match for field author_1xx_search. 
Note that this is not an exact representation of query matching.  As an example, the lucene query parser on the client side breaks things up by whitespace before field analysis is performed for (e)dismax processing.

3.  Visualization of Individual Result Debug Information

The Solr debug information can also contain information on how the algorithm computed the relevancy ranking of a results, either with debugQuery=true, or debug=results for Solr 4.0 or higher.

Given the same Solr query string as above, with debug=results:

 http://(solr baseurl)/solr/select?q={! qf=$qf_author}zaring&debug=results

The explain part of the response (simplified a bit):

<lst name="debug">

... 

<lst name="explain">

  <str name="3928423">
9.0780735 = (MATCH) sum of:
  9.0780735 = (MATCH) max plus 0.01 times others of:
    9.056876 = (MATCH) weight(author_1xx_unstem_search:zaring^20.0 in 1075085) [DefaultSimilarity], result of:
      9.056876 = score(doc=1075085,freq=1.0 = termFreq=1.0
), product of:
        0.999978 = queryWeight, product of:
          20.0 = boost
          14.491322 = idf(docFreq=9, maxDocs=7231138)
          0.0034502652 = queryNorm
        9.0570755 = fieldWeight in 1075085, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          14.491322 = idf(docFreq=9, maxDocs=7231138)
          0.625 = fieldNorm(doc=1075085)
    2.1197283 = (MATCH) weight(author_1xx_search:zare^5.0 in 1075085) [DefaultSimilarity], result of:
      2.1197283 = score(doc=1075085,freq=1.0 = termFreq=1.0
), product of:
        0.24188633 = queryWeight, product of:
          5.0 = boost
          14.021318 = idf(docFreq=15, maxDocs=7231138)
          0.0034502652 = queryNorm
        8.763324 = fieldWeight in 1075085, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          14.021318 = idf(docFreq=15, maxDocs=7231138)
          0.625 = fieldNorm(doc=1075085)
</str>

This can be useful in determining why a particular document is (or isn't) included in the results, but it is difficult to eyeball the above and understand what is going on, even after you format it.

Thankfully, there is a web site in Poland, http://solr.pl/en/, that has a web service, http://explain.solr.pl/, to take your Solr explain info and visualize it as a pie chart.  This presents our result like this:

Suddenly, it is obvious why this document matches.

This is tool is even more useful for more complex (e)dismax formulae with a lot of fields to match, multi-term queries and documents matching different terms in different fields.  Check out some of our actual results while debugging the edismax difficulties here:

edismax:  http://explain.solr.pl/explains/m63o1yhg
dismax:  http://explain.solr.pl/explains/a7bkurhb

Stay Tuned ...

Now that I've explained our testing methodology and some of our debugging techniques, I'm ready to tell you how we overcame the relevancy issues we bumped into when switching to edismax.  That will be the topic of my next post(s).

CJK with Solr for Libraries, part 2

This is the second of a series of posts about our experiences improving CJK resource discovery for the Stanford University Libraries.

We recently rolled out some significant improvements for Chinese, Japanese and Korean (CJK) resource discovery in SearchWorks, the Stanford library "catalog" built with Blacklight on top of our Solr index. If your collection has a significant number of CJK resources and they are in multiple languages, you might be interested in our recipes. You might also be interested if you have a significant number of resources in other languages with some of the same characteristics.

What Solutions Are Out There?

Of course our first thoughts on how to fix our CJK discovery woes were to find existing solutions we could use.  Solr and Lucene are used widely for many different languages, so we hoped to find great ready-made solutions.

Solr Analyzers for Japanese and Chinese

Solr currently ships with a Kuromoji Japanese morphological analyzer/tokenizer (shown in the Solr example schema), and there is also support for Simplified Chinese word segmentation.  Both of these language specific analyzers are mentioned in the README.txt of the Lucene analysis module.  It is possible there is a Korean Solr analyzer available as well, though it might take someone fluent in Korean to find it on the internet - it is not currently part of the Solr distribution files.

Utilizing language-specific analyzers may make sense if you can ask users to indicate language at query time, perhaps by selecting from a small pulldown list.  For example, we could run all our vernacular language metadata through a Japanese analysis, a Chinese analysis, and a Korean analysis and send user queries to the appropriate set of language-specific indexed fields, based on user indication of language.  Given the number of different languages with materials in SearchWorks, we would either have to present our users with a very long list of languages to select from or we would need to navigate a political storm to determine which languages made the short list in the UI.  Neither of these options were desirable for us.

Script Translations

Since Solr release 3.1, the Solr code has been able to utilize some of the Unicode support java libraries from the International Components for Unicode (ICU) project.  There is a Solr tokenizer available, solr.ICUTokenizer as well as some Solr filters for field type analysis: ICU collation, ICU character normalization/folding, and some Unicode script translations.

Han Traditional <--> Simplified

As mentioned in the first part of this series, the top priority for Chinese discovery improvements is to equate Traditional Han script characters with simplified Han script characters.  Similarly, the top priority for Japanese discovery improvements is to equate Modern Kanji (Han) characters with Traditional Kanji characters.  The ICU script translations include a translation to equate Traditional Han with Simplified Han.  In Solr, it could be specified as shown in the filter example:

So the next question is:  should we be translating from traditional to simplified, or from simplified to traditional?

Since multiple traditional characters can map to the same simplified character, we are likely to get the best recall mapping from traditional to simplified than from vice versa.  Some precision may be lost going from traditional to simplified, but our CJK language experts preferred this approach - they would rather get more results with some of them irrelevant than miss results.  As it happens, this approach is also taken by our ILS, which is Symphony by Sirsi/Dynix.

Katakana <--> Hiragana

The second priority for Japanese discovery improvements is to equate all scripts:  Kanji, Hiragana, Katakana and Romanji.  The only other relevant ICU script translation available is a mapping between Hiragana and Katakana.  This is a straightforward one-to-one character mapping, so it doesn't matter which direction the translation is done, as long as it is consistent between the query and the index.  The Solr filter could look like this:

Other CJK Script Translations?

The additional translations we would like would be Hangul <--> Han for Korean, Kanji/Han <--> Hiragana (or Katakana), and Japanese Romanji to one of the other Japanese scripts.  Unfortunately, the Solr use of ICU only supports ICU System transforms at this time, and none of these translations are included.  ICU itself supports user-supplied transforms, but Solr use of ICU does not.

Still, covering the top discovery priorities for Japanese and Chinese with out-of-the-box Solr components is a huge win.

Multi-lingual Solutions

As mentioned above, language-tailored text analysis is not a good solution for SearchWorks.  So we must take a multi-lingual approach to solving CJK discovery.

We were already acquainted with the Arcadia funded 2010 Yale University report "Investigating Multilingual, Multi-script Support in Lucene/Solr Library Applications" by Barnett, Lovins, et. al., (https://collaborate.library.yale.edu/yufind/public/FinalReportPublic.pdf), which explains the problem and suggests some approaches, but gives no test-kitchen-approved recipes.  I conferred with a number of developers working in libraries with large collections of Asian materials using Solr, but all the folks I talked to hadn't tackled this yet and their thoughts of how they planned to do it mirrored my own. It was also suggested repeatedly that I talk to Tom Burton-West of the Hathi Trust Digital Library (http://www.hathitrust.org), which is a very large, full-text multi-lingual digital library containing a significant body of CJK full-text material.

Tom Burton-West was amazingly helpful, and had already documented a number of relevant issues in a blog post in December 2011:  http://www.hathitrust.org/blogs/large-scale-search/multilingual-issues-part-1-word-segmentation.  Tom's research-backed post suggests that the best way to work simultaneously with multiple CJK languages would be indexing with a combination of unigrams and of overlapping character bigrams. As an example, if the original characters are ひらがな then the unigrams would be ひ, ら, が, な and the bigrams would be ひら, らが, and がな, and all seven of these tokens would be in the index.  If the index only used bigrams, it would not find unigram words that have no whitespace on either side, and if the index only used CJK unigrams, then it would produce too many false drops.

Solr CJKBigram Analyzer

Handily enough, Solr makes available a CJKBigramFilter, which creates overlapping bigrams when it encounters adjacent CJK characters (it creates unigrams for CJK characters that aren't adjacent).   The example schema provided with Solr even shows one way you might configure Solr to use it:

(This is not the way we ultimately configured our CJK field type; I will share that in a subsequent post.)

Hooray!   We've got two script translations and a way to do multi-lingual CJK bigramming -- all with Solr out-of-the-box components!  We're ready to go! 

Except we're not.

There was also a bug in Solr affecting the search results of bigrammed fields when using the "dismax" or "edismax" query parser.  (https://issues.apache.org/jira/browse/SOLR-3589).  The Solr dismax and edismax query parsers provide a way to search individual words across a combination of indexed fields and boost values. "Dismax" is an abbreviation of "disjunction maximum", which is a partial description of the way user queries are turned into low level Lucene queries. "Edismax" is an abbreviation for "extended disjunction maximum", where the extensions include improved punctuation handling and boolean syntax among other things.

SearchWorks at this time used the dismax query parser, so this Solr bug affected our CJK search results when we tried using the CJKBigramFilter. In essence, this bug meant that all the tokens created by the filter would be combined into a big boolean OR statement. Using our example above, with a user query of ひらがな, we would get results as if the query was this:  ひら OR らが OR がな OR ひ OR ら OR が OR な.   Clearly, this would bloat the results returned and reduce precision.

I created some test indexes, and worked closely with Vitus Tang, a Stanford metadata expert fluent in Chinese, to confirm SOLR-3589 made our CJK search results unacceptably bad.  I looked at the Solr source code and determined that I was unlikely to be able to fix the bug myself.  I thought of and tried a bunch of workarounds for this bug, but none of them produced acceptable search results.  I even thought about substituting a multilingual dictionary in one of the existing Chinese or Japanese tokenizers, but there was no easy way to modify the dictionary of either one of these analyzers.  I also conferred repeatedly with Tom Burton-West and others, but this bug was stumping us.

Lucky for us, as we asked around wondering who had the chops to fix the bug, Solr expert Robert Muir stepped up and fixed SOLR-3589 for Solr version 4.x and for the "edismax" query parser.  We were using Solr 3.5 and the "dismax" query parser at this point.  Tom Burton-West was kind enough to backport the fix to Solr version 3.6, but it still only worked for the edismax query parser.  Both he and I found it too daunting to port the fix to the dismax query parser.

At this point, the best solution was for SearchWorks to upgrade to Solr 4 or at least to Solr 3.6 with the patch, and to switch to the edismax query parser.   This was not a simple journey, as I will explain in a subsequent post in this series.

CJK with Solr for Libraries, part 1

This is the first of a series of posts about our experiences improving CJK resource discovery for the Stanford University Libraries.

We recently rolled out some significant improvements for Chinese, Japanese and Korean (CJK) resource discovery in SearchWorks, the Stanford library "catalog" built with Blacklight on top of our Solr index.   If your collection has a significant number of CJK resources and they are in multiple languages, you might be interested in our recipes.  You might also be interested if you have a significant number of resources in other languages with some of the same characteristics.

Disclaimer: I am not knowledgeable about Chinese, Japanese or Korean languages or scripts -- the below is an approximate explanation meant only to illustrate the complexities of CJK resource discovery.

Why do we care about CJK resource discovery?

Stanford University Libraries has over 7 million resources in SearchWorks;  over 450,000 of them are shown as resources in Chinese, Japanese, or Korean:

Of these CJK records, 85% have vernacular scripts in the metadata:

We want to leverage the CJK vernacular text to improve resource discovery for our CJK users.

Why approach CJK resource discovery differently?

1.  Meaningful discovery units (words) are not necessarily separated by whitespace in CJK text.

• Solr/Lucene has some baked in assumptions about whitespace separating words. It'sasifthetextalwayslookslikethisbut the software expects it to look like this. 
• This is true for user behavior as well as for resources.

2.  Search results must be as script agnostic as possible.

Chinese, Japanese and Korean each have multiple scripts or multiple character representations for each word ... and search results should include matches from all of them.

Chinese

Uses Han script only, BUT:

• There is more than one way to write each word. "Simplified" characters were emphasized for printed materials in mainland China starting in the 1950s;  "Traditional" characters were used in printed materials prior to the 1950s, and are still used in Taiwan, Hong Kong and Macau today.  Since the characters are distinct, it's as if Chinese materials are written in two scripts.  
• Another way to think about it:  every written Chinese word has at least two completely different spellings.  And it can be mix-n-match:  a word can be written with one traditional  and one simplified character.
• Example:   Given a user query 舊小說  (traditional for old fiction), the results should include matches for 舊小說 (traditional) and 旧小说 (simplified characters for old fiction)

Japanese

Mainly uses three scripts:

• Han ("Kanji")
• Kanji characters can be "traditional" or "modern," akin to Chinese "traditional" and "simplified."  However, given a traditional Han/Kanji character, the corresponding Kanji modern character is not always the same as the Han simplified character.
• That is, "some of the Chinese characters used in Japan are neither 'traditional' nor 'simplified'. In this case, these characters cannot be found in traditional/simplified Chinese dictionaries."  from http://en.wikipedia.org/wiki/Simplified_Chinese_characters#Computer_encoding
• Note:  Kanji characters are still actively used in contemporary writing.
• Hiragana
• syllabary used to write native Japanese words.
• Katakana
• syllabary primarily used to write foreign language words.

Also makes some use of

• Latin ("Romanji")

Korean

Uses two scripts:

• Han ("Hanja")
• some Hanji characters are still actively used by South Koreans.
• Hangul
• in widespread use;  was promulgated in the mid 15th century.

Note:  Han script is used by all three CJK languages BUT:

• the meaning of the characters is not necessarily the same in the different languages.
• you can't translate Han characters for one language without potential degradation of results in the other languages.

3.  Multilingual indexes can't sacrifice, say, Japanese searching precision in favor of Chinese searching precision. 

4.  Automatic language detection is not possible.

• script detection isn't sufficient.  Example: a record has Latin and Han characters.  Is it Japanese?  Or English and Chinese?   Or English and Korean?  Or English and Japanese?
• the indicated language(s) in a MARC record may be insufficient.  For example, the record may be a Korean record for a resource that is mostly in Chinese.  
• the user queries are short:  90% of our CJK queries are less than 25 characters; 50% have 12 or fewer chars.   (Evidence of this will be shown in another part of this series on CJK.)
• the amount of CJK text in an individual record may also be too small.

5.  Artificial spacing may be present in Korean Marc records. 

Cataloging practice for Korean for many years was to insert spaces between characters according to "word division" cataloging rules (See http://www.loc.gov/catdir/cpso/romanization/korean.pdf, starting page 16.)  End-users entering queries in a search box would not use these spaces.  It 's analog ous to spac ing rule s in catalog ing be ing like this for English.

CJK Discovery Priorities

Given the difficulties above, we asked our East Asia Librarians what their priorities were for discovery improvements.

Chinese

1.  Equate Traditional Characters With Simplified Characters

About half of our Chinese resources are in traditional characters, the other half are in simplified characters.  Queries can be in either traditional or simplified characters, or a combination of the two;  search results should contain all matching resources, whether traditional or simplified.

2.  Word Breaks

Search results should match conceptual word breaks whether or not whitespace is used to separate words in the results or the query.

Japanese

1.  Equate Traditional Kanji Characters With Modern Kanji Characters

Kanji (Han) Queries can be in either traditional or modern characters, or a combination of the two;  search results should contain all matching resources, whether traditional or simplified.  It is important to restate that Modern Kanji characters are not always the same as Simplified Han characters for the equivalent traditional character.

2.  Equate All Scripts

Search results should contain matches in all four scripts: Hiragana, Katakana, Kanji or Romanji.  Queries can be in any script, or any combination of scripts.

3.  Imported Words

Japanese represents some foreign words with Romanji and/or Katakana:
"sports" --> "supotsu" <==> スポーツ
Search results should contain matches in all representations and allow queries in any representation.

4.  Word Breaks

Search results should match conceptual word breaks whether or not whitespace is used to separate words in the results or the query.

Korean

1.  Word Breaks

Search results should match conceptual word breaks whether or not whitespace is used to separate words.

2.  Equate Hangul and Hancha Scripts

Search results should contain matches in both scripts: Hangul and Hancha (Han).  Queries can be in any script, or any combination of scripts.

Next ...

We'll be looking at what Solr offers in the way of CJK tools, and some of the recently fixed and current Solr bugs that get in the way, including two that almost sunk us. We'll also examine what current CJK queries look like and where the CJK characters are in our Marc data. And of course we'll cover our testing methodology and the final recipes. No guarantees on the order of these topics!

Solr Acceptance Tests: introducing rspec-solr (and sw_index_tests)

I've released the rspec-solr ruby gem, which applies RSpec custom matchers to Solr responses.  Rdoc is at http://rubydoc.info/github/sul-dlss/rspec-solr  and the source code is at https://github.com/sul-dlss/rspec-solr/ .

It's pretty simple:  once you have a ruby Solr response, you wrap it in RSpecSolr:

 resp = RSpecSolr::SolrResponseHash.new(yer_solr_resp)

and then you can make useful assertions for acceptance testing:
  

 resp.should include({'id'=>'81234'})

 resp.should include({'title'=>'Harry Potter'}).in_first(3).results

 resp.should include('111'}).before('222')

So you might write specs like this:

it "q of 'Buddhism' should get 8,500-10,500 results" do
  resp = solr_resp_doc_ids_only({'q'=>'Buddhism'})

  resp.should have_at_least(8500).documents
  resp.should have_at_most(10500).documents
end

 it "q of 'Two3' should have excellent results", :jira => 'VUF-386' do
   resp = solr_resp_doc_ids_only({'q'=>'Two3'})
   resp.should have_at_most(10).documents
   resp.should include("5732752").as_first_result
   resp.should include("8564713").in_first(2).results
   resp.should include("5732752").before("8564713")
   resp.should_not include("5727394")
   resp.should have_the_same_number_of_results_as(solr_resp_doc_ids_only({'q'=>'two3'}))
   resp.should have_fewer_results_than(solr_resp_doc_ids_only({'q'=>'two 3'}))
 end

 it "Traditional Chinese chars 三國誌 should get the same results as simplified chars 三国志" do
   resp = solr_resp_doc_ids_only({'q'=>'三國誌'})  
   resp.should have_at_least(240).documents
   resp.should have_the_same_number_of_results_as(solr_resp_doc_ids_only({'q'=>'三国志'})) 
 end

Note that these examples utilize a couple of helper methods.  See the README for more details.

The gem is only at release 0.1.0, but I'm finding it useful already.  You'll see some FIXME and TODO comments, and I suspect there's plenty that can be improved.   I'm happy to take your pull requests.

If it looks too much like code ...
If you can get non-coding colleagues to write your tests, then making the testing syntax easier for them is probably worthwhile.  You could certainly use Cucumber on top of rspec-solr  to write your Solr acceptance tests in more natural language.


Tip:

For most of my tests, I realized I don't check anything but the Solr document id in the results.   It makes it a lot easier to look through RSpec error messages when the Solr response doesn't have extraneous fields or the facet counts ... and it's also a much smaller http response.  So I rigged up a method that adds {'fl'=>'id', 'facet'=>'false'} to the request params I send to Solr.   My spec errors now read like this:

expected {"responseHeader"=>{"status"=>0, "QTime"=>10, "params"=>{"facet"=>"false", "fl"=>"id", "qt"=>"search_author", "wt"=>"ruby", "q"=>"契沖"}}, "response"=>{"numFound"=>3, "start"=>0, "docs"=>[{"id"=>"6675613"}, {"id"=>"6675393"}, {"id"=>"6274534"}]}} to include ["6675613", "6675393", "7191966", "6274534", "4783602"]
Diff:
@@ -1,2 +1,14 @@
-[["6675613", "6675393", "7191966", "6274534", "4783602"]]
+{"responseHeader"=>
+  {"status"=>0,
+   "QTime"=>10,
+   "params"=>
+    {"facet"=>"false",
+     "fl"=>"id",
+     "qt"=>"search_author",
+     "wt"=>"ruby",
+     "q"=>"契沖"}},
+ "response"=>
+  {"numFound"=>3,
+   "start"=>0,
+   "docs"=>[{"id"=>"6675613"}, {"id"=>"6675393"}, {"id"=>"6274534"}]}}
 
and they could have even less output, if I turned off "diffable" -- but I am currently finding it helpful.




Okay, but what good is this, really?

My current project is to improve search results for CJK (Chinese, Japanese and Korean) queries in SearchWorks.   It's nearly impossible for a CJK-ignorant coder such as myself to write good tests.  It's pretty darn hard for our non-coder CJK experts to write good tests, too.  So we have to iterate to figure out a set of acceptance tests.  Doing this without coding repeatable, automatable tests is ludicrous.**

We already have search tests, but our current search tests are slow.  They use Cucumber to mimic a user interacting with the web page to do a search, send the request to Solr, then the SearchWorks Blacklight Rails stack prepares the html that would be served by the application to present the search results from Solr.   The assertions are made against the html.   Given that for search acceptance testing, we don't care about the rails stack, this is a lot of extra processing. 

So it's time to take Rails out of the picture.  With some help from my colleague Chris Beer, we conceived a way to make it really simple -- let's write rspec style language on Solr response objects!   That spawned rspec-solr.

I am already using rspec-solr for our CJK acceptance tests.  All I needed was the rsolr gem, a spec_helper file, and some simple configuration stuff - 4 very small files.  (See rspec-solr README)

I've got CJK tests like this:

  it "should parse out 中国 (china)  经济 （economic)  政策 (policy)" do
    resp = solr_resp_doc_ids_only({'q'=>'中国经济政策'}) 

    resp.should have_at_least(85).documents
    resp.size.should be_within(5).of(solr_resp_doc_ids_only({'q'=>'中国  经济  政策'}).size) 

  end
 

  it "Traditional chars 三國誌 should get the same results as simplified chars 三国志" do
    resp = solr_resp_doc_ids_only({'q'=>'三國誌'}) 

    resp.should have_at_least(240).documents
    resp.should have_the_same_number_of_results_as(solr_resp_doc_ids_only({'q'=>'三国志'}))
  end

  it "hangul  광주 should get results for hancha  光州" do
    resp = solr_resp_doc_ids_only({'q'=>'광주'})
    resp.should include(["7763372", "7773313"]) # hancha  光州
    resp.should have_at_least(110).documents

  end


I'm also migrating our cucumber search regression tests to the rspec-solr approach -- obviously, I want a full suite of regression tests as I make changes for CJK searching.

A sample regression test:

  it "q of 'Two3' should have excellent results", :jira => 'VUF-386' do
    resp = solr_resp_doc_ids_only({'q'=>'Two3'})
    resp.should have_at_most(10).documents
    resp.should include("5732752").as_first_result
    resp.should include("8564713").in_first(2).results
    resp.should_not include("5727394")
    resp.should have_the_same_number_of_results_as(solr_resp_doc_ids_only({'q'=>'two3'}))
    resp.should have_fewer_results_than(solr_resp_doc_ids_only({'q'=>'two 3'}))
  end
 

Both types are very much works in progress, but I've deliberately put the tests up on github as the sw_index_tests repository so you can leverage them however you see fit.

I think it's pretty slick.


** In fact, they already DID do this for our ILS without repeatable, automatable tests ... and without records of their manual tests ... so we're starting from scratch.  How annoying and wasteful!

Upgrading from Solr 1.4 to Solr 3.5 - hiccups

Stanford SearchWorks has been due for a Solr upgrade for a loooong time -- we've been using Solr 1.4 since ... well, forever.   Bob Haschart upgraded SolrMarc to work with Solr 3.5, so I figured I would upgrade Solr as I refactored SolrMarc for the stanford-solr-marc fork.  (See also previous blog entry).
  In the course of upgrading from Solr 1.4 to Solr 3.5, a number of our tests were failing.  Usually the problem was a mistake in my configuration files for Solr 3.5;  sometimes the tests were too brittle.  It took a pass or two to start using the ICU library for unicode normalization, rather than SolrMarc's unicodeNormalizer.  I managed to get most of the failing tests to pass, but a handful stumped me.

Here's what I learned:

I.  (Hyphens) and WordDelimiterFilterFactory

Solr 3.2 (?) added a new setting for field analysis:  autoGeneratePhraseQueries, that defaults to "false".  In Solr 1.4, this setting was always true.  The difference is important for certain settings of WordDelimiterFilterFactory.  Let's say we have a query with a value of  "red-rose" (no quotes).

in Solr 1.4:

<fieldtype name="text" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
     <tokenizer class="solr.WhitespaceTokenizerFactory"/>
     <filter class="schema.UnicodeNormalizationFilterFactory" version="icu4j"
        composed="false" remove_diacritics="true" remove_modifiers="true" fold="true"/>
     <filter class="solr.WordDelimiterFilterFactory"
        splitOnCaseChange="1" generateWordParts="1" catenateWords="1"
        splitOnNumerics="0" generateNumberParts="1" catenateNumbers="1"
        catenateAll="0" preserveOriginal="0" stemEnglishPossessive="1"/>
     <filter class="solr.LowerCaseFilterFactory"/>
     <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/>
     <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
   </analyzer>
</fieldtype>

With debugQuery=true, we find the following query fragment being generated by dismax:
   text_field:"red (rose redros)"

in Solr 3.5:

<fieldtype name="text" class="solr.TextField" positionIncrementGap="100">
   <analyzer>
     <tokenizer class="solr.WhitespaceTokenizerFactory"/>
     <filter class="solr.ICUFoldingFilterFactory"/>
     <filter class="solr.WordDelimiterFilterFactory"         splitOnCaseChange="1" generateWordParts="1" catenateWords="1"
        splitOnNumerics="0" generateNumberParts="1" catenateNumbers="1"
        catenateAll="0" preserveOriginal="0" stemEnglishPossessive="1"/>
     <filter class="solr.LowerCaseFilterFactory"/>
     <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/>
     <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
   </analyzer>
 </fieldtype>
debugQuery=true shows us this query fragment:
   (text_field:red text_field:rose text_field:redros) -- including the parens.

Thus, a match on just "rose" is good enough in Solr 3.5, but not so in Solr 1.4's analysis.

How to fix this?

Add the attribute autoGeneratePhraseQueries="true" to the field type declaration:

  <fieldtype name="text" class="solr.TextField" positionIncrementGap="100"
       autoGeneratePhraseQueries="true"> 

2. StreamingUpdateServer and Binary Updates

In the most recent release of SolrJ (3.5), the streaming update server was not processing binary fields properly.  Two solutions:  1)  use the SolrJ jar provided in Bob Haschart's SolrMarc, as he has modified it to address this problem.  2) use a nightly jar, as this has been fixed in the SolrJ trunk and the SolrJ 3.6 branch.

3. Phrase Slop and Queries with Repeated Terms

Ultimately, I managed to get our tests passing except for two.  I couldn't figure out the difficulty - I looked at debugQuery results on Solr 1.4 and Solr 3.5;  I compared using the analysis debugger from the admin interface - nothing looked different.

Jonathan Rochkind pointed out that both phrases had repeated words;  these were both phrase searches as well.

It turns out that there was a bug in Lucene (that crept in sometime between Solr 1.4 and Solr 3.5).  If there was a non-zero slop setting in a phrase query with repeated terms, then results were incorrect.

https://issues.apache.org/jira/browse/LUCENE-3821

Thanks to Doron Cohen and Robert Muir, a fix was found and a patch was applied to Lucene, which was picked up in the Solr trunk and Solr 3.6 branch as of March 10, 2012.