CJK with Solr for Libraries, part 10

This is the tenth (gah!) of a series of posts about our experiences addressing Chinese, Japanese and Korean (CJK) resource discovery in SearchWorks, the Stanford University Libraries "catalog" built with Blacklight on top of our Solr index.

In the previous post, I described our level 2 evaluation of CJK resource discovery, and noted a significant blocker to moving forward: the CJK improvements we were making broke current production behavior.

CJK Indexing Breaks Production?

I call our sw_index_tests relevancy tests, because these tests submit Solr queries to our index and evaluate the results for "correctness."  In a sense, though, they could be called regression tests - changes to the Solr index shouldn't break any of these tests.  Of course, they do break sometimes when records are added to or removed from our collection, but we try to keep them from being too brittle without sacrificing much test precision.

Let me step back a minute and give a bit of context.  Our data is almost entirely bibliographic metadata in the MARC format, and frequently contains data in scripts beyond English.  When data is in a different script, the MARC 21 standard expects it to be put in a linked 880 field.  SearchWorks has always indexed the data in the 880s, and has always recognized that the data isn't English, and therefore should not be stemmed or have stopwords applied.  This is the fieldtype definition we use for these linked 880 vernacular script fields:

We also use this field type for unstemmed versions of English fields, so we can sort more exact matches first.  For example, we index the title both stemmed and unstemmed, and we boost the unstemmed matches more than the stemmed ones so they are scored higher.

Note that the textNoStem fieldtype uses the WhitespaceTokenizer, so tokens are created by splitting text on whitespace.  When we spun up SearchWorks five years ago, we wanted to have behaviors like those below (all of these are from sw_index_tests) for non-alpha characters due to specific resources in our collection.  You even can see the bug tracker (jira) ids in many of the tests as tags.
numbers mashed with letters:

hyphens:

musical keys (also here):

programming languages:

As you've probably guessed, what the above tests all have in common is that they failed when we changed our textNoStem fieldtype definition to use a Tokenizer that assigns a CJKBigramFilter friendly token type.  So if we used this fieldtype definition instead:

we got a number of test failures:

Most of the failures we got by substituting the text_cjk fieldtype for textNoStem are due to hyphens, programming languages with special characters, and musical keys.

Let's use the handy-dandy field analysis GUI to see what happens to a string with these special characters when it passes through the textNoStem fieldtype:

and what happens when we pass the same string through the ICUTokenizer:

Note that all of the non-alphanum characters have been dropped by the ICUTokenizer.  The same happens with the StandardTokenizer:

So it seems we are unlikely to have a one-size-fits-all fieldtype definition that will work for CJK text analysis and also for non-alphanum characters.  What should we do?

Context-Sensitive Search Fields

Recall that as discussed in previous posts, we need to use a different mm and qs value to get satisfactory CJK results, but we want to keep our original mm and qs values for non-CJK discovery.  So we are already on the hook for the SearchWorks Rails application to recognize when it has a user query with CJK characters in order to pass in CJK specific mm and qs values.   I realized that I could make CJK flavors of all qf and pf variables in the requestHandler in solrconfig.xml, like this:

So now when our Rails application detects CJK characters in a user's query, it needs to pass in special mm, qs, qf, pf, pf3 and pf2 arguments to the Solr requestHandler.  So a normal Solr query for an author search might look like this (I am not url-encoding here):

  q={!qf=$qf_author pf=$pf_author pf3=$pf3_author pf2=$pf2_author}Steinbeck

while a Solr query for a CJK author search would look like this (scroll right to see the whole thing):

  q={!qf=$qf_author_cjk pf=$pf_author_cjk pf3=$pf3_author_cjk pf2=$pf2_author_cjk}北上次郎&mm=3<86%25&qs=0  

You may already be familiar with Solr LocalParams, the mechanism by which we are passing in qf and pf settings as part of the q value above.

Now, how do we test this?  Does it work?

Applying LocalParams on the Fly

First, I wanted to try this new approach in our relevancy tests to see if it worked.  If you look at this test, you'll see there isn't anything in the test itself indicating it uses CJK characters:

This test uses the magic of rspec shared_examples, which were employed to DRY up the test code.  The shared example code looks like this:

The method called by the shared example is in the spec_helper.rb file.  I'll do a quick trace through here:

 So the cjk_query_resp_ids method calls cjk_q_arg, which, for our author search, calls cjk_author_q_arg:

which clearly inserts the correct cjk author flavor qf and pf localparams.

What about mm and qs?  These are added later in the helper code chain; the following method is also called from cjk_query_resp_ids:

which in turn calls the solr_response method:

Detecting CJK in a query string

What I've shown above is that my CJK tests use a method, cjk_query_resp_ids, to ensure the correct CJK localparams are sent in to Solr.  But I haven't shown how to automatically detect if a query has CJK characters.

Look again at the solr_response method.  It first calls a method that counts the number of CJK unigrams in the user query string, which is a snap with Ruby Regexp supporting named character properties:

If CJK characters are found in the user query, then solr_resp calls a method that returns the mm and qs parameters:

Even if you didn't follow all that, we definitely end up with the following being sent to Solr:

  q={!qf=$qf_author_cjk pf=$pf_author_cjk pf3=$pf3_author_cjk pf2=$pf2_author_cjk}北上次郎&mm=3<86%25&qs=0  

Using all that helper code above, and a bit more for other types of searches (title, subject, everything ...), I can assert that we did get all our production tests to pass as well as all our CJK tests -- all the issues with hyphens and musical key signs and so on went away.  Yay!

(I want to mention that the layers of indirection in that code are useful for other contexts than the particular test illustrated.)

Context-sensitive CJK in Our Web App

Great - our tests all pass, and the CJK tests we have pass too!   But what about repeating all that fancy footwork in the SearchWorks web application?  We need to get the correct requests sent to Solr based on automatic CJK character detection in user entered queries.

In fact, the SearchWorks Rails application uses a lot of the same code as what I use for our relevancy tests. Let's go through it.

As you know, SearchWorks is built using Blacklight, which is a Ruby on Rails application.  It all starts with a "before_filter" in the CatalogController:

Which is really a method that calls another method and adds the result to the existing Solr search parameters logic (Blacklight magic):

The cjk_query_addl_params method checks the user query string for CJK characters, adjusts mm and qs accordingly, and applies the local params as appropriate for CJK flavored searches:

The two methods I highlighted with purple boxes above are exact copies of those same methods in our relevancy testing application.  They live in a helper file, because they are used to tweak advanced search for CJK, which will be addressed in a future post.

We put these changes in a test version of the SearchWorks application, asked the CJK testers to try it out, and it all worked!  Also, we have end-to-end test code for the application that ensured these app changes didn't negatively impact any existing functionality while it did afford better CJK results.

D'oh!

In the course of writing this post, I realized that I probably can use the same fieldtype for non-CJK 880 text and for CJK 880 text;  I may only need to distinguish between un-stemmed English fields and all vernacular fields, rather than between CJK and non-CJK 880 text.  Since this approach could reduce the number of qf and pf variables we have in our solrconfig.xml files, I may experiment with it in the future.

Where are we Now?

This post showed how I removed our blocker to putting CJK discovery improvements into production:  how I analyzed how CJK Solr field analysis was breaking production behaviors, my chosen solution, and how we tested and applied that solution. 

With the blocker removed, and two of the three items on the "must do" list fixed (see previous post), we asked the testers via email to vet our work.  They agreed we had fixed the intended problems and not broken anything else.  At this point, we worked out a schedule to push these fixes to production.

Well, in reality, one of the fixes on the "should do" list went to production as well, and we agreed that getting advanced searching working for CJK could be a "phase 2" improvement.  Future posts will address all the other improvements: fixing advanced search for CJK and the four other tweaks we added to improve CJK resource discovery (not necessarily in that order).  And of course we will share the final Solr recipes and our code (already available via github).

CJK with Solr for Libraries, part 9

This is the ninth of a series of posts about our experiences addressing Chinese, Japanese and Korean (CJK) resource discovery in SearchWorks, the Stanford University Libraries "catalog" built with Blacklight on top of our Solr index.

In the previous post, I described our level 1 evaluation of CJK search results using the CJK fieldtype definition described in part 7 of this series, and indicated that with a tweak to our mm setting for CJK, we had lift off for level 2 evaluation of CJK Discovery in SearchWorks: more thorough evaluation of resource discovery in all three languages by our East Asia librarians.

Evaluating CJK Discovery, Level 2

I haven't shared here that our baseline objective for CJK resource discovery was to have parity with Socrates (the name for Stanford's catalog via the ILS).  Thus, we wanted our East Asia librarians to evaluate CJK in SearchWorks with this parity in mind.  It's all about communicating expectations, right?  What to test, how to report back, what to report back.  And most importantly: what is "good enough?"  Here's my agenda from our first meeting:

 and here are the instructions I put together for our testers:

As I've stated before, my colleagues are awesome.  They did a bunch of testing, and they gave us feedback.  The feedback wasn't as specific per search as my highest hopes (rarely were any of the detailed questions above answered), but it was still excellent, useful feedback.  And a crucial point: our feedback link in SearchWorks provides the url of the page from which it is sent - meaning the SearchWorks feedback mechanism provides us with some context of what the user was doing (since they often leave out important details, like what kind of search they were doing, or the exact query string, all of which are in the url).

We took the spreadsheet approach started by Vitus Tang, as shown in the previous post and expanded on it as a crude way of collating the testing feedback.  I used the following color system:

• dark green = SearchWorks is better than Socrates
• light green = SearchWorks is at least as good as Socrates
• dark red = Socrates is better than SearchWorks
• light red = SearchWorks and Socrates both miss the mark.

With all this testing, we also had to split each language into a separate sheet, so I'm only showing a small portion here:

I did ultimately add nearly all the queries from this spreadsheet into our cjk relevancy tests, so don't worry if you can't read it -- I'm just sharing it here for the sake of showing methodology.

To be honest, we were astonished by how positive much of the feedback was.  Some of the problems reported turned out to be newbie SearchWorks users issues -- because CJK resource discovery had always been broken in SearchWorks, the testers were new to it -- new to facets, new to relevancy ranked results, etc.  We also educated the testers that in SearchWorks, you might get too many results (too much recall), but the most relevant results should be first, which satisfied most user needs.  It's a bit of a culture shift to go from an ILS discovery interface like Socrates to SearchWorks.

With feedback messages put directly into our JIRA instance (which I watched assiduously throughout testing) and various email exchanges with the East Asia librarians, we only needed one more meeting on vetting CJK resource discovery.  In the second meeting, we asked each East Asia librarian to indicate whether they felt SearchWorks had achieved parity (or better) with Socrates, and what problems had surfaced.  The decision was unanimous that SearchWorks was at parity or better, especially after we had fixes to the problems noted.

Here is the list of blockers that were found by the CJK testers:

1. (empty)

Here is the list of must-dos found:

1. phrase searching broken
2. author + title searches not working for CJK
3. advanced search not working for CJK

And here are the should-dos found:

1. deal with extra spaces in our data due to cataloging practice
2. variant characters - while most Han/Kanji traditional characters are translating properly to Han simplified/Kanji modern, some characters are not.
3. variant quote marks - it is common to use quote marks in CJK queries other than " (ASCII hex 22).
4. add CJK tests - beef up our CJK test suite with the searches used so we don't break functionality in the future.  (see sw_index_tests cjk tests at https://github.com/sul-dlss/sw_index_tests/tree/master/spec/cjk)
5. publish CJK solutions for others to use (that's these blog posts ...)

The would-be-nice-to-dos (identified by me):

1. experiment with boosting bigrams more than unigrams
2. experiment further with Unicode normalization, after automated CJK tests are beefed up
3. experiment with boosting phrase slop setting, ps

However, there was a huge blocker identified by me:

1. current production (tests) cannot be broken by CJK improvements.

A couple of the must-do issues were identified early on and fixed fairly easily.  I'll cover them first, and note here that once these two "must-dos" were fixed and I got past the blocker of breaking existing behavior, all of which happened in a week, we put what we had in production.  The timing was partly rushed to have it installed before classes started in the fall, but nevertheless, it was a major coup to have such a positive response to our work.  It was exciting to get CJK improvements to SearchWorks rolled out to end users so quickly after expert testing.

CJK Tweak 2: qs setting for phrase searches

To fix the phrase searching problem, again we found assumptions in our setup that worked for English, but not for CJK.

Dismax and edismax have a "query phrase slop" parameter, qs, which is the distance allowed between tokens when the query has explicitly indicated a phrase search with quotation marks.  Probably from back in our stopword days, we use a setting of qs=1, meaning a query of "women's literature", with the quotes, is allowed to match results containing 'women and literature' as well as 'women in literature' in addition to 'women's literature'.  Because of the magic of pf sorting the best matches first, this has worked just fine for our users up until now.  However, with CJK queries, this is undesirable -- an explicit phrase query in CJK should only match the exact characters entered, with nothing inserted between them.

Since we were already changing the mm setting for CJK queries (see previous post), it was a simple matter to also change the qs setting to 0 for CJK queries.  This gave CJK phrase searching the desired behavior, per our testers.  Yay!

CJK Tweak 3: catch-all field

The problem with author + title searching is well described in this feedback from one of our Chinese materials specialists (routed from the feedback link in SearchWorks to our JIRA project):

One of the missing records:

Note that the three characters of the author are in the linked 100 field, and the short title, 245a, starts with the five title characters in the original query.  The second missing record has the same author/title data -- the second book is a reprint of the first. 
Looking at the debugQuery analysis, we can see that even though the individual tokens are being matched, the mm is applying to all tokens within a single field:

The clause is basically ((fld:吕 fld:吕思 fld:思 ... fld:朝)~12^20), where ~12 is the mm (the query is 8 characters, creating 15 tokens, and 86% of 15 = 12) and ^20.0 is the boost we give matches in the field.   So this is looking for 12 of the query tokens to match in a single field.  The 5 character title only produces 9 tokens, and 9 < 12, so it is not enough to produce a match from just the title field either.

We have always had a "catchall" field for our Solr index, but we never had an analogous field for our vernacular script data (in the 880s). Thus, we never created a CJK flavor catchall field.  I rectified this, and voila!  author + title CJK searches worked.

Stay Tuned ...

I have just explained fixes for two of the three items on the "must do" list.  The next post will address the blocker -- our CJK improvements were breaking some production behaviors, as documented by our relevancy tests.  I will follow that with posts discussing how we addressed the rest of the issues on the "must do" and "should do" lists, and of course will share our current production solutions (which are already available here with relevancy tests here).

CJK with Solr for Libraries, part 8

This is the eighth of a series of posts about our experiences addressing Chinese, Japanese and Korean (CJK) resource discovery in SearchWorks, the Stanford University Libraries "catalog" built with Blacklight on top of our Solr index.

In the previous post, I described our initial Solr fieldtype for CJK text. Here's how we evaluated the outcome and analyzed the results.

Evaluating CJK Discovery, Level 1

I am very very fortunate to have fantastic colleagues, and I am fortunate indeed to work with Vitus Tang on many issues related to SearchWorks discovery.  Aside from his extensive knowledge of Stanford's Marc data and data practices, Vitus reads Chinese.  Thus, he was able to evaluate Chinese search results and determine if we had anything good enough to warrant broader human review by our East Asia librarians.  We also had a handful of Japanese and Korean tests from meeting with the East Asia librarians and getting their preferred priorities in writing, with some example strings.

Despite all this, there is a communication gap that is hard to breach between a CJK-illiterate techie such as myself who groks automated relevancy testing and a CJK-literate non-techie expressing search expectations.  In fact, when I started working at Stanford, SearchWorks testers believed that "correct" search results were whatever our ILS (in which the Stanford discovery interface is named "Socrates") gave us; thus it was believed that acceptance criteria should always be "match the ILS results."  For CJK, this was especially true, as the East Asia librarians had spent significant earlier effort working with Sirsi/Dynix to get functional CJK searching into the Unicorn (now Symphony) ILS application.  Sadly, there were no test scripts or any other testing artifacts I could leverage from that work, so we had to start over.

As I mentioned in the third post of this series, the first SearchWorks relevancy tests were written with cucumber and went through the entire SearchWorks application Rails stack.  We made an early effort to get our CJK acceptance test in the cucumber format.  We got a small number of tests like this in August 2012:

but these tests were inadequate - we were still getting far too many hits due to SOLR-3589 (mm set to zero when the analysis chain splits a token), and we weren't evaluating some key factors, such as whether the most relevant results were sorting first.

Before SOLR-3589 was fixed, we tried 10-15 different ideas for workarounds.  Vitus examined results for all of these efforts, and none of them passed muster.  Eventually, I learned he was tracking basic evaluation like this:

What's crucial here is that Vitus was only looking at numbers of results as his first pass -- if the numbers were too far off, then the trial wasn't worth further evaluation.  Clearly, instead of having a person do this, we could do it with code.  Given the information in the spreadsheet (the search flavor (title, author), the numbers of results from our ILS (Socrates), the comparison of simplified to traditional Han script, and specific query strings), a CJK-illiterate techie such as myself could get this info much more quickly by using a program.  

Since my ultimate goal with test code was to have CJK acceptance tests, I coded the above using rspec-solr (http://rubydoc.info/github/sul-dlss/rspec-solr) and grabbed the hit counts from the error messages.   Eventually, my tests looked something like this:

I'm not going to talk about all my tests right now (though they are available at https://github.com/sul-dlss/sw_index_tests);  the point is that once SOLR-3589 was fixed, we could easily tell our numbers looked a lot more promising.

The green column shows the result counts from Socrates;  the purple columns are different variants of the analysis for text_cjk fields, such as StandardTokenizer vs. ICUTokenizer and different normalization choices.  The numbers may be hard to read here, but the consensus was this was a big improvement over the previous attempts.  Yay!  And that the traditional <--> simplified Han equivalence was working - Yay!

Note that this approach also allowed me to evaluate different forms of Unicode normalization without needing Vitus's help for every iteration.  As best I could determine, I got the same results regardless of canonical vs. compatible equivalence and composed vs. decomposed, and all combinations thereof.  Or, if we want to think about it differently, the normalization chosen in the text_cjk fieldtype definition of my previous post was just fine.

At this point, Vitus started looking more closely at the Chinese results.  One of his focal points was bad recall:  when Solr results were missing relevant records found by our ILS.

CJK Tweak 1: mm setting

As Vitus reported specific problems, I analyzed them with the debugging tools mentioned in my third post of this series.  Here is an example of such a problem reported by Vitus:

In this particular case, setting debugQuery=true on the Solr request showed me that our "Min Number Should Match" (mm) setting was causing us difficulty.  The description of the (e)dismax mm setting is here; the spec for the format of mm settings is here.  Our mm setting was  6<-1 6<90%, which translates to:  for 1 to 6 clauses, all are required; for more than 6 clauses, 90% (rounded down) are required.  Unless you have explicit parenthesis in your query, a "clause" is a token.  This was chosen based on user query analysis when we removed stopwords from our index.  I even blogged about it. 

Our mm setting makes sense for whitespace separated tokens, but for overlapping bigrams, it was causing problems.  Take the following example: 婦女與文學 (women and literature in traditional Han characters).  The correct word breaks are 婦女   與   文學.  Our text_cjk fieldtype creates four overlapping bigrams:  婦女, 女與, 與文, 文學 and five unigrams: 婦, 女, 與, 文, 學 for a total of 9 separate tokens or "clauses" as far as mm is concerned.  90% of 9 is 8, rounded down.  So that means all but one of the "clauses" must be found in the record.  Assuming all unigrams are found, that means all but one of the overlapping bigrams must be found, so Solr is requiring at least one of the nonsensical bigrams combining adjacent characters with 'and':  女與 or 與文.  Thus, Solr is only returning results where 'and' immediately precedes literature or where 'and' immediately follows women.

Another example:  董橋 (Dong Quai, an author).  This creates 3 tokens:  董橋, 董 and 橋.  Our mm setting requires all clauses when there are 6 or fewer, so all of these tokens are required.  This means the bigram is required, so the characters must appear in the same order as the query.  Perhaps this is good for our author query ... but should it be true for all 2 character CJK queries?

At this point, I started wondering about the number of characters in our users' CJK queries, and how many of the bigrammed tokens should be "required" via mm.  How many CJK queries have whitespace in them?  Do CJK queries also include non-CJK characters?  How many characters are in most CJK words?

To answer these questions, I asked the Stanford ILS support folks for CJK queries in our usage logs from Socrates.  They were kind enough to provide me with 1033 (mostly) CJK searches for 2 months. I did a rough analysis of these queries, sorting them by length and getting a sort of histogram.  It's a bit tricky since we have to count characters for CJK, but some of the query strings were in quotes, e.g.  '昌化縣志', had whitespace (妇女  与  婚姻) or included boolean operators (戴季陶 AND 國民黨).  Ignoring the boolean queries and non-CJK queries, and not counting non-CJK characters such as quotes, the data looked like this:

 or, if you prefer it in a table:

Note that nearly 95% of our queries in the data sample contained 8 or fewer CJK characters.
In the interests of broader validation, Tom Burton-West of HathiTrust was kind enough to share like information.  I was not able to so carefully vet the character counts in the HathiTrust queries (e.g.  婦女與文學 is five characters, but I consider 婦女  與  文學 to also be five characters, because I'm not counting the whitespace), but if you look where 10 characters crosses the graph (the horizontal line labeled 10 on the y axis), and go down two steps, you can see that well over 50% of the queries in Tom's data are 8 characters or fewer.

I'm guessing that if we removed spaces and punctuation from the char counts, the character count distribution would look more like ours.

So the data supports an assumption that the vast majority of our CJK queries will be 8 CJK characters or fewer.  What does this mean about our mm setting? At this point, I checked in with Vitus to buttress another assumption: that most Chinese written words are 2 or more characters.  This is true, and since Chinese has more materials than our Japanese and Korean collections combined, I decided it was a good starting place for picking an appropriate mm value for CJK bigrammed + unigrammed strings.

The second column in the table shows the number of tokens from the overlapping bigrams + unigrams if all the CJK characters are adjacent -- the maximum number of tokens that will be parsed from a CJK string.  For example, I showed above that a 5 character string has 4 overlapping bigrams, and 5 unigrams for a total of 9 tokens.  If there is a space or punctuation mark within the string, there are fewer tokens, because one of the bigrams is no more:  婦女  與  文學  still has 5 unigrams, but only 2 bigrams, for a total of 7 tokens.
Next I came up with the Desired mm values by thinking of how many adjacencies I could subtract if words were 2 characters. For three and four character queries, I wanted to require all but one adjacency:   舊小說 (old fiction) can split into 舊 (old) and 小說 (fiction). For five and six character queries, I wanted to required all but two adjacenies: 婦女與文學 (women and literature) can be separated into 婦女 (women) 與 (and) 文學 (literature).   This sort of thinking would have similar recall to Socrates, which is what Vitus's analysis was investigating.
Using the table above, I chose this mm setting for CJK:

3<86%

Per the mm spec, this says for three or fewer "clauses" (tokens), all are required, but for four or more tokens, only 86% (rounded down) are required.  This is perfect for CJK queries of 6 or fewer characters, a tad high for 7 characters, and perfect again for 8 characters, and seems to be the best fit available.
Recall that with (e)dismax's pf settings, in a result set where some results have the query characters as adjacent and some don't, the hits with the characters adjacent should score higher and sort to the top.   - so when we get too many results (too much recall), we can mitigate it by making sure the best (most precise) results are first.
This tweak to our mm setting for CJK queries got more test searches through the Vitus-meter.

I will talk later about how we use code in the SearchWorks application to only apply this mm setting to queries with CJK characters, retaining our original mm setting for all other queries.

Evaluating CJK Discovery, Level 2

With the text_cjk fieldtype shown in the previous post and the mm tweak above, Vitus felt we had results ready for broader testing by the East Asia librarians.  I will discuss how we approached that testing, and what else we had to tweak for CJK resource discovery in future posts in this unbelievably long series.

CJK with Solr for Libraries, part 7

This is the seventh of a series of posts about our experiences addressing Chinese, Japanese and Korean (CJK) resource discovery in SearchWorks, the Stanford University Libraries "catalog" built with Blacklight on top of our Solr index.

The last three posts were focused on diagnosing and mitigating the differences between the edismax and dismax query parsers, since edismax (and Solr release of 4.1 or later) is a prerequisite for a critical bug fix for CJK analysis.  Note that without this bug fix, one of our CJK experts referred to the search results we were getting as "inexplicable."

As a refresher, we recall our original mission from the first blog post in this series:

CJK Discovery Priorities

Chinese

1. Equate Traditional Characters With Simplified Characters
2. Word Breaks

Japanese

1. Equate Traditional Kanji Characters With Modern Kanji Characters
2. Equate All Scripts
3. Imported Words
4. Word Breaks

Korean

1. Word Breaks
2. Equate Hangul and Hancha Scripts

Solr Solutions to Leverage

We borrow from the second post in this series to remind ourselves which Solr tools interest us for our Solr CJK processing needs:

1.  CJKBigram Analysis

Tom Burton-West's blog post advises the use of a combination of overlapping bigrams and unigrams of CJK characters for multilingual discovery.  The CJKBigramFilter, new with Solr 3.6, allows us to generate both the unigrams and the bigrams for CJK scripts only:  perfect.

In the future, Solr may provide CJK dictionary-based segmentation (see LUCENE-4381 and SOLR-4123), but it doesn't exist yet.  The ability to provide your own rules for tokenizing has arrived, though!  See the javadoc for the ICUTokenizerFactory for more information.

An important note:  The CJKBigramFilter must be fed appropriate values for the token type.  This can be learned from the source code:

or from looking at Solr admin analysis. Currently, the tokenizers that assign such types are:

a. ICUTokenizer

Here is output from Solr admin analysis on a string with four different CJK scripts as well as latin.  Note the type attribute, which I highlighted in the label column on the left:

b.  StandardTokenizer 

Note that the StandardTokenizer output does not separate the Hangul character from the Latin characters at the end.

c. ClassicTokenizer 

I believe ClassicTokenizer will also assign token types, probably the same way StandardTokenizer does.

The WhitespaceTokenizer is an example of a Tokenizer that does not assign the types.  I have added spaces to the string to emphasize the difference in the token types assigned:

So we probably want to use the ICUTokenizer when we use the CJKBigramFilter.

2.  ICU Script Translations

Solr makes the following script translations available via the solr.ICUTransformFilterFactory:

1. Han Traditional <--> Simplified
2. Katakana <--> Hiragana

We would like to get more script translations (e.g.  Han <--> Hangul, more Japanese script translations), but to my knowledge, they are not readily available via Solr at this time.

3.  ICU Folding Filter

We have already been using solr.ICUFoldingFilterFactory for case folding, e.g. normalizing "A" to "a."  Even though CJK characters don't have the concept of upper and lower case, many CJK and other Unicode characters can be expressed multiple ways by using different code point sequences.  The ICUFoldingFilter uses NFKC Unicode normalization, which does a compatibility decomposition (using a broader, "compatible" notion of character equivalence and "decomposing" to distinct code points to represent separate-able parts of a character) followed by a canonical composition (the narrower, "canonical" notion of equivalence, and the more compact code point representation of characters).  We experimented with different flavors of Unicode normalization and found NFKC to be just fine for CJK discovery purposes.  I may talk about this further in a later post.

Solr Fieldtype Definition

The example schema provided with Solr shows one way you might configure a fieldtype to do CJKBigramming:

While it does use the CJKBigramFilterFactory, there are no script translations and it isn't using the ICUFoldingFilterFactory.  Note that this email clarifies that solr.CJKWidthFilterFactory normalizations are a subset of those in solr.ICUFoldingFilterFactory.

So our improved version of the above might look like this:

Spoiler Alert:  this is NOT our final fieldtype definition.

Let's examine it more closely:

1. positionIncrementGap attribute on fieldType
   
This setting is all about trying to keep your matches within a single field value for a multivalued field.  Bill Dueber explains it nicely in this blog entry.  Basically, the values in a multivalued field are stored adjacently, and this setting is the number of pretend tokens between the field values.  Thus, a large value keeps phrase queries from matching some words at the end of one field value, and some words in the beginning of the following field value.  Given that we will have tons of tokens for CJK strings (roughly 2 per CJK character, from the unigrams and overlapping bigrams), setting this to a nice high value is a good idea.

2. autoGeneratePhraseQueries attribute on fieldType
As LUCENE-2458 describes, prior to Solr 3.1, if more than one token was created for whitespace delimited text, then a phrase query was automatically generated.  While this behavior is generally desired for European languages, it is not desired for CJK.  Oddly, when this bug was fixed and a configuration setting was made available, they changed the default behavior - the default is now false. "false" is the correct value for CJK, while "true" is most likely what you want in other contexts.  The Jira ticket description is pretty readable;  Tom Burton-West also mentions this in his blog, and I briefly talk about it in my blog entry about upgrading from Solr 1.4 to Solr 3.5.

3. ICUTokenizerFactory
The section above on CJKBigram analysis shows that the ICUTokenizer is likely to be better than the StandardTokenizer for tokenizing CJK characters into typed unigrams, as needed by the CJKBigramFilter.

4. CJKWidthFilterFactory
It may be that this is completely unnecessary, but on the off chance that the script translations don't accommodate half-width characters, I go ahead and normalize them here.  For the curious, written CJK characters can be very opaque, so old printers used to print each CJK character twice as wide as a Latin characters.  However, some of the simpler characters could be printed regular size, or "half-width" compared to other CJK characters. (http://en.wikipedia.org/wiki/Halfwidth_and_fullwidth_forms)
5. ICUTransformFilterFactory  from Han traditional to simplified
This is the translation of traditional Han characters to simplified Han characters, which will allow a query to match both traditional and simplified Han characters.
6. ICUTransformFilterFactory from Katakana to Hiragana
This is the translation of Katakana characters to Hiragana characters, which will allow both of these scripts to be matched when there are characters in either script in the user query.
7. ICUFoldingFilterFactory
This will perform Unicode normalization as described in the Solr solutions section above.
8. CJKBigramFilterFactory
Note the settings to bigram all four CJK scripts, and to output unigrams in addition to bigrams.

So we now have a Solr fieldtype to create overlapping bigrams as well as unigrams for CJK script characters.  Super!

Our next step is to index all of our CJK data using this fieldtype, and then to include the new CJK fields in the boosted field lists searched by our edismax request handlers.

Where is our CJK Data?

In our preliminary testing, we confirmed SOLR-3589 made our CJK search results "inexplicable" (recall that this bug being fixed only for edismax is what drove us to switch to edismax).  One way we assessed our results was by comparing them to search results in our traditional library OPAC (Symphony ILS by Sirsi if you're curious).   For some of the very small result sets (15 hits or less), we could examine why each result was included by Solr and by the ILS.   These close examinations surfaced CJK characters that weren't in the appropriate Marc linked data 880 fields, but instead were included in Marc fields 505 and 520.  Many of these records came from external vendors.  We then asked our East Asia librarians to estimate how often CJK scripts would occur outside 880s, and the guesses were about 30% of the Chinese records, 15% of the Korean records, and 5% of the Japanese records.  With this in mind, we decided to apply the text_cjk fieldtype to 505 and 520 fields in addition to 880 fields.

Our CJK Solr fields are all copy fields (more on that in a subsequent post);  here are the relevant bits from our Solr schema.xml file, in addition to the fieldtype definition for text_cjk:

Note that the dynamicField declaration has indexed=true and stored=false:  the display of CJK text in the SearchWorks application is accommodated via other fields.  This whole effort is about CJK searching.  CJK search fields must be multivalued because many of the relevant Marc fields are repeatable.

Ready, Set, Index

The next posts in the series will talk about how we evaluated our indexing viz-a-viz CJK resource discovery, what additional indexing changes were required, and surprise changes needed in the SearchWorks rails application code.

CJK with Solr for Libraries, part 6 (Edismax woes, part 3)

This is the sixth of a series of posts about our experiences addressing Chinese, Japanese and Korean (CJK) resource discovery in SearchWorks, the Stanford University Libraries "catalog" built with Blacklight on top of our Solr index, and the third in the sub-series on problems we had switching to  edismax from dismax.

You might be interested in this post in particular if you use Solr's edismax query parser or if you want to get more exact matches for user queries.

Edismax Woes, Part 3

In part five of this series, I showed that our edismax results needed to give more weight to exact matches in the short title field in order to fix the failing relevancy acceptance tests for journal titles (and in order to behave more like dismax).

I also mentioned that Bill Dueber of the University of Michigan wrote a wonderful blog post on using "fully-anchored" text fields to get an "exactish" match.  This blog post will cover our solution to the journal title results using a fully-anchored text field.

What Do We Mean When We Say Exact?

Recall the results for an edismax title search on 'the press':

When we say "exact" match, our spec might be this:

1. case insensitive.  
2. field contains only those tokens we intend to match - it matches the entire contents of the field.  
3. punctuation insensitive  (e.g. the nation should match "The nation.")
4. Leading, trailing and consecutive whitespace insensitive.  
5. Unicode folding.  
6. possibly include synonyms.

It's not a phrase match, because that doesn't cover point 2 -- we don't want the first two results returned by edismax, even though they contain the phrase "the press".

A "good enough" solution, especially for short metadata fields, is to anchor the query string to the beginning and end of the field value, and to boost this "anchored" field type very high so will dominate the score when there's a match.

Anchored Text Fieldtype

Bill's blog post does a fine job of explaining how to do this.  Essentially, you add a prefix, like 'aaaa' to the beginning of the text, and you add a suffix, like 'zzzz' to the end of the text.

In our case, we have an unstemmed text fieldtype like this:

And we want to use a PatternReplaceCharFilterFactory to add the prefix and suffix to this field.  Bill's example is nice and simple:

Note that Bill's filter adds two tokens, which will affect mm counts.

I'm going to save you a lot of time and tell you right away:  be careful about your regular expression, especially concerning punctuation and other symbol characters.  Due to the Marc metadata standard having been originally designed to accommodate the printing of catalog cards, the short title fields in Marc data (245a) often end in punctuation and/or whitespace. 

Also, I didn't want to affect mm counts, because when we do bigramming for CJK characters, mm counts are going to be plenty tricky.  So here is what my version of the PatternReplaceCharFilterFactory to add anchoring prefix and suffix ended up as:

It's ugly, but it works.  It doesn't add tokens, like Bill's does, because it adds no whitespace to the token stream. 

Let's walk through the pattern:

   ^\s*

says ignore whitespace at the beginning of the text.

   [^\.\,:;/=<>\(\)\[\]\&\|] 

is a character class excluding these characters: .,:;/=<>(){}&|    Let's call this character class z.

z is inside another character class:

   [\S&&z]

which, in total, says any non-whitespace character or a doubly escaped ampersand, but none of the characters excluded by z.  Let's call this modified non-whitespace character class \S'.

So the first part

   ^\s*(.*[\S&&[^\.\,:;/=<>\(\)\[\]\&\|]]) 

is simplified to

   ^\s*(.*\S')

which is saying ignore beginning whitespace and capture anything after that ending in one of the characters in \S' (the modified non-whitespace character class).

The part after the capturing group contains another character class:

   [\s\.\,:;/=<>\(\)\[\]\&\|]

which includes:  whitespace as well as .,:;=<>()[]&|.  Let's call this character class Y.  So after the capturing group, we have:

   Y*$

which is saying to ignore any trailing characters in the Y character class.  So the whole expression simplifies to:

   ^\s*(.*\S')Y*$

So it's not really so far off from Bill's pattern.  It ignores initial whitespace, it ignores trailing whitespace and punctuation, and it captures everything in between only if it ends with a non-whitespace character other than .,:;/=<>(){}&|

So yay!  We have a charFilter to place at the beginning of the fieldtype analysis chain and we're ready to create a new field for exactish matches to address the edismax relevancy discrepancy when compared to dismax!  Our anchored text fieldtype looks like this:

And we're ready to go, right?  We re-index our data, adding a Solr field:

in which we put our short title, (245a for Marc data wonks).  Then we add this Solr field to our edismax formula with a higher boost value than the other fields used by edismax in the failing journal title relevancy tests.  We can tweak the boost value as needed, since we have a safety net of 600 relevancy tests to let us know if we're breaking existing functionality.

And all the relevancy tests now all pass, right?

Wrong.  Do you see the problem?  I didn't.

The journal title search results passed the tests (after we addressed the problem of trailing punctuation, ahem).  But then we were left with a bunch of failing synonym tests, in addition to the already failing synonym tests from our 21 tests failing with edismax.

Synonyms and Anchoring

A synonym can be thought of as a substitution.  If you have

   c++  =>  cplusplus

in your synonym file, then analysis from the text_anchored fieldtype above will change

   Great C++ programming

to

   aaaaaagreat cplusplus programmingzzzzzz

This is perfect.  But if you start with

   C++ programming

then you get

   aaaaaac programmingzzzzzz

Why?   This is where the Analysis form in the Solr admin GUI came in handy.  It tells us the analysis for the text_anchored fieldtype above proceeds like this:

   C++ programming

   aaaaaaC++ programmingzzzzzz   <== pattern replace char filter

   aaaaaaC++ programmingzzzzzz   <== whitespace tokenizer (makes 2 tokens)

   aaaaaac++ programmingzzzzzz   <== ICU folding filter

   aaaaaac++ programmingzzzzzz   <== synonym filter  (no-op) 

   aaaaaac programmingzzzzzz     <== word delimiter filter

The synonym filter doesn't find "c++" as a token, it finds "aaaaaac++" which it doesn't know about.

Because our list of synonyms is short, and because I wanted to keep synonyms in our text_anchored fieldtype, my solution was to create additional synonyms for right, left and both anchored versions of each synonym to be mapped.  This covers cases where the synonym is the first, last or only word in the token stream for the text_anchored fieldtype.  For c++, this becomes these synonyms:

   c++  =>  cplusplus

   aaaaaac++  =>  cplusplus

   c++zzzzzz  =>  cplusplus

   aaaaaac++zzzzzz  =>  cplusplus

Here is what our text_anchored fieldtype actually looks like:

(you can also look here to see it with syntax highlighting:  https://github.com/solrmarc/stanford-solr-marc/blob/master/stanford-sw/solr/conf/schema.xml#L371-388).

Edismax Woes Begone!

Recall that our goal is to improve our edismax journal title results by giving great weight to exact-ish matches of the query string in a Solr document's short title field.  We have a field type for this, per above, that we use for an "exactish" short title field (245a for Marc data wonks).  Then we add this Solr field to our edismax formula with a higher boost value than the other fields used by edismax in the failing journal title relevancy tests.  We can tweak the boost value as needed, since we have a safety net of 600 relevancy tests to let us know if we're breaking existing functionality.

This is precisely how we fixed the failing journal title tests when using edismax.

Recall that our relevancy test failures with edismax were in the following categories:

1. Journal titles
2. Hyphens preceded by a space (but no following space)
3. Boolean NOT
4. Synonyms for musical keys

We addressed the journal titles with the fully-anchored exactish matching short title field.  We determined that the hyphens and boolean NOT failures were due to a Solr bug, and that we can ignore these failures for now.  And I'll go ahead and tell you that the failing synonym test was for a musical key (F#) that also maps to a computer language, and we decided to live with that failure for now.  (Maybe I'll talk more about our synonyms in a future post.)

Relevancy/Acceptance Tests

I want to make a point of mentioning that while we were trying to figure out why searches on "the nation" were failing our tests, we had human testers looking for other errors and trying to find the pattern to the problems we were already aware of.  So we got a lot more tests for similar journal titles, such as "the sentinel," "the chronicle", "the times" (which we already had), etc.  Eventually I realized the culprit for our journal title results was the punctuation at the end of the Marc 245a fields.  But in the meantime, we beefed up our test suite in this area.

Our relevancy tests were also instrumental in determining an appropriate boost value for our new short title exactish match field, both in title searches and in "everything" searches.   I was able to find out what was too low a boost and what was too high a boost by running my test suite against different values.

And of course, our relevancy tests were crucial in my effort to simplify our boost factors.

Please feel free to take ours (https://github.com/sul-dlss/sw_index_tests) and modify them for your own needs.   Ignore the ones that don't apply to you; add tests for your local needs. Improve on my methodology.  Feel free to get in touch.

Next:  Actually Working on CJK

Now that we've gotten past the difficulties with changing from dismax to edismax query parsing, we can utilize Robert Muir's fix for SOLR-3589, the bug that was setting mm to 0 for fields using CJKBigramming (this is discussed in the second post in this series).  We also upgraded to Solr 4.3 at this point (from 3.5), partly to get Robert's bug fix without having to apply a patch to Solr manually.

So the next post in this series will return to discussing work specific to CJK.