You might be interested in this post in particular if you use Solr's edismax query parser or if you want to get more exact matches for user queries.
Edismax Woes, Part 2In the third part of this series, I mentioned that we had 21 relevancy acceptance tests fail when we used the edismax Solr query parser instead of dismax. Recall that our test failures were in the following categories:
- Journal titles
- Hyphens preceded by a space (but no following space)
- Boolean NOT
- Synonyms for musical keys
Digging Into Relevancy DifferencesHere is example output from some failing tests in category 1:
rspec ./spec/journal_title_spec.rb:22 # journal titles The Nation as everything search
rspec ./spec/journal_title_spec.rb:32 # journal titles The Nation (National stems to Nation) with format journal
I used manual searches to confirm that users would perceive the edismax results as worse than dismax. I also tried a number of similar searches (and wrote tests) to better pinpoint this problem.
Here are the first five results for a title search on The press using using dismax and edismax with the same index:
Looking at these results, I immediately have two questions:
- Why are the first two results of edismax not exact title matches?
- Why are the scores of the three documents that appear in both sets of results different?
Edismax Formula DifferenceTo answer these questions, I first looked at the Solr output with debugQuery=true. The Solr query analysis was shown as exactly the same EXCEPT for this difference noted in SOLR-2058 in the comment from Michael Dodsworth on 25 Sep/12.
I'll try to express it more succinctly (thanks to Tom Burton-West) :
A = field1:"term1 term2"
B = field2:"term1 term2"
C = field3:"term1 term2"
Dismax: DisjunctionMaxQuery (A|B|C)
returns the score of whatever field has the highest score.
returns the sum of the scores of each of any of the above queries that match. So if your phrase is in all 3 fields, you get the sum of the scores for each matched field.
Unfortunately, SOLR-2058 is marked as fixed, despite this difference. So we have an initial diagnosis, but not a treatment plan.
Visualizations To The Rescue!I was stumped as to what to do about the above until I used the visualizations of Solr scoring data made available at explain.solr.pl. The full visualizations I created for scoring the top 5 results of a search for 'the press' are available here:
Here is the visualization of the score of the first dismax result, id 9162486 with title The press:
However, with edismax, we have two basic patterns within the first 5 results, and the phrase match in the unstemmed short title field is not nearly as dominant:
Here is a closer look at the visualization of the score of the first edismax result, id 8192320 with title MEET THE PRESS:
What I learned from these visualizations is that to make the edismax results more like dismax, I needed a way to give more weight to exact matches of the entire string in the title_245a_unstem_search field.
Another visualization example, of the top 5 results of a search for 'the nation':
Tie Parameter"DisMax" is an abbreviation of "disjunction maximum", which is a partial description of the way user queries are turned into low level Lucene queries. From http://wiki.apache.org/solr/DisMax, dismax is "designed to process simple user entered phrases (without heavy syntax) and search for the individual words across several fields using different weighting (boosts) based on the significance of each field." From the same document:
So basically, dismax pays attention only to the highest scoring query match in any of the document's fields. The tie parameter, documented here, can be used to dial up the influence of other matches:
A tie value of 0.01 was used for both dismax and edismax searches. That value is very close to zero, so the highest scoring matching clause should already be dominating the total score. I tried a tie value of 0.99 to see if I could make a difference in the relevancy ranking this way, and while my document scores changed, the result order remained the same. Below are results for edismax search for 'the press' with different tie values:
tie 0.01: http://explain.solr.pl/explains/hwi1ma6x
tie 0.99: http://explain.solr.pl/explains/jfmdli56
I coerced a colleague into trying a similar experiment, and was able to confirm the tie parameter affected his results as expected. Further experimentation indicated that setting the tie value to something like 0.00001 for my search DID have the effect of making the highest scoring matching clause highly dominant; this led me to the conclusion that my field boosting values were ridiculous (as high as 200,000 !) and affected my ability to use a reasonable value for the tie parameter..
Re-adjusting Boost ValuesOur boost value settings were a big nasty kludge that I partly inherited and partly created. Simplifying them was definitely in order ... but perhaps not at the same time as a change to edismax, nor during a Solr upgrade from 3.5 to Solr 4. Sure, having about 600 relevancy tests gave me a decent amount of confidence I could revise my boost values without degrading the user experience of relevancy, but upgrading to edismax was causing me plenty of difficulty without revising boost values.
Eventually, I did greatly simplify our boost values and our relevancy tests were indispensable to that process. Of course I de-coupled this from fixing the problems we had with edismax, and again from the Solr upgrade from 3.5 to Solr 4. You can see our simplified boost values in our solrconfig.xml file at https://github.com/solrmarc/stanford-solr-marc/blob/master/stanford-sw/solr/conf/solrconfig-slave.xml. The highest boost value is now 5,000, a great reduction from 200,000.
However, in terms of addressing the relevancy test failures with edismax, adjusting the boost values and the tie parameter wasn't really part of the route taken.
Back to Edismax WoesThe visualizations of Solr scores helped us see that our edismax results need to give more weight to exact matches of the entire string in the short title (title_245a_unstem_search) field. The tie parameter wasn't really working for us. What now?
Bill Dueber of the University of Michigan wrote a wonderful blog post on using "fully-anchored" text fields to get an "exactish" match. This approach seemed well worth a try. I will discuss our fully-anchored solution in the next post in this series.