Tuesday, February 8, 2011

Expressing (Search Result) Expectations as Cucumber Scenarios

As many of you know, SearchWorks is Stanford's Blacklight instance providing a "next generation" User Interface for materials at the Stanford Library.  What follows is a document I wrote for internal use so that motivated staff could provide feedback in the form of cucumber scenarios.  This blog post might make more sense in the context of my presentation at Code4Lib 2011 ... but this seemed a worthy blog post nevertheless.  Don't be put off by the length - a lot of what follows is examples.   

   

How to Write SearchWorks Search Result Expectations as Cucumber Scenarios

Sometimes we ask folks to check something new in SearchWorks. (thanks for your help!)
Sometimes people notice problems and report them: via the feedback form (thanks!), or a direct JIRA ticket (thanks!) or via email (less optimal, thanks!)
Occasionally people tell us specifics of something that is working "correctly."
When you ask yourself questions like these, then you are doing a "manual" test:
  • "Is SearchWorks getting the right search results?"
    • "SearchWorks is getting the right results because ..."
    • "The results in SearchWorks aren't ordered correctly. They should be ..."
    • "I know SearchWorks is wrong because ..."
  • "Are things displaying correctly?"
  • "The vernacular title should be ..."
Whenever you test something ("manually") in SearchWorks, we would like to capture your expectations as a cucumber scenario so we can run the test repeatedly and automate it.
Benefits:
  1. we won't have to keep asking you to check the same things over and over. Imagine never having to perform a given test search again!
  2. we can ensure that applying a fix for one problem won't inadvertently break something we've already fixed.
  3. we can automate running a large suite of tests nightly so we keep checking that we haven't broken anything.
  4. as we add specific searches and expected results against our own (meta)data corpus, we are accruing relevancy tests for our own data, based on human review of search results.
Sadly, this does not mean we can make all tests pass – sometimes, we can't achieve the ideal. There may be unacceptable tradeoffs, or it might just be too difficult technically to warrant pursuit. We do have a way to hang on to tests that should pass but are not passing at the current time, so these sorts of tests are welcome as well.
We would still like JIRA issues filed for FAILING cuke tests, and the JIRA issue identifier put in the scenario description.
The tests are easy to write.
Here are some sample cucumber scenarios:
Scenario: Query for "cooking" should have exact word matches before stemmed ones (VUF-123)
  Given a SOLR index with Stanford MARC data
  And I go to the home page
  When I fill in "q" with "cooking"
  And I press "search"
  Then I should get ckey 4779910 in the first 1 result
  And I should get result titles that contain "cooking" as the first 20 results

Scenario: relevance sort explicitly selected: score, then pub date desc, then title asc (SW-175)
  Given a SOLR index with Stanford MARC data
  When I go to the home page
  And I follow "Newspaper"
  And I select "author" from "sort"
  And I press "sort_submit"
  And I select "relevance" from "sort"
  And I press "sort_submit"
  # alpha for 2007
  Then I should get ckey 7141368 before ckey 7097229
  # newer year (2007) before older year (2005)
  And I should get ckey 8214257 before ckey 5985299
There are more samples at the bottom of this page.

Basic concepts

We use Cucumber (http://cukes.info) to automatically test the behavior of SearchWorks. It matches specific language in the "scenario" (cucumber parlance) with actions to perform, like filling in the search box, and then hitting return. Or clicking a facet link. Or going to a particular record to ensure information is properly displayed.
Cucumber does string matching (using regular expressions) to turn the natural language expressing expected behavior into executable test code. But you don't need to worry about it - just follow the specific language rules and you'll be supplying tests to the grateful engineers.

Scenarios

Each cucumber test is called a "scenario." It can have multiple actions taken, and what is displayed after each step can be examined. The idea is to capture how you're interacting with the web page (clicking buttons, selecting from pull downs, typing in text boxes) and what you expect to be displayed.

Be as precise as possible, BUT

We want the tests to be as useful as possible. Relevancy of search results can sometimes be as clear as
  • "record 666 should be the first result"
  • "the first 4 results should be"
  • "record 777 should be before record 999"
    There are more possibilities given in the "statements" section below.

Try to leave wiggle room for changes to our collection.

If a test is too rigid in its expectations, then small changes can make the test fail. These are the sorts of questions that help determine if the test is too brittle:
  • Are we likely to get more resources of the exact title you expect as the first result?
  • Are we likely to get more resources for the subject heading?
  • Are we likely to get resources that are a better match to the search terms?

Statements

  1. The quotes or absence of quotes in the statements below is important.
  2. When, Then, and And at the beginning of the statements are interchangeable.
  3. If you can't express your expectations with the statements below, please file a JIRA ticket telling us what you are writing a test for. We may be able to add more statements to enable the scenario.
(Step Definition Code for the following is available at
http://www.stanford.edu/~ndushay/code4lib2011/search_result_steps.rb)

All Scenarios must start:
All Scenarios must start:
Scenario: (free text description, keep it short) (JIRA issue identifier)
  Given a SOLR index with Stanford MARC data

Indicate Your Starting Point

  • When I go to the home page
    • Use this for searching scenarios.
  • When I go to the advanced search page
  • When I am on the show page for "________"
    • Use this when you are talking about a particular record.
    • Fill in the blank with an id (ckey).

You're At Your Starting Page; Now Do Something.

Fill in a Text Box
  • When I fill in "q" with "___________________"
    • Use: searches without quotes.
    • Fill in the blank with any string for the search text box (no quotes allowed).
      • When I fill in "q" with "gobblety gook"
      • When I fill in "q" "under the sea-wind"
      • When I fill in "q" with "Shindy AND Delilah"
  • When I fill in the search box with "_________________"
    • Use: searches containing quotes.
    • Fill in the blank with any string for the search text box, and if there are quotes, escape them with a backslash
      • When I fill in the search box with "\"under the sea-wind\""
Pressing a Button
  • And I press "________"
    • Use: pressing a button
      • And I press "search"
      • And I press "per_page_submit"
Selecting from a Pulldown
  • And I select "_____" from "______"
    • Use: selecting a value from a pull-down. (If you don't know the official name of the pulldown, we'll figure it out.
    • Fill in the first blank with the selected value; fill in the second blank with the name of the pulldown.
      • And I select "Title" from "search_field"
      • When I select "author" from "sort"
      • And I select "100" from "per_page"
Following a Link
  • And I follow "____________"
    • Fill in the blank with the link text (NOT the url it goes to)
    • This is how we select facets in testing.
      • And I follow "Journal/Periodical"
      • And I follow "Hoover Library"
      • And I follow "Chinese"
Checkbox Selection and Un-Selection
  • I check "____________"
  • I uncheck "____________"
    • Fill in the blank with label of the checkbox (the text displayed next to it)
Radio Buttons
  • I choose "____________"
    • Fill in the blank with label of the selected radio button (the text displayed next to it)

Look at What You Got Back

  • Then I should get results
  • Then I should not get results
    • Use this only if
      1. you can't provide at least one id (ckey) OR
      2. you can't provide a ballpark number of expected results
  • Then I should get (at least|at most) ___ results
    • Use when the number of results is less than the default number per page (currently 20)
    • Pick the appropriate qualifier; fill in the blank with a number
      • Then I should get at least 2 results
      • Then I should get at most 19 results
  • Then I should get (at least|at most) ___ total results
    • Use when the number of results is more than the default number per page (currently 20)
    • Pick the appropriate qualifier; fill in the blank with a number
      • Then I should get at least 250 total results
      • Then I should get at most 50 total results
  • Then I should get ckey _______ in the results
  • Then I should not get ckey _______ in the results
    • Fill in the blank with a single ckey expected (or not expected) in the first page of results.
    • The latter can be used to exclude false positives.
  • Then I should get ckey _______ in the first ___ results
  • Then I should not get ckey _______ in the first ___ results
    • These are good statements when a particular record should be "above the fold" or should clearly be the first result, or when you want to ensure a particular false positive isn't polluting the top search results.
    • Fill in the first blank with a single ckey, and the second blank with a number lower than the default number per page (currently 20). The last word may be result or results.
      • Then I should get ckey 12345 in the first 1 result
      • Then I should get ckey 12345 in the first 3 results
  • Then I should get ckey _______ before ckey _______
    • Use this to specify result ordering, such as after a particular sort.
  • Then I should get (the same number of|fewer|more) results (than|as) (a|an) (.)search for "_______"*
    • Use: compare number of results with different search
    • "than" and "as" are interchangeable, as are "a" and "an"
    • "title" "author" "subject" may be put before search to indicate a specialized search.
    • query string may contain quotes - but they must be escaped with a backslash
      • Then I should get fewer results than a search for "wonderbread"
      • Then I should get more results than an author search for "\"James Herriot\""
      • Then I should get the same number of results as a title search for "jack in the beanstalk"
  • Then I should get at least ____ of these ckeys in the first ___ results: "______________"
    • fill in the first two blanks with positive integers, fill in the blank with a list of ckeys separated by comma-space: "1234, 23324, 1523"
      • Then I should get at least 4 of these ckeys in the first 4 results: "7637875, 336046, 6634054, 2130330"
  • Then I should get ckey _______ and ckey _______ within ___ positions of each other
    • Then I should get ckey 6974167 and ckey 5757985 within 2 positions of each other
  • Then I should get result titles that contain "______________" as the first ___ results
    • Use when you think a term or phrase in the title will be a less brittle test than ckeys. (originally used to detect if exact matches sort higher than stemmed matches.)
      • Then I should get result titles that contain "arabic" as the first 20 results
  • Then I should see "______________"
  • Then I should not see "______________"
  • Then I should see "______________" (at least|at most|exactly) ___ times
    • Use when you want to find visible text somewhere on the page. Generally too vague for search tests.
      • Then I should see "Carnoy, Martin"
      • Then I should see "Refine" exactly 2 times

Facet Expectations

  • Then the facet "______________" should display
  • Then the facet "______________" should not display
    • Then the facet "Russian" should display
    • Then the facet "Choctaw" should not display
  • Then I should get facet "_____________" before facet "_____________"
    • Then I should get facet "Croatian" before facet "Czech"

Call Number ordering in show view

  • Then I should get callnumber "_____________" before callnumber "_____________"
    • Then I should get callnumber "505 .S343 V.20 1972" before callnumber "505 .S343 V.21:1 1973"

Example Scenarios

Examples: Simple Searches
Scenario: Query for "cooking" should have exact word matches before stemmed ones (VUF-321)
  Given a SOLR index with Stanford MARC data
  And I go to the home page
  When I fill in "q" with "cooking"
  And I press "search"
  Then I should get ckey 4779910 in the first 1 result
  And I should get result titles that contain "cooking" as the first 20 results

Scenario: Expect specific match and non-match for "french beans food scares" without quotes (VUF-123)
  Given a SOLR index with Stanford MARC data
  And I go to the home page
  When I fill in "q" with "french beans food scares"
  And I press "search"
  Then I should get ckey 7716344 in the first 1 result
  And I should NOT get ckey 6955556 in the results
Examples: Specialized Searches
Scenario: Single Author Title search matches Socrates results (SW-5)
  Given a SOLR index with Stanford MARC data
  When I go to the advanced search page
  And I fill in "author" with "McRae"
  And I fill in "title" with "Jazz"
  And I press "advanced_search_button"
  Then I should get at least 4 of these ckeys in the first 4 results: "7637875, 336046, 6634054, 2130330"

Scenario: Search for non-existent author should yield zero results (VUF-5)
  Given a SOLR index with Stanford MARC data
  When I go to the home page
  And I fill in "q" with "jill kerr conway"
  And I select "Author" from "search_field"
  And I press "search"
  Then I should get at most 0 results

Scenario: Stopwords in title searches should be ignored - 3 terms total (SW-14)
  Given I am on the home page
  When I fill in "q" with "alice in wonderland"
  And I select "Title" from "search_field"
  And I press "search"
  Then I should get at least 100 total results
  And I should get the same number of results as a title search for "alice wonderland"

Scenario: Thesis advisors (720 fields) should be included in author search (SW-3)
  Given a SOLR index with Stanford MARC data
  And I go to the home page
  When I fill in "q" with "Zare"
  And I select "Author" from "search_field"
  And I press "search"
  Then I should get at least 10 results
  And I should see "Thesis"
Example: Multi-Button Presses
Scenario: relevance sort explicitly selected: score, then pub date desc, then title asc (SW-666)
  Given a SOLR index with Stanford MARC data
  When I go to the home page
  And I follow "Newspaper"
  And I select "author" from "sort"
  And I press "sort_submit"
  And I select "relevance" from "sort"
  And I press "sort_submit"
  # alpha for 2007
  Then I should get ckey 7141368 before ckey 7097229
  # newer year (2007) before older year (2005)
  And I should get ckey 8214257 before ckey 5985299

Scenario: Call Number
  Given a SOLR index with Stanford MARC data
  When I am on the home page
  Then I should see "Archive of Recorded Sound"
  When I follow "Archive of Recorded Sound"
  Then I should see "[remove]"
  And I should get at least 10 results
Example: Non-Latin Script, Per Page selection
Scenario: Cyrillic (VUF-22)
  Given a SOLR index with Stanford MARC data
  And I go to the home page
  When I fill in "q" with "пушкин pushkin"
  And I select "Title" from "search_field"
  And I press "search"
  And I select "50" from "per_page"
  And I press "per_page_submit"
  Then I should get at least 12 results
  And I should get ckey 216398 in the results
  And I should get ckey 7898778 in the results
Example: Selecting Facet
Scenario: japanese journal of applied physics PAPERS - 780t, 785t indexed (VUF-11)
  Given a SOLR index with Stanford MARC data
  And I go to the home page
  When I fill in "q" with "japanese journal of applied physics papers"
  And I select "Title" from "search_field"
  And I press "search"
  Then I should get at least 7 of these ckeys in the first 8 results: "365562, 491322, 491323, 7519522, 7519487, 460630, 787934"
  When I follow "Journal/Periodical"
  Then I should get at least 5 of these ckeys in the first 5 results: "7519522, 365562, 491322, 491323, 7519522"
Examples: Call Number Sorting in Record
Scenario: The show view call numbers should be in volume reverse sort order for serials (VUF-666)
  Given a SOLR index with Stanford MARC data
  When I go to the show page for "370790"
  Then I should get callnumber "570.5 .N287 V.25-26 1935" before callnumber "570.5 .N287
Posted by at 3 comments:
Subscribe to: Posts (Atom)