Search for content in message boards

Announcing Relevance Test II

Announcing Relevance Test II

Posted: 26 Oct 2012 3:59PM GMT
Classification: Query
Hi all,

We are in the process of rolling out our next test to verify the default behaviors on new search are the correct default behaviors. You may recall that our earlier test indicated we should remove Soundex matches from the default experience. This test will help us determine if we should, by default, tighten search results even more.

The changes we will be testing include:

1) Whether Phonetic algorithm matches should be included by default on new search. For this test, we will be comparing how successful searchers are under the current default behavior (with Phonetic matches on) vs. Phonetic matches off (we'll still include "similar" matches).

2) Whether there should be some degree of first name matching. Currently the website does not require any matching on first name at all. This is why you can get results for Peter Blow when you search for Joe Blow. For this test we will compare successes from the current default behavior to one where the results must match exactly, or phonetically, or similar, or because of initials.

We will be analyzing performance of the different experiences across different user types and deciding upon the configuration that works best for the most users. (I think I have a pretty good idea about what configuration you all will be pulling for.)

As with our last test, this test reflects our commitment to taking a reasoned approach to significant changes to the user experience.

Thanks,

John

Re: Announcing Relevance Test II

Posted: 26 Oct 2012 4:40PM GMT
Classification: Query
John,

Glad to see this is going forward!

Andy

Re: Announcing Relevance Test II

Posted: 27 Oct 2012 2:37PM GMT
Classification: Query
I'm still a little puzzled how you define "success" for these kinds of tests. Perhaps your methods are proprietary and cannot be publicly disclosed (which is fine!), but it makes me no less curious. :)

For example, if one user searchs for Joe Blow and gets a match on Peter Blow, the results might be total garbage. But another user might also search on Joe Blow and get a match on a Peter Blow who is really Joseph Peter Blow and who therefore may be the exact person who was being searched. How do you tell the difference?

Jerry

Re: Announcing Relevance Test II

Posted: 27 Oct 2012 3:34PM GMT
Classification: Query
John,

Testing search engine parameter improvements is good.

If "success" is principally measured by whether users save-to-tree, I hope that some factor such as subtracting 50% will be applied.

A great many newbies have little idea of what constitutes a "match." One of the recent examples I have noted is frequent saving of a US Census item for southwestern PA to a tree family who never left North Carolina according to the myriad copied trees. This item's relevance is suggested by the hints system because one treebie did the initial tree save, and the hints system may be giving it greater relevance (to wrong family, mind) weighting as others followed suit.

Cheers,
Jade

Re: Announcing Relevance Test II

Posted: 28 Oct 2012 1:20PM GMT
Classification: Query
Also, I never "save to tree" because I'm not presently keeping a tree online. Rather, my tree is kept 100% on my local computer. But I do have tremendous success when searching (exact mode only, of course, but I still would love to be able to use ranked searches effectively).

Jerry

Re: Announcing Relevance Test II

Posted: 29 Oct 2012 6:33PM GMT
Classification: Query
Successes include activities such as saving to computer, saving to tree, printing, etc. We look at these metrics in aggregate as well as through segmented groups to make sure benefits to one user type (e.g., newbies) don't make things worse for experienced users.

Re: Announcing Relevance Test II

Posted: 11 Mar 2013 5:10PM GMT
Classification: Query
"We look at these metrics in aggregate as well as through segmented groups to make sure benefits to one user type (e.g., newbies) don't make things worse for experienced users."

Newbies aren't using the search function. They rely on hints, and attach without evaluation of the data. If the metrics are relying on attachment, well then that accounts for why the search function becomes more useless every day.

This mindless attaching in concert with the massive volume of *alternate* data being entered by inexperienced users and then indexed by the system is having an adverse effect on search results, the pages of irrelevant returns is staggering. Something's gotta give.


per page

Find a board about a specific topic

  • Visit our other sites:

© 1997-2014 Ancestry.com | Corporate Information | Privacy | Terms and Conditions