Search for content in message boards

Announcing Web Search

Announcing Web Search

Posted: 13 May 2011 3:36PM GMT
Classification: Query
Edited: 13 May 2011 3:39PM GMT
If you are here, you probably saw the blog post announcing the ability for Ancestry users to find relevant records about their ancestors – even if they are on other websites.

Our users keep telling us they want more content to help them understand their family history and build and source their family trees. At Ancestry.com, we are continually digitizing and adding more records to our service to help our users with their family history. At the same time, we also realize that Ancestry.com will never hold all of the content out there and there are many organizations and individuals who are publishing a lot of complementary genealogical and historical content. We have also heard from a number of these publishers that they want to have more people find their websites and this information.

This information lives on thousands of websites but, as a family historian, it isn’t always easy to find and it can take a lot of effort to figure out which websites contain information about your ancestors. We wanted a way to help Ancestry.com users find this great content. At the same time, we wanted it to be a win for the website publishers.

Last fall, we launched this idea in Ancestry.com Labs. We provided the first glimpse of how we might approach this. We got feedback from many users and publishers. Some of it was very glowing and some of it definitely wasn’t. We listened to all of it to try to develop an experience that was helpful to users and, at the same time, respectful to publishers. It’s not easy trying to balance all of the many ideas and needs but we hope we are doing it right.

PRINCIPLES OF WEB SEARCH

In building this system, we want to be forthright about the principles we are striving to follow. Some of the things we heard from the start are that many publishers want both credit and traffic from any such service, so they want us to make it easy for users to find their records and get to their site.

To accomplish this, and in the spirit of openness, we want to state up front the key principles we are following as we build this service. For simplification, I will refer to the system that searches this information as “web search” and the content itself as “web records.”
• Free access to web records – Users do not have to subscribe or even register with Ancestry.com to search and view these records.
• We will always strive to follow web standards for web crawling permissions. For example, some websites have a robots.txt file that instructs search engines (like Google) to not crawl the site, or to only crawl certain areas.
• Proper attribution of web records to content publishers - we will link prominently to the original site within the search experience.
• We have in place processes to remove content from the index if a website owner requests us to do that and we will publish how to contact our team to do this. Website owners can also contact us to ask questions or to request their site be indexed – see this page to learn how to contact us: http://www.ancestry.com/websearch.
• Ancestry.com users will be able to save key information to their trees but it will list the website as the source and will have an easy way to link back to the original site.

We really want to maintain trust and openness with our users and with website publishers. This is part of an ongoing dialog starting back last year and is continuing with this launch.

If you have questions, concerns, or suggestions, please feel free to comment here on these boards or email us at websearch@ancestry.com. Either way, we will respond.

We are only launching a search to a few websites initially because we want to continue to let people know what we are doing and to learn as we go from our users and from you, the people and organizations that publish these websites. As we continue to learn, you will see this service expand to include more websites.

HOW DO I FIND OUT WHAT INFORMATION OR WEBSITES YOU HAVE SEARCHED?

This information will show up in search results on Ancestry. To see an example of this, you can do a general (global) search on Louise M Chrisman in Indiana, USA. This link will show the search results:
http://search.ancestry.com/cgi-bin/sse.dll?gl=ROOT_CATEGORY&...

Looking at the search results, you will see a record from “Web: Allen County, Indiana Deaths 1870-1920.” We use the “web:” prefix to denote records found via web search and to let users know that they are not from Ancestry.com. If you are finding these records from global search, you will also find a link that goes straight to the website.

You can also go to the card catalog and search on the word “web” in the title to see a list of what has been indexed. If you search just on one collection, you will find a link to the website on the search form and on the index page.

Here is an example of a way to search just one web collection: http://search.ancestry.com/search/db.aspx?dbid=70001.

HOW DO I PREVENT ANCESTRY.COM FROM INDEXING MY SITE?

The easiest way to prevent us, or other search engines, from searching your site is to add instructions in the website’s robots.txt file. Ancestry.com’s crawler is called “ancestrybot”. You can easily instruct Ancestry and other search engines which areas of your site you want searched, or not searched. You can also restrict all search engines, or just specific ones, like ours, through this file. Because we follow web standards, we will honor the instructions in the robots.txt file.

To learn more about how to use a robots.txt file, there are many sites with tutorials or instructions. Here is one such site: http://www.robotstxt.org/robotstxt.html.

If your site is included in the search but you don’t want it there, you may contact us at websearch@ancestry.com and we will remove it from our search index.

HOW DO I ASK ANCESTRY.COM TO INCLUDE MY WEBSITE IN THE SEARCH?

You can contact us at the same email address: websearch@ancestry.com to request us to include your site in the index. Please also make sure that your robots.txt doesn’t prevent crawling of your site.

SUMMARY

As I stated at the start of this long post, we hope this is a great service for family historians and for web publishers as well. We expect it will help many more people find your website and the great content you are publishing and hope that we have done this in a way that is win-win for everyone.

We are happy to answer questions, either here on this message board or via the email address listed above.

Please try out the service, kick the tires, and let us know what you think.

Thanks,

Brian Edwards
Director of Search Product Management

Gentry Davies
Search Product Manager

Re: Announcing Web Search

Posted: 14 May 2011 5:14PM GMT
Classification: Query
Edited: 14 May 2011 5:16PM GMT
Mr. Edwards,

Although I have written to you directly, I will also post my question here for the benefit of others.

Are "publishers" (as you call us) of RootsWeb-hosted sites allowed to "opt out" of this new search feature? I ask because of the 'Acceptable Use Policy' for such sites stating "By submitting Submitted Content to RootsWeb.com, you grant Ancestry, the corporate host of the Service, a limited license to the Submitted Content to use, host, and distribute that Submitted Content and allow hosting and distribution on co-branded Services of that Submitted Content."

Ancestry.com uses this statement to deny copyright infringement claims DESPITE the fact that Ancestry's tree users are violating both the Terms & Conditions ("You should submit only content which belongs to you and will not violate the property or other rights of other people or organizations.") and the Content Submission Agreement ("Be aware that content, including photographs, even if submitted to a site of which you are a member, belongs to the creator or submitter and you should not reproduce it without permission of the owner").

Is it possible to allow a site to be searched/indexed on Web Search but NOT allow it to be "saved" to user trees? It would be far more preferable, to publishers such as myself, to change this option to "Create a Web link" to the web-site containing the info. In my opinion, there is absolutely no reason for Ancestry.com users to copy from the original site when they could just as easily utilize the Web Link feature and, therefore, not infringe copyrights. However, the users I have corresponded with regarding the matter have all stated they want the information/item (such as photos) ON their tree. Ancestry not only does not discourage this, it actually ENcourages copyright violations from one tree to another with the "Save this to my tree" button. The first user may indeed have permission, but subsequent users simply add it to their tree without any interaction with either the owner of the material or the first Ancestry tree user.

Quite frankly, (in my opinion) this new feature is going to lead to more copyright infringement, not less.

Re: Announcing Web Search

Posted: 14 May 2011 5:18PM GMT
Classification: Query
By the way, why isn't this board (Content Publishers Feedback) listed in the Message Board hierarchy? I could not access it from there but had to use the link on the Web Search page
http://www.ancestry.com/websearch

Re: Announcing Web Search

Posted: 15 May 2011 3:22AM GMT
Classification: Query
Thanks for the question. There are a lot of different types of Rootsweb sites and I don't think I have a definitive answer for all the different types of sites and areas.

The general idea is that when people own a website and publish information, they control that information and can opt out at any time. From my understanding, that does apply to sites that are controlled and put up by societies and only hosted by Rootsweb/Ancestry.

I am going to dig into this to get a better handle on the different types of pages and information on Rootsweb and see if I can get a better definition.

If you have specific examples, or if there is an area where you are the administrator, please send me those examples and I can give you a better answer.

Thanks!
Brian

Re: Announcing Web Search

Posted: 15 May 2011 3:28AM GMT
Classification: Query
When we set it up, the intent was the it be a little more private, or at least semi-private, area to discuss topics and questions that are specific to website publishers.

Since it was focused on one specific group and wasn't intended for the general Ancestry user, we removed it from the general message board hierarchy.

It's not deeply hidden, as you can tell, but we wanted to encourage an open discussion with just the publishers and thought having it be a little more personal space might encourage that frank conversation.

Re: Announcing Web Search

Posted: 15 May 2011 5:48AM GMT
Classification: Query
Edited: 15 May 2011 5:49AM GMT
That was pretty much what I thought - and as a result, I'll be removing my site from RootsWeb.

What about my suggestion to "Add a Web Link" rather than "Save to my tree"? Most Ancestry users do not know the Web Link feature even exists because it is in such an obscure location.

And what about the suggestion to add a cautionary notice about copying from other web-sites? Quite frankly, Ancestry claims to be "sensitive to the rights of others' but my experience is that statement is only lip service (or legally covering your backside).

Re: Announcing Web Search

Posted: 15 May 2011 5:50AM GMT
Classification: Query
If someone can't access it from the Message Board hierarchy and it does NOT appear on searches (even for the exact name of the board), then it is "deeply hidden".

Re: Announcing Web Search

Posted: 15 May 2011 8:03AM GMT
Classification: Query
This post was deleted by the author on 16 May 2011 6:46PM GMT

Re: Announcing Web Search

Posted: 15 May 2011 9:01AM GMT
Classification: Query
_Nan_ Posted: 15 May 2011 3:03AM

"I clicked on "go to website" again and this time correctly clicked on "open" and I am now at the Allen Co site...but...I now have to do a search for Mary's information! "

Nan, had you bothered to read the message on the Ancestry Index page it plainly says:

[Quote]
This record is not from Ancestry and will open in a new window. You may need to search for the record when the web page opens. For more information on web records, click here.
[End Quote]

You may not have been lobotomized but you sure as heck aren't very observant.

Re: RootsWeb Collections

Posted: 16 May 2011 3:08PM GMT
Classification: Query
I am going to split my reply to your questions into different posts to focus on one thing at a time.

You can certainly remove your site from RootsWeb but, let me explain a little more about what I meant in that earlier post.

We are trying to be careful about publishers' desires to allow, or prevent, their site or collection from being searched on Ancestry. Any site owner/publisher can request their site be removed from the search and we will comply.

What I am not sure about is, when a site has content contributed by many people, how do we determine who has the right to determine whether to remove the site from the search?

I can certainly see situations where many people contribute to a single database and where some of them want it to be searched and found by Ancestry users while others do not. Ideally, there would be one person who is the owner or administrator of the site who would be able to make the decision. I want to avoid a tug of war where we end up with many people arguing about whether it is searchable or not and we make the wrong call.

Do most sites and databases have a single administrator or owner? For sites and collections where the ownership is clear, it will be easy to respond and add or remove it from being searched.

I am going to talk to a few people who know about Rootsweb's policies and organizational structure better than I do to better understand this. I think in most cases, this will be easy. I just don't understand all of the different levels of ownership we might encounter on this.

I would also appreciate your thoughts and suggestions here. How would you suggest we determine who is responsible for a collection or site?

Thanks!
per page

Find a board about a specific topic

  • Visit our other sites:

© 1997-2014 Ancestry.com | Corporate Information | New Privacy | Terms and Conditions