When to perform a Search full crawl in SharePoint 2013?

Search is indeed a mission critical component in SharePoint 2013 and it’s very important that it functions properly so that you get the desired results. As we all know, the search results and their relevancy is directly proportional to how often your content sources are crawled and what sort of crawling you’re running in your SharePoint farm (i.e. full crawl, incremental crawl and continuous crawl). So, in this post I’m not going to discuss about the different type of search crawls or the SharePoint 2013 search architecture, perhaps I would be discussing on when and under what circumstances should a SharePoint administrator perform a full search crawl. The reason for me picking up this topic is because I see a lot of misconception among SharePoint administrators in understanding when the Search full crawl has to be performed. For the most part, I’ve seen many folks turning on full crawl when it’s not required at all and before doing so we need to understand that turning on Search full crawl is going to consume a lot of your server’s resource and at worst case it could even make your SharePoint farm go to an unresponsive state and hence it’s very important that we do this only when it’s required.

1

Alright, let’s get into the details ….

Listed below are the reasons why and under what circumstances should a SharePoint farm Administrator perform a full search crawl:

1.You just created a new Search Service application and the default content source (i.e. Local SharePoint sites) that gets created along with the newly created Search service application hasn’t been crawled yet.

22. You recently added a new content source and it hasn’t been crawled yet (Note: This is applicable for all the types of content sources (i.e. Local SharePoint sites, File shares, Exchange public folders and External line of business data)

3.png

3.When there has been, a change made to the existing content source (meaning, when you’re trying to edit the existing content source for making some changes)

4.png

4.When you’re patching your SharePoint 2013 farm by installing a Cumulative update, Service packs and hot-fixes etc. For some reason I see a lot dilemma on this specific point because it brings up a question on why should a full crawl be performed post the patching .The reason for this is really simple ,  if you read my article on patching a SharePoint farm you would notice that I’ve mentioned a step where you need to suspend the search crawl before patching your farm and the reason for mentioning that is because it’s quite possible that when you check the crawling schedule  before patching you farm there may not be any instance of crawl running. However, if a crawl is triggered by schedule which occurs during the installation, the search application may crash or lead to inadvertent results. In worst case, you might end up rebuilding the entire search application. Hence, as a best practice it’s very important that you suspend the search service application before patching your farm and once you’re done with patching your farm please go ahead and resume it and run a full crawl.

5

6

7

5.When changes have been made to managed properties in search. A full crawl of all affected content sources is required for the new or changed managed property to take effect.

8.png

6.If you want to detect security changes that were made to local groups on a file share after the last full crawl of the file share

9.png

7.When the incremental crawl keeps failing continuously. If an incremental crawl fails many consecutive times for any content, the system removes the affected content from the search index. In such case, please look into the search crawl logs and try to identify the issue and fix it after which you need to run a search full crawl so that the failed content gets updated in the search index.

8.If you have made some changes to the search Crawl rules such as adding, deleting or modifying the crawl rule.

10.png

9.When your search index gets corrupted you need to perform a search index reset after which you need to run a full search crawl. Please check my article on search index reset to understand how to perform an index reset and under what circumstances should you be performing a search index reset.

11.png

10.The permissions given to the default content access account has been changed.

11. Apart from the above mentioned one’s the system by itself would be performing a search full crawl even when an incremental or continuous crawl is scheduled under the following circumstances:

   a)The SharePoint administrator stopped the previous crawl.

    b)A content database was restored, or a farm administrator has detached and reattached a content database.

    c) A full crawl of the content source has never been done from this Search service application.

     d)The crawl database does not contain entries for the addresses that are being crawled. Without entries in the crawl database for the items being crawled, incremental crawls cannot occur.

Thanks for reading this post. Happy SharePointing!!!

Resetting Search Index in SharePoint 2013:

searchadministration_thumb2

Search is indeed a mission critical service application in SharePoint and it’s very important that it remains healthy so that you get the results in the Search center whenever you execute a search query. However, it’s quite possible that you may notice some errors in the “Crawl log” stating that the crawler is not able to crawl the contents in the content source and at times it might get even worse such that you may notice all “failures” and no “success” in the crawl logs. In scenarios like that you might have to reset the search index so that the content in the index gets flushed away after which you need to perform a full crawl so that search index component receives the newly processed items from the content processing component and writes them to the search index. Of course, directly resetting the index is not a fix for any search related issue, please ensure that you do all the necessary troubleshooting steps from your end to identify and fix the issue and once you reach a point where nothing really helped, that’s when you should think about resetting the search index.

What happens when you reset the Search Index?

When you reset the search index, all the contents will be immediately removed from the search index and users will not be able to retrieve search results when they execute a search query on the search center.  Once you’re done resetting the search index, you must perform a full crawl on one or more content sources to create a new search index. Users will be able to retrieve search results again once the full crawl is finished and the new search index is created. So in a nutshell there will be a downtime when the search index is getting reset. After a search index reset, the full crawl won’t restore all the analytics features that are powered by the Analytics Processing Component. All the analytics results will be erased after resetting the search index.

When we should reset the search index?

As I already mentioned above, resetting the search index should be the final step of troubleshooting for any search related issue. However, based on my experience this when we should reset the search index ….

  1. Index is corrupted
  2. One or all index component status is degraded
  3. You crawl completed successfully but you are not getting the search results.
  4. When you move the index location
  5. When Index location run out of space.
  6. In a scenario where we have Index partitions spread across multiple servers and the indexed document count is out of sync. For example, we have 2 servers set as index partitions, server 1 has 150K indexed documents and server2 has 145k index documents
  7. If we make any changes to search topology such as removing the search components ,adding the components & activating a different search topology

 Alright, now let’s take a look at the steps to reset the search index …

Pre-requisites: 

Please ensure that you take care of the below mentioned pre-requisites before resetting the search index .Failing to do so will cause adverse effects and at worst case you may end up recreating the entire search service application.

  1. Make sure the crawl status for each content source is “Idle”. If any crawl is running, wait for the crawl to complete or follow the steps to “Stop the active crawls” below.
  2. Make sure continuous crawl for content sources is disabled [if applicable].This step is only specific to SharePoint 2013 and above.
  3. Make sure the crawl rate is 0
  4. Make sure background activity status is “None” on search administration page.

Stopping the Active Crawl(s):

  1. Navigate to search administration page.
  2. Click on Content Sources link towards your left hand side as shown in the image below.

4

3. If any content source is crawling, click on the drop down of the content source and click on “Stop Crawl”.

Disable Continuous Crawl:

  1. Navigate to search administration page.
  2. Navigate to Content Sources link appearing on quick launch.

4

3. Click on the drop down and select “Disable Continuous Crawl” to disable it.

5

4. Click “Ok” when below warning message appears.

6

5. Follow the same steps to disable continuous crawl for other content sources. Finally, all the content sources would show “Idle” status as shown in the image below.

7

Suspend Search Service Application:

This step is to ensure that there is no crawling activity in place while we perform the Index reset.

  1. Open SharePoint Management Shell using service account
  2. Run the below command.

Suspend-SPEnterpriseSearchServiceApplication “Search Service Application Name”

Note: If you don’t know the name of search service application, you can get it by running below command before executing the above command.

Get-SPServiceApplication | Where-Object {$_.TypeName –eq “Search Service Application”}

After suspending the search application, search administration page will show the status as “Paused for: External request” as shown in the image below.

8

 Ensure Crawl Rate is 0:

  1. Navigate to search administration page.
  2. Check the Recent Crawl rate.

9

Ensure Crawler Background Activity is none:

  1. Navigate to search administration page.
  2. Check the status of the Crawler background activity and ensure it shows as “none” as shown in the image below.

10

Once the necessary pre-requisites are taken care, please follow the below steps to perform an Index Reset.

There are 2 ways to perform Index reset.

Using UI [User Interface]: This method can be used when the indexed content in your farm is not much in size i.e. index size is not huge and also when you don’t have large content sources.

Using PowerShell: UI method is not recommended for SharePoint farms with large search index. [i.e. huge count of searchable/indexed documents]. Using the UI method in large farms for resetting the search index can result in time out error. In such cases, we need to use PowerShell method to reset the index.

USING UI:

  1. Navigate to SharePoint Central Administration.
  2. Under “Application Management” category, click on “Manage Service Applications”.
  3. Find out “Search Service Application” and click on the Search service application.
  4. In the quick launch, click on “Index Reset

reset-crawl-index-sharepoint-2013

5. Click on confirm once done .

USING POWERSHELL:

  1. Login to the SharePoint server using Administrator credentials.
  2. Open “SharePoint management shell” with elevated permissions.
  3. Run the below command to reset search index.

12

Post Index Reset Steps:

  1. Check whether the number of searchable items is 0.
  2. Resume the search service application by running the command shown in the screenshot below.

133. Enable “continuous crawl” for the content sources if it was enabled prior to index reset.

4. Run full crawl for the content sources one by one.

This confirms that you’re done resetting the Search index successfully and please keep monitoring the full crawl and ensure that it doesn’t get stuck in the middle. The time duration for the full crawl depends on the size of the content source, if you have very large content sources then it might take days for the full crawl to complete successfully. Once done, please verify the search results and ensure that they return fine. Also check the “Crawl log” and make sure you’re not seeing any errors this time.

Thanks for reading this post …..Happy SharePointing!!!

Search Service application components in SharePoint Server 2013:

 

1.png

As you’re all aware ,there has been a major change in the search architecture on SharePoint Server 2013 when compared to SharePoint server 2010 and in this post I’ll  be discussing on the six different search service components in SharePoint 2013 .It’s very important that all these components are running fine without any issue for the Search service to function seamlessly and surface the results in the search page when a user executes a query in the search center.

Listed below are the six components available in SharePoint 2013 search service:

  1. Crawl Component
  2. Content processing component
  3. Indexing component
  4. Query processing component
  5. Analytics processing component
  6. Search administration component

 Now, let’s take a look on all these components separately …

1.Crawl Component :

This component takes care of crawling the content sources such as (SharePoint sites, websites & file shares etc…) and extracts the crawled properties and metadata and sends that to the content processing component.

2. Content processing component:

This component receives the information from the crawl component and then processes and sends it to the indexing component. It also interacts with the analytics processing component and is responsible for mapping crawled properties to the managed properties.

3. Indexing Component :

This component takes care of receiving the information from the content processing component and writes it to the search index. It also takes care of handling the queries and sends back the results to the Query processing component.

4. Query Processing Component:

This component handles incoming query requests and sends them to the indexing component for results. It also takes care of query optimization.

5. Analytics Processing Component :

This component takes care of analyzing what users are querying on and how they interact with the results.  This information is used to determine relevance, generate recommendations and also used for generating search reports.

6. Search administration Component:

This component manages administrative processes as well as changes to the search topology, such as adding or removing search components and servers.

Please note that these 6 search components can be distributed across multiple servers to provide high availability as well as improve performance as shown in the image below.

2.png

Search service application databases:

  • Search Administration database :The Search Administration database hosts the Search service application configuration and handles crawl state orchestration, including the content source crawl history.
  • Analytics Reporting database :The Analytics Reporting database stores the results for usage analysis reports and extracts information from the Link database when needed.
  • Crawl Store database :The Crawl Store database stores the state of each crawled item and provides the crawl queue for items currently being crawled.
  • Link database :The Link database stores the information that is extracted by the content processing component and the click through information.

Thanks for reading this post. Happy SharePointing!!!