.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Win Surprise Gifts!!!

Top 5 Contributors of the Month
Gaurav Pal
Post New Web Links

How does SharePoint Incremental crawl works

Posted By:      Posted Date: October 28, 2010    Points: 0   Category :SharePoint

Hi All,

I am investigating a lot on how the incremental crawl works in SharePoint.

I know that it uses Change Log.

But when it comes to Deletes, how does it work

As i have observed, Chnage Log has only Top level delete event ie., If a List containing List Items has been deleted,

Change Log has only SPChangeList event.

How does SharePoint incremental crawl deletes all its children as there will be only dead reference to the List.

Can someone please help me in understanding on how does this work?





View Complete Post

More Related Resource Links

Why SharePoint incremental crawl doesn't use Audit Logs


Hi All,

I am just wondering why SharePoint (2007) search incremental crawl uses change log. Why is it not using Audit logs if they give more information.

Can someone help me understand this.




SharePoint crawl errors on files which are not present

All, I'm noticing 2 errors in my crawl logs. Neither of the files exist anywhere on our site. The URLs are http://.../forms/repair.aspx and http://.../forms/combine.aspx and the error message is 'Error in the Microsoft Windows SharePoint Services Protocol Handler'. Our crawl normally takes about 3 and a half hours. Recently, it's been taking 5-6 hours. These 2 errors are logged at the end of the crawl. While the crawl is running, I see the success count growing and at about 3 and a half hours into the process, the success count stops growing. I'm not sure what the crawl is doing for the next 2 or so hours, but if finally logs the 2 errors mentioned earlier at the end of the crawl, then completes. I have tried resetting the crawled content and changing the index location of the SSP, but neither have worked.  I have also tried excluding the path to these two files with crawl rules, but that hasn't worked. I am on SharePoint 2007 SP2. Any ideas? Thanks

Cannot crawl sharepoint site and mysite after database attach upgrade form sharepoint 2007 to 2010.

After database attach upgrade site and mysite from sharepoint 2007 to 2010 , I have full crawl and get "The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly. If the repository was temporarily unavailable, an incremental crawl will fix this error. ( Error from SharePoint site: HttpStatusCode ServiceUnavailable The request failed with HTTP status 503: Service Unavailable. )" for mysite and get "Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled." for sharepoint site. The content access account for search is "db_owner" of both of site and mysite. How do I solved this problem ?

Crawl Error in Sharepoint 2010 RTM

Hi ,       I am getting the following error on doing a crawl (both Full & Incremental). The SharePoint item being crawled returned an error when requesting data from the web service. ( Error from SharePoint site: *** Index was outside the bounds of the array. ) My Search service account is a Farm administrator and has full control on the entire farm. Thanks,   Vinod.    vinod kumar Bhasyam

SharePoint 2010 AD FS 2.0 Integration - login works then fails, looping back to the adfs server

Hello, I setup SharePoint 2010 Claims auth with AD FS 2.0 following this post: http://blogs.technet.com/b/speschka/archive/2010/07/30/configuring-sharepoint-2010-and-adfs-v2-end-to-end.aspx.  I have no access to the ADFS server, but I think they followed the steps as outlined.  If I do an iisreset (or just recycle the app pool of the SharePoint site) I can login successfully using the remote credentials.  Also I can under security in SharePoint search for and find users and roles from the remote ADFS server. If I close my browser, open a new browser and go to the site I am prompted to login again (this is fine).  I do, using the same credentials, and the login fails because I am redirected between the SP and ADFS servers until the ADFS server stops the redirecting.  I have been doing some digging and I found this post: http://blogs.technet.com/b/speschka/archive/2010/08/09/setting-the-login-token-expiration-correctly-for-sharepoint-2010-saml-claims-users.aspx, which describes the situation I have in point # 3 at the bottom.  I have tried setting the token time out as low as 1 second but this hasn't helped. I have also, just to see, tried setting all the other lifetime settings to 1 second - but no luck.  Because the initial login works I think the setup is correct, but something is maintaining some state that is causing the loop

Trying to build a custom application to crawl SharePoint Search

Hello,      I have been tasked with building a custom application to develop a report dynamically based upon a search ran in a SharePoint site using user-defined keywords. What I need the application to do is this: 1.) Access the results of a search, and do a full fidelity save of each item. 2.)  If the item returned is a document in a library and it has other versions associated with it, access the meta-data of those other versions and group those results with the item found in the search. 3.) Store the results in XML (probably the easiest), and build a custom display for reporting purposes. If anyone has any advice on how I can accomplish this, I would be very grateful, Thanks!

Crawl Component Resilience Requires Sharepoint Server Search Administration Component?

I have a single crawl database and two crawl components pointing to that crawl database. One of the crawl components also has the Sharepoint Server Search Administration component running. When I close down the server running the Sharepoint Server Search Administration component crawling stops. I removed the crawl component from the server running Sharepoint Server Search Administration component and reindexed and this worked. So the additional crawl component works fine. Checked that incremental crawl would run in 5 minutes. Downed the Sharepoint Server Search Administration component server and uploaded some new content. Crawling did not run. Can anyone confirm that this is expected behaviour? I am now trying to transfer the Sharepoint Server Search Administration component to the other server as part of Disaster Recovery preparation and this is failing as per another recently opened problem.  

SharePoint search server 2010 crawl rules

My client wants to create a number of scopes by crawling specific subsites of a CMS 2.0 site.  The CMS site is crawled as a website and security is ignored (e.g. results are not security trimmed). As an example, they want to create a scope called “Audit”.  This scope will use a content source which crawls all content starting at http://server/services/audit and http://server/wssservices/audit. The first is the CMS 2.0 site, the second is a WSS 3.0 site that contains documents for the CMS site. I setup the content source with start address of http://server/services/audit and http://server/wssservices/audit with the crawl settings set to ‘only crawl within the server of each start address’. Additionally, I have created rule with path http://server/services/audit* and set the configuration to “Include all items in this path”, with “Crawl SharePoint content as http pages” also selected.  I have created rule with path http://server/wssservices/audit* with the same configuration settings, except  “Crawl SharePoint content as http pages” is not selected . I have also performed a full crawl after creating the content source and crawl rules. What I would expect to happen is that only results from http://server/services/audit or documents linked from http://server/wssservices/audit would show i

Edit in Datasheet view does not work on one SharePoint 2007 farm, works on another.

The ability to edit in datasheet view on one of our environments no longer works.  I've tried different client machines on different builds and regardless of the install every library, list, etc. when attempting to edit in datasheet view shows as read only.  Our other environments those systems work as intended with those same clients. I've searched all over and haven't found anything that helped.  Any ideas?

Custom masterpage Layout works fine, but WIDE sharepoint lists flow through, over design

Hello all, and thanks in advance for any help. We've built a custom masterpage layout and so far it works great.  We have specified a layout based on tables that fix the width at 960pixels.  The tables are centered on the screen and have a particular background that is different from the page's background to make the content pop.  It works fine in most cases and still displays all MOSSs dynamic contents like the quicklaunch bar at left, etc.  The theme sort of looks like this : http://www.sharepointpackages.com/index.php?main_page=product_info&cPath=65&products_id=268&zenid=a045f5b7dba91fdab94d754569e0b611 The problem is that when a list has too many columns and is therefore wider than the 700 or so pixels that are left over for the placeholders inside the table, the list continues on, but does not affect the table itself.  In other words it appears as if the content was placed OVER the tables in the layout.  This is odd first of all, but it is actually bad in a way, because even if I use styles on the table cells that hold these placeholders, and have the overflow use scrolls, the table ACTUALLY does not see an overflow and the contents continue to run through without activating the scrollbars (they don't even show up).  Can anyone help.  Please, Please, I've been looking at HTML for two days straight now trying to fi

what is sharepoint profile crawler?How it works?

what is  sharepoint profile crawler?How it works?

Reg: Does Custom site definition created in SharePoint 2007 works in Sharepoint 2010



If one has created a custom site definition in sharepoint 2007, Once the whole site is migrated to sharePoint 2010 will that "custom site definition" created in 2007 will work without breaking in SharePoint 2010?

If No, what has to be done to make it work on 2010.






SharePoint 2010 search crawl error: The SharePoint item being crawled returned an error when request

SharePoint 2010 search crawl error:
The SharePoint item being crawled returned an error when requesting data from the web service. ( Error from SharePoint site: Specified argument was out of the range of valid values. )

This error is for both document and folder. The document and folder were created/uploaded via SharePoint User Interface only. Later these doc/folder were deleted. And we found the search result showing the documents that were deleted from the doc library resulting in '404 page not found error' upon clicking. Ofcourse we did an incremental crawl after  deleting the file/folder. The error is same even after full crawl.

Environment details:
1 server - WFE role
1 server - App server role
1 server - index server role

Another information if helpful: We used some third party tool to migrate few sample data in the same server but on different site collection, different content database but on the same web application. The tool migrated the data and set unique permissions too for the migrated files. We tried deleting them via SharePoint interface and that files too gave a similar error!

Does Sharepoint new Client Object Model works with SharePoint 2007


I am able to test this new Client OM successfully with SharePoint 2010.


Would it support SharePoint 2007 & SharePoint 2003?


How do you retrieve data from older SharePoint Server ?

Ronak Patel

An incremental crawl on a file share always has about 9000 successes although nothing has changed


Hi everybody,


We have several file shares (based on Windows servers) that we want to crawl with our SharePoint 2010 system. It is all set up and running, not too fast but that might be another topic. I noticed that every incremental crawl on a file share with about 50 GB of data leads to 9000 successes (according to the crawl log). I did some investigation and saw that there are about 9000 folders on this file share.

It looks like the crawler always recrawls all folders! Is this how it should be?

I think that if there aren't any changes no recrawl should be necessary. Moreover this could speed up the incremental crawl.


Any ideas?


Thanks for your help!





BCS Incremental Crawl Configuration

I tried unsucessfully to implement incremental crawling on my external list data.  The external list is created from an external content type using a BDC model.  The BDC model is created using a .NET conenctivity assembly which makes web service calls to retrieve the data.  The LastModifiedTimeStampField property is configured as the name of a DateTime field (DateModifiedTimestamp) on my entity object.

I am able to view all of the data in the external list.  However, when I try to index the external list in SharePoint Search Server 2010 through a full crawl, it fails after a couple hundred (of 12K+ records) sucessful record indexes with the error "Error while crawling LOB contents. (Not a valid Win32 FileTime).

The SharePoint log contains this: FLTRDMN: Errorinfo is "Not a valid Win32 FileTime."    [fltrsink.cxx:553]  d:\office\source\search\native\mssdmn\fltrsink.cxx

The obvious reason for this error might be that the DateTime field configured with the last modified date data does not contain a valid datetime value.  However, I have confirmed that I can parse all of the dates returned from the web service (I created a console applicstion to pull the records and parse the modified date).

SharePoint 2010 - Crawl Fails - Access is denied


Please help!!  I have recreated the Search Service Application and when I try to do any type of crawl - I get the following error.  I can not figure out where the problem is.  The event logs indicate the same error as well.  I think it should be pretty straight forward.  I have already disabled the loopBackCheck in the registry.

Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled.

Please help!  Thank you.


ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend