.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Win Surprise Gifts!!!

Top 5 Contributors of the Month
david stephan
Gaurav Pal
Post New Web Links

Crawler only indexes 1,000,002

Posted By:      Posted Date: October 04, 2010    Points: 0   Category :SharePoint

I have setup an external data type pointing to a SQL Database with just over 8 million records.  I need share point to be able to search these records so I setup a crawler but it only indexes 1,000,002 it does report 1 error (which happens near the beginning of indexing) but that error won't display when going through the errors screen.  The crawler has indexed data as I do get some search results back.  Is there a limit on the amount of data a crawler will process? 

Do apologize is this is obvious but I am only just learning sharepoint.

View Complete Post

More Related Resource Links

Web Spider/ Crawler


hello everybody,

i'm trying to develop a WebSpider that retreives data related to "Sports" from twitter, Facebook and other sites/blogs, to diplay it all in my page.


i just need to retreive  the information diplayed on that page, but the problem i'm facing is that when i'm reading the twitter page, there is only a javascript code that display these data and not the data itself


is there any possbility to extract this information si can log it in a Database for exemple ?

Thank you for your usual help  




Crawler fails to register date properties of user profiles with the month of January, April, August


This seems to be a bug when the crawler search the user profiles in MOSS 2007.  When crawled, user profiles with a SPS-HireDate in the months of January, April, August and December will be detected, but a full-text (SQL) search returns those profiles without the HireDate field.

User profiles with HireDates in other months work correctly, returning the HireDate in the search.  And changing the month of a problematic user profile also fixes the problem.

This problem is also reflected in the fact that while we have 499 user profiles using the SPS-HireDate property,  the managed property page from the search section only has 350 items with the HireDate property.

We're running MOSS 2007 32bit with SP2 with an English language base and the Spanish language pack. I'd considered date format problems, but I can't imagine how some months would work, while others wouldn't.

Any ideas?

ArrayList indexes not set properly


I have some basic code to fill items in an ArrayList as follows:

Dim intSteps As ArrayList = New ArrayList()
Select Case Session("type")
Case 1
Case 2
Case Else
End Select

Dim j As Integer
	For j = 0 To intSteps.Count - 1
        Response.Write(intSteps.IndexOf(j) & " : " & intSteps.Item(j) & "<br />")

The issue is that for Case 1 the output is:

0 : 0
1 : 1
2 : 2
-1 : 6
-1 : 11
-1 : 12

For Case 2, it's:

0 : 0
1 : 1
2 : 2
-1 : 5
-1 : 6
3 : 11
4 : 12

And for Else, it's:

0 : 0
1 : 1
2 : 2
3 : 3
4 : 4
5 : 5
6 : 6
7 : 7
8 : 8
9 : 9
10 : 10
11 : 11
12 : 12

Why is the index being set as -1 for a few items?

How to remove indexes

How to remove indexes..as it is created by mistake I have read that if it will be deleted then data will be deleted also...PLease clear the case of removal of clustered and non clustered indexes..."SQLSERVER DBA" "INDIA"

Find all compressed indexes?

I've inherited a database on which a contractor was toying around with compressing indexes. As the work she did was never documented I've been asked to remove all compression from the database so that the front end vendor can test a service pack. Is there a simple way to either uncompress all of the compressed indexes within a database or to rebuild all existing indexes within the database without compression? Thanks for your time, XOXO CD.

MOSS Crawler Issues

Today I noticed, the scheduled crawls are not running. I can manually start incremental crawls on the content sources and it's working fine. When the time comes for a scheduled crawl the job never runs, but surprisingly the time for the next scheduled crawl adjusts to the next proper time. I have crawled (incremental) manually on 4:49 PM today and indexed item count was 53287. Till now some more issues are created into my Application and after the time mentioned  automatic incremental crawling is performed 3 times (5:00 PM, 5:30PM and 6:00PM) but unfortunately the indexed item count is same i.e. 53287. Surprisingly next crawling time has updated to next scheduled time i.e. 6:30PM but crawl logs showing last crawling time is 4:49PM.why automatic incremental crawler is not working but adjust the next crawling time time?Please help me to solve this problem.THanks! Arup R(MCTS) SucCeSS DoEs NOT MatTer.

How to tell if 2 or more indexes are interleaved?

Does anyone know if there is a quick and easy way to tell if indexes are interleaved or not?John M. Couch Insight Enterprises Inc. 6820 S. Harl Ave, Tempe, Az 85283 Email: john.couch@insight.com

Search crawler 'breaks' Document ID link

Hi all - we have a Sharepoint 2010 development server up and running (very much Out of the Box configuration). The Document ID functionality has been enabled and appears to be working OK - each document has its own unique ID as predicted. In a team site (Corporate Comms), we have a document library where our corporate logo has been loaded (a small .bmp file). This image file has a unique Document ID. On the top level site, I have created a link to the corporate logo so that it appears on the 'entry' web page. The link to the image is : http://<servername>/_layouts/DocIdRedir.aspx?ID=CH07-7-1  where CH07-7-1 is the Document ID When first created, everything works OK and the image appears on the front page. However, once a search crawl has been run (either full or incremental), the image disappears and is replaced by a red cross (i.e. the URL link has been broken). I have tried this with a word document link (as opposed to a .bmp image) and this still works OK. I have also tried storing the image in a picture library (as opposed to a document library) and the same thing happens. Is there something funny happening specific to the .bmp image/Document ID/crawler ? Anything I need to configure ? Any help would be appreciated. Thanks, Richard.    

not all indexes used in query?

hello, I have this query with several joined tables, and i have created an index on the columns used in the joins. in the query below I join tables hr200, hr300, hr301 and dim_datum. I created the following indexes: In hr200: index on hr200_obj in hr300: index on hr300_obj, hr200_obj in hr301: index on hr301_obj, hr300_obj in dim_datum: index on datum_id, datum But when i run below query, only the index on dim_datum is used. What to do to malke the query use the other indexes too?   SELECT   dbo.hr300.hr300_obj AS ziekmeldingID, dbo .hr300.hr300_date_start AS ZiekStartDatum, CASE   WHEN dbo.hr300.hr300_date_end IS NOT NULL   THEN dbo.hr300.hr300_date_end   ELSE getdate() END   AS ZiekEindDatum, dbo .hr300.hr300_time_start AS ZiekStartTijd, dbo .hr300.hr300_time_end AS ZiekEindTijd, dbo .hr300.hr300_open AS ZiekOpen, dbo .hr300.hr104_obj AS ZiekTypeID, dbo .hr300.hr038_obj AS ZiekRedenID, dbo .hr300.hr200_obj AS hrmID, CAST   (CONVERT(char(8), dbo.hr300.hr300_date_start, 112) AS int) AS ZiekStart_KEY, CAST   (CONVERT(char(8), dbo.hr300.hr300_date_end, 112) AS int) AS ZiekEind_KEY, DATEDIFF   (dd, dbo.hr300.hr300_date_start, dbo.hr300.hr300_date_end) AS Verzuimduur, dbo .hr301.hr301_date, dbo .hr301.hr301_percentage_sick / 100 AS PercentageZiek, dbo .hr301.hr301_percentage_recovered

Is there a way to make SPS 2003 search fields search the MOSS indexes?

Here's my situation: We're doing a gradual migration from SPS to MOSS. Can anyone tell me if there's an easy way to make the SPS search fields search the MOSS indexes? The easiest way I can think of is to great a web part for SPS which will gather the scope and search param and redirect the user to the MOSS search result page. If there's an easier way. I'm all ears.

will indexes auto sort?

  is it true that if we create indexes for our tables, we must still explicitly declare "order by <the index>" when we return a select*from table or can i assume that it is ordered as defined by my index without me having to explicitly state order by?    

Multiple indexes using same column

I am writing an extensive query and have multiple non-clustered indexs on a table.  I have one index for a single column [Column1] and another index for [Column2], [Column1].  I am using different querys where I sometimes just use [Column1] and other times use many columns including both [Column1], and [Column2]. My question is: If you have multiple indexes that contain the same column will they ever cause a conflict?

Strategy In creating indexes

Hi Team, Need a help strategy in creating indexes.. I have a Query which returns me more then 4 billion rows, which i am trying to insert into a table. There are views and table in the query. I am working on SQL Server 2000. So i thought if we create indexes on primary key columns on the table(which are used in views), on the columns used where condition of the main query. Will creating clustered index on the table which more than one primary key helpful? Is my strategy correct? any suggestions will be of great help! Regards, Eshwar.


Hi i am storing some web stats into a table, i have noticed that Request.Browser.Crawler does not catch all crawlers, so i am getting incorrect visitors. what is the best way to make sure that only valid web browsers are stored?

No Indexes - Linking to MS Access

Hello, I am using MS Access 2000 as a front end to a SQL Server 2000 db.  When I link to the SQL Server db using a DSN, the SQL Server table pk's and indexes are not coming over with the link.  The pk's and indexes do exist in SQL Server.  I haven't had to link to these tables in over a year, but they used to link just fine. Do you have a suggestion as to where to look at this problem? Thanks you - Darryle

Problem with cursor and Rebuild indexes

(SQL Server 2008 SP1) Hi I have a stored procedure that runs frequentely (uses a cursor), that SP never have errors on execution, but when runs at the same time with a rebuild index task (Database maintenance task on sundays), the stored procedure show the following error: "Could not complete cursor operation because the table schema changed after the cursor was declared" My question: Is there any way to avoid this problem running the two task at the same time? Is this a bug of the SQL Server 2008?JAC

SharePoint 2010 Search Crawler excluding items

Hi All I'm trying to setup the Search functionality in SharePoint Server 2010, but I'm having trouble getting it working. I have set up a crawler for the site with a user that has Full Read permissions to everything. I also set up a crawl rule for the site including all items in the path with the "Follow links on the URL without crawl itself" option checked. The crawler runs across the entire Site Collection and sub-sites, but I keep seeing warnings that content was excluded and the search can return no results. The error message I keep seeing is "The content for this address was excluded by the crawler because this item was marked with a no-index meta-tag. To index this item, remove the meta-tag and recrawl." I have looked for a no-index meta-tag, and have been unable to find it anywhere. I have also done searching on this issue, and set site settings to Always index all WebParts and to Appear in Search Results. Neither of those settings appear to have helped.   Does anyone know what I am doing wrong, or know of good documentation that could point me in the right direction? Thanks
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend