.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Win Surprise Gifts!!!

Top 5 Contributors of the Month
Gaurav Pal
Post New Web Links

SSIS 2008 - Fuzzy lookup warming index takes too much time

Posted By:      Posted Date: October 08, 2010    Points: 0   Category :Sql Server

Hello All,

We are using Fuzzy lookup transformation in our SSIS package.

We have selected option to store and maintain index for first time. And for subsequent executions, we use option to use existing index. So it will not create new index each time. We are doing fuzzy lookup on single source table with single column. The source table contains more than 2.5 million rows.

So while executing the package, although its not taking time to build complete index. But still it takes considerable time and uses large memory whenever we execute it.

We have also set the WarmCache option to true. Although all data is loaded into memory and removed after execution.

Is there any other way so we can keep the data in memory, so for subsequent execution, it gets result directly from memory?

Please let me know if you have any suggestion? Its urgent.


Thanks in advance.

Darshan Vyas

View Complete Post

More Related Resource Links

SSIS - OLE DB from VIEW takes long time



I have a question regarding SSIS 2008 and how it handles views. I hope the answer for this question will be a quick "Get a hotfix" heh.


I am experiencing a very odd behavior with setting up a View as a data-source in OLE DB. I have set up a a very simple package (for experimenting) with a Data Flow Task that has only two components, Data-Source and Data-Destination. I have set the data-source to a view and the destination to a table in the same database.

Note: I am developing the package on the server itself.


  • While trying to preview data-source I get an error, "Query Timeout expired (Microsoft SQL Server Native Client 10.0)
  • When executing the package Validation, Pre-Execute phases are complete but no rows are passing through the buffer. I have waited over 12 hours before stop the job. Checking the monitoring verified that IO was on 1. And that no object was locked , which may have caused the package to run so long or not to retrieve data.
  • When connecting to SSMS and doing export of data from the view - it also retrieves no data and the process seems to be stuck.


Additional data:

  • T

Using Conditional Split data Transfer in SSIS 2008

This article uses the Integration Services Conditional Split Data Transformation element to filter and transfer data from a set of flat text files to SQL Server database table. The concept can be easily extended to apply to any other source or destination such as Microsoft Excel. This scenario is useful in creating denormalized database tables in a reporting and analysis situation.

Issues with SSIS and SQL 2008 - Integration services not running

Hey guys, I installed SQL 2008 with complete BI studio including Integration Services. When i check in my services - i cant see my integraiton services in there When i try to connect to Integration Service using localhost, I cant connect it And when i try to run setup of SQL 2008 - It shows that Integration Services already installed and that what i checked when i fetched my Installation tool report from SQL Server to verify i have it installed. Why can i access Integration services, if i already have it, Please suggest with ideas, checks i can make to ensure every setting is in place. Thanks. Would appreciate for quick response.  

SSIS Lookup Transformation Issue

I am having a strange issue, In my data flow i have a Lookup Transformaton which will match on key columns of the fact and is followed by a condetional split that would deside if it should insert (old db destination) or go to update (oledb command) or ignore if no change. I have packages running for the last 1 year with the same logic. But in the recent packages I am experiencing a problem.  example: Key columns for join are - type_no (varchar 16) with all numeric values except one record wihh ' '(space) in it  and type_cd (decimal(18,0)) with values(0,1,2,3,4,5) It worked fine when I test the package. After couple of day running in schedule I get integrity violation and huge file with failed records which are supposed to be blocked at the condetional split as they are already in the fact. When i add a data viewer what i found is for all the llokup columns its having nulls (no match found). Workaround that is working for me for now is - I select full cash and say ok in the lookup transofrmation and again open it and set it back to no cash. Then it starts working as expected. Did anyone come accross this kind of issue? is the some standard that I have to follow to make sure this doesnot happen again  

Output parameters in OLEDB Command component - SSIS 2008

I have a package that I developed in SSIS 2005 and recently ported to 2008. Everything runs great except an OLE DB Command component that calls a stored procedure that uses output parameters. When I try to run this, I get this error: Error: 0xC0202009 at Data Flow Task, OLE DB Command [100]: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Syntax error, permission violation, or other nonspecific error". Error: 0xC004701A at Data Flow Task, SSIS.Pipeline: component "OLE DB Command" (100) failed the pre-execute phase and returned error code 0xC0202009. When I do the same thing in SSIS2005, it works fine. So I setup a new package with 1 data flow. In that data flow, I added a source component, an OLE DB Command, and a destination.  I then tested this with a stored procedure that took no parameters. It worked fine. I then tested it with a stored procedure with 1 parameter defined as OUTPUT. This generated the error.   Is there an issue with output parameters in an OLE DB Command in SSIS 2008?   Thanks!http://bobp1339.blogspot.com

WCF Service: takes very long time to start up

Hi, What could be the reasons, why my WCF service takes up to 40 seconds to get started? My service class is set to [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single)] I am running a thread inside this class. The WCF service gets initialized and opened within the context of this thread. The Initialization: private void InitializeWCFService() {      // initialize servcie base addrresses      this.SetupBaseAddresses();      //MetaData behaviour : allow HttpGet      ServiceMetadataBehavior metaDataBehavior = new ServiceMetadataBehavior();      metaDataBehavior.HttpGetEnabled = true;      // initialize service host instance      this.seviceHost = new ServiceHost(this, this.httpBaseAddress, this.tcpBaseAddress);      // add service endpoints and behavior.      this.seviceHost.AddServiceEndpoint(typeof(IUpdateServer), this.GetServerBinding(BindingType.Tcp), EndpointName);      this.seviceHost.AddServiceEndpoint(typeof(IUpdateServer), this.GetServerBinding(BindingType.Http), EndpointName);      this.seviceHost.Description.Behaviors.Add(metaDataBehavior);      // MEX endpoint      this.seviceHost.AddServic

WCF - First call to Callback instance from Server takes considerable time to make it to the client.

Hello. There's a performance problem we are facing that we are unable to fix using DuplexChannels. The first call made by the client to the server reaches the server instantly. The first call made by the server to the first client using a Callback proxy will take several seconds to reach the client. All subsequent calls to the client (as well as other clients) will reach and execute instantly. The client connects to the Server using a proxy to IServiceProvision, supplies an instance to an implementation of IClientCallback and calls IServiceProvision.Start(). The server instantly processes the call and makes an asynchronous call to IClientCallback.DoWork(), which takes several seconds to enter the client's DoWork() implementation. The remainder of the execution then runs smoothly. Also, the server will experience such a slowdown only for the very first call of the first client that connects to it. Is it normal for the Callback interface to take so much time to warmup, only on the first call to the first client? Below is the full source code. Thanks for any information you may provide. Cheers Mike Shared Class Library using System.ServiceModel; namespace Test { [ServiceContract(Namespace = "http://test/IService", CallbackContract = typeof(IClientCallback))] public interface IServiceProvision { [OperationContract] void Start(); [OperationContract] void

Post Upgade task, Upgrading SSIS Pacakges to SQL Server 2008.

Hi, I am trying to upgrade sql 2005 packages to sql 2008 after doing in-place upgrade of DE and SSIS. Can I know what packageformat column in msdb.dbo.sysssispackages refer to, as according to http://msdn.microsoft.com/en-us/library/cc879336.aspx the value should be 2 if the package is in sql 2005 and it should be 3 if it is upgraded. But I am seeing only 0 or 1.   Can I know any other method to figure out version of the ssis packages? I am having issues upgrading SSIS Packages from 2005 to 2008, using SSIS package upgrade wizard.   Thanks for your help. Regards, KRanp.

SQLDataAdapter/SQLDataReader takes lot of time for executing a query

All, I have a webpage with 6 drop down lists. User can start with selecting any drop down list first, and then proceed in any order.  1. Assume when the 1st dropdown is selected first, rest 5 drop downs (except 1st) are filtered/updated based on the selected value in the 1st dropdown. 2. Assume user selects the 4th dropdown second, rest 5 drop downs (except 4th) will be filtered/updated but the condition would be based on both 1st drop down and 4th drop down. But while filtering the first drop down, only the 4th drop down is used as filter (self-filter will be ignored). The code works fine but I see a very bad performance hit in time when the user selects the fourth dropdown after selecting three drop downs. I tried using both SQLDataAdapter and SQLDataReader but neither used to solve the issue. The code takes a lot of time on one line in either of the code (which is highlighted below). When I run the query in SQLServer Mgmt Studio, the query completes in a few seconds.Using SQLDataReader: SqlConnection conn = new SqlConnection(CONNECTION); SqlCommand ObjCmd = new SqlCommand(query, conn); ObjCmd.CommandTimeout = 600; conn.Open(); SqlDataReader daSelTools = ObjCmd.ExecuteReader(); //This line takes a lot of time ddl.Items.Clear(); // Call Read before accessing data. while (daSelTools.Read()) { ddl.Items.Add(new ListItem(String.Format("{0}",daSelTools[0])));

SSIS - Lookup

Hi, I am using Lookup component in my SSIS workflow. I am comparing a string. Is there a way to ignore the case while comparing?

ssis sql 2000 image to 2008 varbinary Failed to retrieve long data for column

Hi, I have a task that to migrate the image type column from sql 2000 to varbinary type in sql 2008. The source column having 3812353 max datalength size for the column. The package always failed with following error message. [OLE DB Source [13177]] Error: Failed to retrieve long data for column "attch_file_content_t". [OLE DB Source [13177]] Error: There was an error with output column "attch_file_content_t" (13209) on output "OLE DB Source Output" (13187). The column status returned was: "DBSTATUS_UNAVAILABLE". [OLE DB Source [13177]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "output column "attch_file_content_t" (13209)" failed because error code 0xC0209071 occurred, and the error row disposition on "output column "attch_file_content_t" (13209)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure. Meng Chew

Diffrences in SSIS 2005 and SSIS 2008?

Hi All,   How to convince interviewer in  major differences in ssis 2005 and ssis 2008.In my system I have ssis 2008.Can I install ssis 2005? How to install? Thanks CMK…

SQL Server 2008 bug with outer join and index

Hello! We found a problem when migrating from SQL Server 2005 to SQL Server 2008. SQL-statements like   SELECT <table1>.<fields>, <table2>.<fields>, <table3>.<fields> FROM <table1> JOIN <table2> ON <table2>.<field> = <table1>.<field> LEFT JOIN <table3> ON <table3>.<field> = <table1>.<field> WHERE (<table2>.<field> = 2 AND <table2>.<another_field> = 'some_text' OR <table3>.<field> = 2 AND <table3>.<another_field> = 'some_text')   give different results on 2005 / 2008. On 2008 I get lots of exessive rows which have NULLs instead of the values required in the WHERE-term. Changing the Compability Level does nothing. It also doesn't matter from where the query is executed (direct via ODBC, in a view or from the Enterprise Manager). 2005/2008 also use different execution plans. (Btw: how can I make an screenshot of a EP larger than the screen?)   After some hours I drilled it down to the existence of a index! By deleting or deactivating the index 2008 gives the correct results like the 2005 did, after activating the index I get these exessive NULL-rows again!   Anyone else ran into this problem? Any tips (dropping all indexes in a 25+ GB DB is NOT an option :-) ?   Thanks in advance! Holger WellenkötterMCTS SQL 2005

ssis 2008 Loading MULTIPLE EXCEL file

HI it is possible for me to load multiple text file using for each loop,but if i try the same process for EXCEL file then my excel source throws an error. Also when i created my variable in value  i have given the path of my excel file ,still it didnt help me..because the moment i map the variable with the connection string of source excel connnection(property->expression->connection string) it start giving error if you could help me or provide some reference it will be of great help http://bi-polar23.blogspot.com/2007/08/loading-multiple-excel-files-with-ssis.html i tried this link but got stuck with it on step 12 and 14 (below  is my connection string) and variable is @Filename which i have created Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Users\mushtaq.sheikh\Desktop\ForEach\New Microsoft Excel Worksheet1.xlsx;Extended Properties="Excel 12.0;HDR=YES"; The moment i try to edit my connection string by comparing connection string provided in link it gives error(an OLEBD error has occured with error code 0x80040E73) on excel source Thanks in advance

How to create a SSIS package to import records form SQl Server 2008 tables to text files

I am a newbie to SSIS and would like to create a package to accomplish the task referenced in the above title. Will appreciate any links and pointers in the right direction.   Thanksakoranteng

SSIS Lookup slow

Ik have a SSIS package which does a lookup for a WorkID in a employee_work table. The lookup is based on date and employeeID. It then inserts the correct workID in an sick leave fact-table. The lookup table has "only" 20,000 rows, it's indexed. The fact table is about 4,000,000 rows. This is the lookup  query: select TOP(1) * from    (SELECT WerkID, werk_start_KEY, ms120_obj    FROM dbo.DimMedewerker) [refTable] WHERE [refTable].[ms120_obj] = ? and [refTable].[werk_start_KEY] <= ? ORDER BY  [refTable].[werk_start_KEY] DESC But whe I run the package it only processes about 35000 rows, the it takes a time, then it processes the next 35000 rows, etcetc. How can I make it go Faster? At the moment it takes about 1.5 hours

SSMS 2008 & Activity Monitor's Processor Time graph is disabled

Hi,I just upgraded one of my SQL 2005 clusters to SQL 2008 with CU9.  Now, when I connect to it with SSMS, either from my workstation or directly on a cluster node, the % Processor Time graph in the Activity Monitor screen is disabled.  The other three graphs are fine.  If I connect to a different SQL 2008 server, the % Processor Graph is fine.  I don't see any errors in any app event log.  Has anyone else seen this behavior?Thanks!
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend