.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Win Surprise Gifts!!!

Top 5 Contributors of the Month
Gaurav Pal
Post New Web Links

2008 SSIS and NUMA Memory issues and BLOB Data Types

Posted By:      Posted Date: December 04, 2010    Points: 0   Category :Sql Server

We're running SSIS 2008, moving data from an Oracle 10g database to a SQL 2008 database.  The SSIS is running on the same machine as the SQL Destination server.  The server has 8 gigs and is windows 2003 sp2

The issue we're having is when our package pulls a blob data type from oracle it will just quit with no errors at about 1.4 million records.   We know there are 6 millions records in the dataset. 

A friend of mine said it was a NUMA Memory issue that it is running out at some point and SSIS thinks the incoming data is finished.  Unfortunately, she said there was no answer. 

I was wondering if someone else had a simliar situation moving blobs from Oracle?


View Complete Post

More Related Resource Links

(SSIS 2008) What are the precise numerical sizes (in bytes) of the date data types in SSIS?


I have been reading the famous Integration Services: Performance Tuning Techniques document and I want to use the guidance in the Buffer Sizing section.

In order to optimize my settings for DefaultMaxBufferRows and DefaultMaxBufferSize, I need to calculate the Estimated Row Size for my Data Source.

However, when I look to the Integration Services Data Types document I find that several of the data types do not explicitly list their size in bytes.

(DT_BOOL also isn't listed but the assumption must be it's one bit)

Does anyone know how big (in bytes) these data types are? The Estimated Row Size can't be found without them.


Peter Kral

Using Conditional Split data Transfer in SSIS 2008

This article uses the Integration Services Conditional Split Data Transformation element to filter and transfer data from a set of flat text files to SQL Server database table. The concept can be easily extended to apply to any other source or destination such as Microsoft Excel. This scenario is useful in creating denormalized database tables in a reporting and analysis situation.

Issues with SSIS and SQL 2008 - Integration services not running

Hey guys, I installed SQL 2008 with complete BI studio including Integration Services. When i check in my services - i cant see my integraiton services in there When i try to connect to Integration Service using localhost, I cant connect it And when i try to run setup of SQL 2008 - It shows that Integration Services already installed and that what i checked when i fetched my Installation tool report from SQL Server to verify i have it installed. Why can i access Integration services, if i already have it, Please suggest with ideas, checks i can make to ensure every setting is in place. Thanks. Would appreciate for quick response.  

ssis sql 2000 image to 2008 varbinary Failed to retrieve long data for column

Hi, I have a task that to migrate the image type column from sql 2000 to varbinary type in sql 2008. The source column having 3812353 max datalength size for the column. The package always failed with following error message. [OLE DB Source [13177]] Error: Failed to retrieve long data for column "attch_file_content_t". [OLE DB Source [13177]] Error: There was an error with output column "attch_file_content_t" (13209) on output "OLE DB Source Output" (13187). The column status returned was: "DBSTATUS_UNAVAILABLE". [OLE DB Source [13177]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "output column "attch_file_content_t" (13209)" failed because error code 0xC0209071 occurred, and the error row disposition on "output column "attch_file_content_t" (13209)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure. Meng Chew

SSIS Excel Connection Manager Data Type Conversion Issues with SS Agent Job

Hi All! I have an issue I've been trying to fix but can't seem to figure it out. I was hoping a kind person would point me in the right direction. :o) I have an SSIS package that uses an excel connection manager source, and I want to run this package through a job scheduled in the SQL server agent. The data types for the excel file fields are 2 (DT_WSTR) and 5 (DT_R8). When I run the package directly through the SSIS package (VS solution) all of the data fields are properly imported into the database table. But...when I run this package through the SQL server agent job, ONLY the string (DT_WSTR) fields in each row are being imported, all of the float fields are imported as NULL. I set the data types for these float fields as "float" in the SQL server import table (data type). Even though the excel source float fields are indicating a type of DT_R8 in the excel connection manager and I set the data types in the SQL server table to "float", I also used the data conversion component and set the type to "float" as a fail-safe. I guess I should add to that the data access mode in the excel connection manager is using a custom code to select only those columns that I needed and to trim rows that I didn't need. Here's my code that I have in the excel source editor: select f1, f2, f3, f5, f6, f7, f8 from [mdo$] where f2 <> 'Rep Name'


Hi Folks, I downloaded SQL 2008 and installed recently.Everything perfectly works except one thing that I can't use VARCHAR AND CHAR data types anymore.I was using SQL 2005 and I had the same problem there as well. Please helpe to solve this problem

SSIS User Defined Data Type (Alias Data Types) and OLE DB Command validation

Hello everyone, I've been having an issue with trying to run my SSIS package on a server, and it seems to be failing on the OLE DB Command step.  What we have in our SQL 2005 DB, is a User-Defined Data Type (base type char(7)) and the OLE DB Command is supposed to call a proc that passes in a value of this data type. ie:  CREATE PROCEDURE myProcedure ( @passedInFromSSIS MY_DATATYPE ) AS .... In my SSIS package, I have the type defined as DT_STR with a length of 7.  Now, when I run the package locally (via Visual Studio), the process runs with success.  However, once the package is deployed on a server and run from an application (note: it is run under a different user), the process fails on a validation step with a "Invalid Parameter Number" error. Now, if I change the input parameter in my proc to the base type of the user-defined data type, the process works again. Has anybody run into a similar issue or know what may be causing this issue?  I first suspected perhaps I needed to grant permissions on the user-defined data type (since I was able to run it under my security context, but not under the application's), however noticed that there isn't security tied to the types.  Any other thoughts?  Please let me know if you need further explanation.  Thanks!

SSIS data flow Data Types vs. SQL Server Data Types


How do I join a SSIS text field type cast from the Derived Column Transformation Editor with a varchar from an SQL data table in a lookup?

Also, is there any reference on which SSIS data flow datatypes can be JOINed to SQL Server datatypes?

Here's the problem:  I have records in a flat file.  The flat file name contains the data of the records.  When importing, I read the flat file name into a data field using SSIS.  Then, I use FINDSTRING to create a derived column that contains just the file date as YYYYMMDD.

There's a SQL Table that contains Currency Rates, indexed by currency code and date, in DD/MM/YYYY format, but it's stored as a VARCHAR

My problems:

A.) I tried to convert the string, '07302010' into the following formats, without success:





However, none of these can understand the string '20100730' as a date!


B.) Furthermore, if i convert the field into a DTW_STR, it still doesn't match the varchar(50) in my db.


So, basically, despite using all available typecast date formats, and several string formats as well, I can't find a way to

A.) Parse text out of a field

B.) Convert that text to either a DATE format or a TEXT format

C.) Use that resulting field to

SSIS 2008 data type bug



I am using SSIS 2008, I set a varaible @t1 to data type CHAR and set the value to Y next I used an Execute SQL Task with an ADO.NET connection to SQL 2008 R2 database to insert this the value of @t1 into a test table with only 1 column of data type NCHAR(1):

insert into test1

On the Parameter Mapping section of this task I put the Data Type as String and initially set the Parameter size to -1  

The only problem is it tries to insert a 2 character numeric value into the SQL table. If I set the value of @t1 to N the parameter mapping seems to convert this to 78, when I set the value of @t1 to Y the parameter mapping converts this to 89 it looks like it converts the character to an ASCII code.

Is this meant to be the intended behaviour, as all I wanted to do was insert the actual character into the destination column and not the ASCII code?



Incremental data load from MySQL database to SQL server 2008 using SSIS


I am looking for best solution to perform incremental data load from MySQL database to SQL Server 2008.  I tried to follow instructions from articles on incremental data load in SSIS  using MD5 hashes but I failed to make connection in Lookup transformation editor between MySQL Id (4 byte unsigned integer) and Id in SQl Server 2008 table which is  bigint. i tried to add  data conversion and convert MySQl Id to string. It works fine when you map this field directly to OLEDB destination but in Lookup component it failed. 


Any input greatly appriciated.



Visual Studio 2008 crashes on SSIS Data Flow Task opening


Visual Studio 2008 crashes on SSIS Data Flow Task opening.
This is a random behavior on random SSIS packages on Data Flow Task opening on packages that I've finished developing last week. The packages does not executes with error.

Error Description from Event Viewer:
Faulting application devenv.exe, version 9.0.30729.1, stamp 488f2b50, faulting module msdds.dll, version 9.0.30729.1, stamp 488f2e31, debug? 0, fault address 0x00012b6a.

I'm runnig SSIS development against SQL SERVER 2008 ENT 64.

There is already a fix for this ?

My environment description:
Microsoft Visual Studio 2008
Version 9.0.30729.1 SP
Microsoft .NET Framework
Version 3.5 SP1
Installed Edition: Professional

Microsoft Visual Basic 2008 91605-270-4167253-60984
Microsoft Visual Basic 2008
Microsoft Visual C# 2008 91605-270-4167253-60984
Microsoft Visual C# 2008
Microsoft Visual Studio 2008 Tools for Office 91605-270-4167253-60984
Microsoft Visual Studio 2008 Tools for Office
Microsoft Visual Web Developer 2008 91605-270-4167253-60984
Microsoft Visual Web Developer 2008
Crystal Reports    AAJ60-G0MSA4K-68000CF
Crystal Reports Basic for Visual Studio 2008
KB944899, KB945282, KB946040, KB946308, KB946344, KB946581, KB947171, KB947173, KB947180, KB947180, KB947180, KB94

SSIS - Global Data SOurce - Password issues


I have a simple solution with (right now) 4 packages, 3 configuration files and one "global" datasource.  Here is the View Code on the data source:

<DataSource xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dwd="http://schemas.microsoft.com/DataWarehouse/Designer/1.0" xsi:type="RelationalDataSource" dwd:design-time-name="02d8345e-665e-4bd8-9323-1e846af220e5" xmlns="http://schemas.microsoft.com/analysisservices/2003/engine">
  <ID>PPDM37 Staging</ID>
  <Name>PPDM37 Staging</Name>
  <ConnectionString>Provider=SQLNCLI.1;Data Source=engsql03.dev.ihs.com;Persist Security Info=True;Password=;User ID=;Initial Catalog=PPDM37_Staging</ConnectionString>

As you can see, the <ConnectionStringSecurity> node is set with PasswordRemoved.  When editing the data source and checking the Keep Password option on it still saves wit

SQL Server 2008 R2 on Windows Server 2008 R2 Memory Usage issues and not giving memory back


Okay so i have a physical server i am trying to replace with a virtual server. Its doing a large load process every night, about 190GB on one disk.

The VM is:

On vSphere 4.1 ESXi

O/S is Windows Server 2008 R2 Enterprise

SQL version is SQL Server 2008 R2 Enterprise.

4 vCpus

Now 20GB of memory

Disks assigned to this virtual are all seperate LUNs except for the O/S which is on a shared LUN which is not busy. So 1 LUN/Datastore per 'disk drive on this SQL virtual. Data/Log/TempDB/Backup are on their own LUNs.

All the recommended exclusions are in place for McAfee on a SQL Server.

ESXi 4.1 Enterprise host is not overcommitted. Has 4 x 6core 2.6Ghz AMD processors. BL685c G6 and 128GB of memory. Almost nothing else is running at the time i am testing this load and host is not overcommited or stressed or anywhere near it.

SQLServer process shows 127,236K Working Set and 130,040K Peak working set and memory Private working set 88,828k and Commit Size of 302,792k.

It seems when i check the memory usage in task manager it doesn't show much. In Windows 2003 X64 in would show the GB being used, like 8GB right in task manager. But i don't see that in Windows 2008 R2 task manager. I downloaded SysInternals and it shows the memory on the Windows 2008 r2 box as 15Gb of AWE memory. How can this be? The case is th

Variable types are strict, except for variables of type Object- SSIS 2008


Hi , I declared a user variable MINIODate  as string in package level and assigned this variable in an execute sql task 2008 , where my source is ado.net and sql command is Select CSTR(MIN(IODate) ) as MINIODate  from IOData ( This is source table available in MSAccess DB) , The above senario working fine in 2005. During the run time in 2008  i am getting the following error

[Execute SQL Task] Error: An error occurred while assigning a value to variable "MINIODate": "The type of the value being assigned to variable "User::MINIODate" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.

checking data types in SSIS

hi i have a "gender" column in my database in which o/1 value exists.i want to change 0 to male and 1 to female in SSIS.
2) i also have DOB table in which dates are in dd-mmm-yy format.i want to chechk some of them are erronous like 31-feb-90 ,22-apl-88,21/03/67.
I want to know method how to do remove these erronous data? how to convert it?
how to send dirty data to "error table" ???
need quick response plzzzzz

SSIS 2008 SCD Performance Issues



We are having 20 dimension tables and each table will be having around 20 million records.

These tables would be loaded on a daily frequency with 5 files, each of 3 million records.

We are currently using SCD transformation for TYPE2 load of data.( to maintain history in the dimension table.)

But SCD is taking a long time to insert the data and below are the statistics that I recorded when I executed the package with sample files:

Run1: File1(0.5 million records)  -2 minutes  (Dimension Table is empty)
Run2: File2(0.5 million records)  -13 minutes (Table has  589,000 records)
Run3: File3(0.5 million records)  -26 minutes (Table has 1,140,000 records)
Run4: File4(0.5 million records)  -37 minutes (Table has 1,680,000 records)
Run5: File5(1 million records)   -51 minutes (Table has 2,780,000 records)

Package elapsed time : 2 hr 9 min

1. How do i improve the performance of the SCD? If not, is there any way of loading a table parallely from file so that i can achive performance?

2. In informatica, we have a partitioning feature to load the data parallely which greatly improves performance. Is there any equivalent feature or workaround in SSIS?

Any help would be greatly appreciated.



Data Types - Date and Time in SqlServer

Date and time values can be stored with either the DATETIME or SMALLDATETIME data type. The difference between the two is that SMALLDATETIME supports a smaller range of dates and does not give the same level of precision when accounting for time. The DATETIME data type can hold values from January 1st of 1753 to December 31st of 9999. The time is stored to the 1 three hundredths of a second and each value takes up 8 bytes of storage. The SMALLDATETIME data type can hold values between January 1st 1900 and June 6th of 2079. The time is tracked to the minute and each value takes up 4 bytes of storage. The majority of business applications can live happily with SMALLDATETIME, however, if you are in an environment where each second matters or you need to make estimates to the distant future (or past) then you have to resort to DATETIME. If you fail to specify the time when inserting a value into a DATETIME or SMALLDATETIME column, a default of midnight is used. If you fail to specify the date portion the default of January 1, 1900 is used.
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend