.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Win Surprise Gifts!!!

Top 5 Contributors of the Month
Gaurav Pal
Post New Web Links

Join 2 flat file data flows - retain unmatched rows

Posted By:      Posted Date: August 28, 2010    Points: 0   Category :Sql Server
I have two data flows from two separate flat files. They may contain matching IDs (account number), in this case specific data from each flow should be used to create one row. When there is no match, the rows would stand on their own. At the end of the flow, I need both flows combined into one flow, with one record for each key record (account number). If I were able to use a look-up, I could easily union the no-match data flow back into the match data flow and have the desired result. I cannot use a look-up, since the source is flat files, but this is exactly the functionality I am trying to achieve. Solutions I want to avoid: staging tables, and cache transformations. Any ideas are appreciated.

View Complete Post

More Related Resource Links

SSIS multiple data flows accessing the same raw file

I have multiple data flow tasks accessing the same raw file at virtually the same time.  I would like to know if the data flow tasks lock the raw file during read?  Can multiple data flow tasks read the same raw file at the same time.  I'm currently developing with a very small data set, but I would like to know if I should be concerned with the possibility of the data flow tasks competing for access to the same raw file. Thanks.

Import a flat file with combined data into separate SQL tables using SSIS

I have a flat text file (comma delimited) that is essentially multiple files, each with its own format, combined into one file. The file is coming from an external software vendor so unfortunately we don't have much choice but to work with what we are receiving. Here is an example of what the file could look like: Customer Data CustID,FName,LName,PhNum,Email 12345,John,Smith,,jsmith@gmail.com 12346,Jane,Doe,8001111111,jdoe@hotmail.com Customer Plan CustID,PlanType,PlanName,PlanStart 12345,0,Plan1,01/01/2010 12345,2,PlanVis,01/01/2010 12346,3,PlanLf,04/01/2010 12346,0,Plan1,01/01/2010 Customer Payment CustID,LastPayment,Amount 12345,09/01/2010,100.00 12346,05/01/2010,50.00 There is an empty line between each 'section' of data. I adapted a VB script I found online that can take the incoming file and save off each section as its own file so that each one can be separately imported, but this seems inefficient. I'm really new to SSIS in general, but it seems like it shouldn't be that difficult to take the data, split it where there is an empty line, and then import each section into the appropriate SQL table. Any ideas would be most welcome. Thanks!  

Do not create a flat file if no rows are read

Hi, I have a number of flows in a package that are reading to check for changes in data and then outputing those changes to flat files.  If there are no changes, that is, I read no rows in my oledb source, an empty flat file is created.  Consequently, the file is sent and the destination is empty for the next run time. This is not really the worst thing in the world, but I would rather not FTP off empty flat files, if only for the sake of efficiency.  Is there a way to prevent the creation of the flat file if no rows are going to be written? Mark

Data conversion failed --flat file connection manager


Hello All,

Can any one help me in this .

[Flat File Source [16]] Error: Data conversion failed. The data conversion for column "acct" returned status value 2 and status text "The value could not be converted because of a potential loss of data.".

[Flat File Source [16]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "output column "acct" (373)" failed because error code 0xC0209084 occurred, and the error row disposition on "output column "acct" (373)" specifies failure on error. An error occurred on the specified object of the specified component.  There may be error messages posted before this with more information about the failure.

[DTS.Pipeline] Error: SSIS Error Code DTS_E_THREADCANCELLED.  Thread "WorkThread1" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown.  There may be error messages posted before this with more information on why the thread was cancelled.


can any one help me in this error :


[Lookup [465]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "component "Lookup" (465)" failed because error code 0xC020901E occurred, and the error row disposit

Import flat file from the report to filtered data



Is there a way to use a flat file which could be imported from the report in order to filtered the data displayed ?



SSIS: Error in the package when the data in the flat file source is modified


Hi All,

I have a package which loads data from a flat file source to an OLEDB destination, which is executed successfully and data is loaded perfectly.

But later when the data in the flat file source is modified i.e. if an extra column gets added to the text file, the package is throwing an error because it is unable to detect the extra added column.

How can i avoid this error??? I need my package to execute successfully ignoring the changes(added columns) happened in the flat file source.

Please provide me wth your suggestions and solutions....

Thanks in Advance!!

SQL 2005 Import of Flat File always truncates data longer than 50 chars

My source data is a flat file, semicolon-delimited, and has three columns.  The first is 10 chars, the second can be up to 240, and the third can be up to 240.  For the import I select "Flat File Source" for the Data Source.  When I select my source file, I use the Flat File Manager, Advanced tab, to configure it as follows:

I leave the defaults for Column 0.
For Column 1 I select a DataType of Unicode String and set the OutputColumnWidth to 240.
For Column 2 I select a DataType of Unicode String and set the OutputColumnWidth to 240.

On the Destination page I choose "SQL Native Client", and point it at my database.  The destination table has 3 columns.  Column 0 is CHAR(1), Column 1 is NVARCHAR(240), and Column 2 is NVARCHAR(240).

However, when I get to the "Select Source Tables and Views" page of the wizard and hit the "Edit" key under "Mapping", it shows a size for all three columns of "50".  Sure enough, when I perform the import, it truncates everything to 50 characters.

How do I get more than 50 characters to import?

MS SQL Server: Search All Tables, Columns & Rows For Data or Keyword Query

If you need to search your entire database for specific data, this query will come in handy.

So when a client needs a custom report or some sort of custom development using Great Plains, most of the time I will have to track down the data in the system by running this query and find the table(s) it is in.

Need Oracle Data Provider .CS File for Oracle 10g Database connection !



I need a 'Wrapper.cs' file which takes care of the Database connection ( Oracle 10g) where

i can just call the method with my SQL Query


Gridview1.DataSource = SampleWrapper.ExecuteDatatable("THE SQL QUERY");


Plz Post the link if there is any open source !    

convert SQL data to EDI 835 file


I was assigned to create a app to convert SQL data to EDI 835 file. (Electronic Data Interchange file for Health Care Claim Payment/Advice,  outbound process only).

Can someone help me where to start?

Linq join fields for return data


  how would I join fields together?

           return (from c in storedb.Product_Categories

                         where c.Category_Name.Contains(searchText)
                         orderby c.Category_Name
                         select new {
                                    c.Category_Key && " ;" && c.Category_Name // HOW CAN I DO THIS.....

Create excel file from Binary data on SqlServer



My requirement is that i upload an excel file then i need to validate that data.


1. Uploading file using using upload control saving binary in SqlServer

2. Then i am creating file from binary data  on Sql Server using below command and then using OPENROWSET to dump data to Sql table

3. Then reading Sql Table row wise and validating data.


Alter Procedure spUploadExcelFile
  @PKID int,
  @BatchID int


	Declare @VarBin varbinary(max)
	Declare @FileName varchar(100)
	DECLARE @sql nvarchar(MAX)
                DECLARE @errMsg nvarchar(MAX)
	DECLARE @ObjectReturn INT
	DECLARE @ErrorSource VARCHAR(255)
	DECLARE @ErrorDesc VARCHAR(255)
	DECLARE @ObjectToken INT

	Select @VarBin  = Content , @FileName = [FileName] from MultilangBinaryData Where PKID = @PKID
	SET @FileName = 'C:\'  + @FileName
	EXEC sp_OACreate 'ADODB.Stream' @ObjectToken OUTPUT
	EXEC sp_OASetProperty @ObjectToken, 'Type', 1
	EXEC sp_OAMethod @ObjectToken, 'Open'
	EXEC sp_OAMethod @ObjectToken, 'Write', NULL, @VarBin
	EXEC sp_OAMethod @ObjectToken, 'SaveToFile', NULL, @FileName, 2
	EXEC sp_OAMethod @ObjectToken, 'Close'
	EXEC sp_OADestroy @ObjectToken  
	SET @sql = 'INSERT INTO dbo.UserBulkData SELECT 0,NULL,'+ @BatchI

Using the single *.rpt file with multiple data sources


I've created a set of CrystalReports (*.rpt files) for an ASP.NET web app on a development server. I call each report using the following code:

protected void BTN_RunReport_Click(object sender, ImageClickEventArgs e)
CrystalReportViewer_ClientLetter.Visible = true;

ConnectionInfo con = new ConnectionInfo();
con.ServerName = Constants.ServerIP;
con.DatabaseName = Constants.DatabaseName;
con.UserID = Constants.UserID;
con.Password = Constants.Password;

CrystalReportViewer_ClientLetter.ReportSource = Server.MapPath(Constants.ClientLetters);
ParameterFields parameter = CrystalReportViewer_ClientLetter.ParameterFieldInfo;
ParameterField batchdate = new ParameterField();
batchdate.Name = "@BatchDate";
ParameterDiscreteValue batchdate_value = new ParameterDiscreteValue();
batchdate_value.Value = Convert.ToDateTime(txtBatchDate.Text);

foreach (TableLogOnInfo tlf in CrystalReportViewer_ClientLetter.LogOnInfo)
tlf.ConnectionInfo = con;

Writing the windows forms data to an HTML file

I have a windows forms application ready now i need to transfer data from the windows forms to the web and to do this i need to write an HTML file that contains the form data, i mean the data contained in the form should be written to an HTML file to be submitted and this HTML file should be generated on the local PC. I have the code for the HTML file but how can i write this HTML file with the data values contained in my windows forms. Please give your suggestions. Best Regards, Syed

Insert Excel Data via File Upload into sql server database???

Hi all, I have requirement that User can Upload the Excel Sheet Data to sql server Database at once. How i do that Any Article or hint is apperciated.   Regards 

How to send data from WPF UI to excel file?

How to send data from WPF UI to excel file? Thanks.

Issue: Load {NUL} from flat file to "Flat File Source" faild.

Hi guys, I've met a strange issue when I was working to use SSIS to load data from flat file to database. It is a story about {NUL}. Sample Flat File: FIELD_SAMPLE|OTHERS ABC{NUL}DEFG|Any Others   I placed an Flat File Source to load these into package, and the configuration of the Connection Manager are as following: FIELD_SAMPLE: Unicode string [DT_WSTR], Length 8   Then I click the "Preview" button and I can see this data in the "Preview" window correctly (with the value "ABC{NUL}DEFG"). But when I tried to run this package, I've got an error that the "ABC{NUL}DEFG" is too long for this field. Then I changed the length to 20 and it was loaded into SSIS package. But I found the value in SSIS package is "ABC", and "{NUL}DEFG" was not there~ I have no ideal why it is ok in Preview but it is not ok when the package running~     Anyway, I hope to load the {NUL} from the flat file into target database. So that is there anybody could give me a hand?
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend