.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Win Surprise Gifts!!!

Top 5 Contributors of the Month
david stephan
Gaurav Pal
Post New Web Links

Flat File input - records dropped, reformatted.

Posted By:      Posted Date: May 22, 2011    Points: 0   Category :


I have struggled with flat txt. file input truncation error for some time. Finally appended a dummy column at the end 500 byte long. Truncation error disapeared but now half of my records get bad formatting.

Any advice....?

Thanks, Dmitry


View Complete Post

More Related Resource Links

How capture bad records from "Flat File Source" in SSIS


The flat file is having 10 columns so used 9 delimitor. Few records in the Flat file is having more than 10 columns. I need to redirect in to some error table and rest valid records needs to be loaded in target. Pls help.

How to do wide search and export the records to excel file in ASP.net C# ?


This is my MYSQL  database connection :-

public static OdbcConnection Connection = new OdbcConnection("Driver={MySQL ODBC 5.1 Driver};Server=localhost;Database=jdmcrm; User=root;Password=admin;");

My table name is "customer" and  fields liek customerId , companyName , address and date ;

How to do wide search and export the records to excel file ? im quite new in asp.net .

Pls advise . Thanks





Join 2 flat file data flows - retain unmatched rows

I have two data flows from two separate flat files. They may contain matching IDs (account number), in this case specific data from each flow should be used to create one row. When there is no match, the rows would stand on their own. At the end of the flow, I need both flows combined into one flow, with one record for each key record (account number). If I were able to use a look-up, I could easily union the no-match data flow back into the match data flow and have the desired result. I cannot use a look-up, since the source is flat files, but this is exactly the functionality I am trying to achieve. Solutions I want to avoid: staging tables, and cache transformations. Any ideas are appreciated.

Issue: Load {NUL} from flat file to "Flat File Source" faild.

Hi guys, I've met a strange issue when I was working to use SSIS to load data from flat file to database. It is a story about {NUL}. Sample Flat File: FIELD_SAMPLE|OTHERS ABC{NUL}DEFG|Any Others   I placed an Flat File Source to load these into package, and the configuration of the Connection Manager are as following: FIELD_SAMPLE: Unicode string [DT_WSTR], Length 8   Then I click the "Preview" button and I can see this data in the "Preview" window correctly (with the value "ABC{NUL}DEFG"). But when I tried to run this package, I've got an error that the "ABC{NUL}DEFG" is too long for this field. Then I changed the length to 20 and it was loaded into SSIS package. But I found the value in SSIS package is "ABC", and "{NUL}DEFG" was not there~ I have no ideal why it is ok in Preview but it is not ok when the package running~     Anyway, I hope to load the {NUL} from the flat file into target database. So that is there anybody could give me a hand?

Inserting a row on an extract flat file

Hi, I want to add in a header row to an extracted file. The file contains 5 columns of data but I only want to add in a 5chr id (stored in a variable) in the very first line of the extracted flat file.. please can you assit and tell me the best way of doing this..

Method/Way to know whether a input path of FilesSystem contains any Child file/folder or not

Hi,I am looking for a Method/Way to know whether a input path of FilesSystem contains any Child file/folder or not. To know it currently I am using "Directory.GetFileSystemEntries(Path).Count" and checking if count > 0 then returning True else False. For this first method calculates all the files and folder and then return status to me but I want to exit out as soon as my method gets first element(file/folder) in specified path. Because above method is timing out when my input path contains more than 1 million files. I just need to check whether input folder path contains any child file/folder or not. I have tried to use Win32 API also but on ther server it is crashing so I donot want to use that solution. In .NET any method or way is present to know it. Please suggest me.

Export db to flat file

I'm trying to export data from a database to a flat file using the tool import and export of SQL Server 2008, I is generating an error that says: Error 1 Validation error. Data Flow Task 1: Data Flow Task 1: The code page on input column "Tienda_Nombre" (201) is 1252 and is required to be 20 127. anaquel.dtsx 0 0 Error 2 Validation error. Data Flow Task 1: Data Flow Task 1: The code page on input column "Proveedor_Nombre" (203) is 1252 and is required to be 20 127. anaquel.dtsx 0 0 Error 3 Validation error. Data Flow Task 1: Data Flow Task 1: The code page on input column "Depto_Nombre" (205) is 1252 and is required to be 20 127. anaquel.dtsx 0 0 Error 4 Validation error. Data Flow Task 1: Data Flow Task 1: The code page on input column "Subdepto_Nombre (207) is 1252 and is required to be 20 127. anaquel.dtsx 0 0 Validation Error 5 error. Data Flow Task 1: Data Flow Task 1: The code page on input column "Clase_Nombre (209) is 1252 and is required to be 20 127. I saved the package and opened it, I saw that in the metadata columns that I got error I get the value of 1252, flat file connection says that the code page is 20127. What should I do to correct these errors?

How do you specify special characters as flat file delimiters?

I have a flat file that has fields delimited by tilde {~} and rows delimited by {CR}{LF}.  The problem is, my source is spitting out the {NULL} (&H00) at the end of the row. The data displays correctly in the pre-viewer (with a non-print symbol at the end of the last field), but gives an error when I try to run the package. Example: "data"~"data"~"data"{NULL}{CR}{LF} I can process the row in c# using: "\0\r\n" as the row delimiter.   What is the equivalent notation for SSIS?   Thanks,

Load and parse a file where the file path is located in a column of a flat file source

Hi, I have a situation where I have a CSV that contains a bunch of data that populates a bunch of related tables with FK constraints directly.  Thats no problem.  The problem is that one of the columns specifies a filepath to an XML file that is supposed to contain data to be populated in a table with an FK constraint.  How do I specify the file name to the XML source component at runtime? For example TableA has columns: TableAID GroupName TableB has columns TableBID TableAID (FK to Table A) MemberID (the data in question) TableA has a 1 to many relationship with Table B. The CSV has a bunch of columns including: TableAID File path to list of members for each group. I don't know how to solve this problem.  I tried using a ForEach ... but I couldn't get it to work. Thanks in advance for any assistance.

Upload millions of file records to a table

I have six input files having millions of record. I need to upload these data to my table. How can I upload the details in the most efficient way. Any help on this will be greatly appreciated.

Flat file upload: Error when inserting into datetime field.

Hi All I am having an issue with uploading a txt file into an SQL table and one of the fields needs to go into a datetime column. The issue is with column3 I have been into the advanced editor of the flat file source and set the output column to decimal and it fails. The error I am getting is below: Error: 0xC02020A1 at Materials Transfer, Flat File Source [1]: Data conversion failed. The data conversion for column "Column 3" returned status value 2 and status text "The value could not be converted because of a potential loss of data.". Error: 0xC0209029 at Materials Transfer, Flat File Source [1]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "output column "Column 3" (48)" failed because error code 0xC0209084 occurred, and the error row disposition on "output column "Column 3" (48)" specifies failure on error. An error occurred on the specified object of the specified component.  There may be error messages posted before this with more information about the failure. Error: 0xC0202092 at Materials Transfer, Flat File Source [1]: An error occurred while processing file "D:\Development\r3_downloads\mat.txt" on data row 1. Any help would be great.   Thanks BigGopher

Adaptive Flat File Destination Columns

I'm sure it must possible to adjust the columns for a flat file output based on the input, but so far have not found the right tools. I have to convert quite a few tables to flat files, where the input columns have not been safely specified yet. The tables are still being changed.   e.g. I have a table with columns A,B,C. Columns D,E,F... could be added later, maybe B or C will even be removed or renamed. I have to transfer all data from the table except for column A to a flat file.   I find it annoying having to adjust the columns manually every time. Would be extreemly thankful for your advice.

Import a flat file with combined data into separate SQL tables using SSIS

I have a flat text file (comma delimited) that is essentially multiple files, each with its own format, combined into one file. The file is coming from an external software vendor so unfortunately we don't have much choice but to work with what we are receiving. Here is an example of what the file could look like: Customer Data CustID,FName,LName,PhNum,Email 12345,John,Smith,,jsmith@gmail.com 12346,Jane,Doe,8001111111,jdoe@hotmail.com Customer Plan CustID,PlanType,PlanName,PlanStart 12345,0,Plan1,01/01/2010 12345,2,PlanVis,01/01/2010 12346,3,PlanLf,04/01/2010 12346,0,Plan1,01/01/2010 Customer Payment CustID,LastPayment,Amount 12345,09/01/2010,100.00 12346,05/01/2010,50.00 There is an empty line between each 'section' of data. I adapted a VB script I found online that can take the incoming file and save off each section as its own file so that each one can be separately imported, but this seems inefficient. I'm really new to SSIS in general, but it seems like it shouldn't be that difficult to take the data, split it where there is an empty line, and then import each section into the appropriate SQL table. Any ideas would be most welcome. Thanks!  

Web Services Task Editor: The input Web Services Description Language (WSDL) file is not valid

I am trying to prove I can use SSIS to connect to a web service.  The WS I am trying to connect to was developed by a vendor and covered by a NDA, but I was able to reproduce the issue with a public WS. Here are the steps to reproduce the issue: In the Web Services Connection Manager, I entered http://office.microsoft.com/Research/Providers/MoneyCentral.asmx?wsdl in the URL window.  I am able to successfully "test" the connection I pasted the above link into IE and saved the resulting XML as a .wsdl file on my local machine.  In the Web Services Task Editor, General Tab, I specify the path to the .wsdl file and click on "Download WSDL" button.  No Issues When I click on "Input" and select "MoneyCentralRemote" from the drop-down for Service, I receive an error message saying "This version of the Web Services Description Language (WSDL) is not supported" So the questions are: Did I perform the above steps correctly? What WSDL versions are supported in SSIS? How can I tell what WSDL version was used to create the .wsdl I am trying to access? If the WSDL is an unsupported version, is there a work-around to fix the issue?

\ in flat file forcing truncation

I am using a Flat File Source to import a csv file. The connection manager uses Delimted, Text Qualifier None. I am importing a row at a time then breaking it up. It is not importing a row because of the "/" char. Any way to fix? Thx in advance.   "September 11, 2010","article","51811 - government\ not loaded",1 "September 11, 2010","article","51811 - government gets loaded fine",1  

Do not create a flat file if no rows are read

Hi, I have a number of flows in a package that are reading to check for changes in data and then outputing those changes to flat files.  If there are no changes, that is, I read no rows in my oledb source, an empty flat file is created.  Consequently, the file is sent and the destination is empty for the next run time. This is not really the worst thing in the world, but I would rather not FTP off empty flat files, if only for the sake of efficiency.  Is there a way to prevent the creation of the flat file if no rows are going to be written? Mark

DataRows Coming cotinuoulsy in flat file

Hi, Iam using flat file destination using a flat file connection manager to write data into a file with .FILE extension. The source data is of a single column of width 40 characters. When setting up connection manager, I used Fixed Width mode and also used CR-LF as Header row terminator. The source is from a Stored procedure witha table select which returns data with 40 characters width. But when I execute the package and open the file using notepad I can see that all the rows are coming one after the other instead of one below the other. I have been trying to solve the issue, but am not able to. Can some one help where I would have gone wrong. Regards, Sreekanth
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend