.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Register
 
Win Surprise Gifts!!!
Congratulations!!!


Top 5 Contributors of the Month
Easy Web
Imran Ghani
Post New Web Links

SFTP is converting text file format, now SSIS sees an empty file

Posted By:      Posted Date: April 14, 2011    Points: 0   Category :
 

Using BIDS that came with SQL Server 2008 R2, I created a text file import package.  I used files generated from my local machine for local testing.  However, I find that these files get their format changed after being SFTP'd to the data warehouse.  Not being able to visually see a problem with the text file to be imported, I copied it down to my local machine and compared to one generated at my machine.  Not until I chose "Show All Characters" in Notepad++ did I see the difference: my local file ends its lines with a carriage return-linefeed character, whereas the file that went through SFTP ends its lines with a linefeed character only.

I suspect there's something very basic going on here that I don't know.  I won't ask anybody to explain it, but I'm wondering if there's some way I can import both file formats.  The column numbers and lengths are identical, it's the just line ending character that's different and causing zero rows to get imported.  The only other thing I know to do is recreate my flat file connection, and I haven't even tried that yet.

Thanks,
Eric




View Complete Post


More Related Resource Links

Saving SSIS results to Log or text file using dtexec

  
Hi, How to save the SSIS package results to log file using dtexec command............please help regaridng this...........   Thanks in advance,

How to redirect bad date(xxx,9999) in datereceived column in text file in ssis

  
I want to redirect bad date format from flate file source to log table. I tried with redirect row but it is not working. Thanks in advance

SSIS - load table from source text file with multiple record lengths

  

I have a text file that has mulitple records lengths that I need to load into a table.  The file has multiple record types and each record type detemines the length of the record.  Let me try to explain...the file has a balance line 1, multiple detail records, balance line 2, and a trailer record.

The first record is a balance line 1, which has a record length of 144.

Detail records with a record type of inv or crd, will have a record length of 147.  The remaining record types will have a length of 144.

The second to the last record is balance line 2, which has a a record length of 156.

And the trailer record, which has a record length of 162.

Here is a portion of my source file.  I have made each line bold where you can see the rec type.  The record starting with 9999 is the trailer.

077700001BL100000002010100120101008BALANCE FORWARD 0000000000000000001081175D0000000000{0000000000{0000000000{000001081175D00 0000000000{

077701000INV01953172010100120101008GROCERY - DRY 0000000000000000000073504I0000016678K0000000852G0000000000{000000057679D011N 0000093201E010

07779008099000000002010100820101008KC CASH REC. 0000000000000000001081175M0000000000{0000000000{0000000000{000001081175M00 00000000

Load multiple record length text file into table using SSIS

  

I have a text file that has mulitple records lengths that I need to load into a table.  The file has multiple record types and each record type detemines the length of the record.  Let me try to explain...the file has a balance line 1, multiple detail records, balance line 2, and a trailer record.

The first record is a balance line 1, which has a record length of 144.

Detail records with a record type of inv or crd, will have a record length of 147.  The remaining record types will have a length of 144.

The second to the last record is balance line 2, which has a a record length of 156.

And the trailer record, which has a record length of 162.

Here is a portion of my source file.  I have made each line bold where you can see the rec type.  The record starting with 9999 is the trailer.

077700001BL100000002010100120101008BALANCE FORWARD 0000000000000000001081175D0000000000{0000000000{0000000000{000001081175D00 0000000000{

077701000INV01953172010100120101008GROCERY - DRY 0000000000000000000073504I0000016678K0000000852G0000000000{000000057679D011N 0000093201E010

07779008099000000002010100820101008KC CASH REC. 0000000000000000001081175M0000000000{0000000000{0000000000{000001081175M00 00000000

Prevent SSIS from creating an empty flat file

  

I created a SSIS to export to a flat file (from a SQL command : a stored proc).

I don't wan't my SSIS to create an empty file if there is no data.

 

How can I achieve this ?

 

Thanks,

 

Vince


Converting a portion (numbers) of a text file name into a date type

  

I am working with a number of Flat Files that will be processed daily. The FileName is a combination of text and numbers. The numbers are the date of transaction. Therefore, I need to pick up the numbers and use it as a Date (DateAdded) inserted into the database. However, the positioning of the numbers is not fixed.  

Examples of what the File Names:   Dial20100610.txt and Dialog20100611.txt

Now, I have inserted all the data into a staging table called Raw_Staging with File Name as the name of the column (eg. Dial20100610.txt, Dialog20100612.txt)

Is there a way to strip the text character from the columns with tsql so I am only left with the numbers and then convert it to a Datetime datatype so I can insert it into the destination table?

Upload Text File data to SQL table Using SSIS

  
I have a text File with following format  (  HED - Header Row, CL-Data (Sample 3 Rows) , TRL-Trailer Row)


HED201008240001PCMCL


CL 012352908607309000014090 2010-03-04 16:12:53.077000000 Inserted / Log ICode Action on Order Number / 33949591 / with / I-Code / IDEC/ / Memo: / cci questions about alert/


CL 012352958107309000020378 2010-04-14 10:20:03.340000000 Inserted / Log ICode Action on Order Number / 36992594 / with / I-Code / ISRB,INEW,ILKD/ / Memo: / cci to cxl, saved with benefits and gave her instructions to move account online/


CL 012353013807309000021681 2009-11-24 17:09:05.973000000 Inserted / Log Customer Info Change Action CL 012353013807309000021681 2009-11-24 17:09:05.973000000 Updated / Log Customer Info Change Action


TRL201008240001PCMCL 0000007349



I have to Import The above Mentioned data in to Following Table



CREATE TABLE [dbo].[Log]

[Type] [varchar

SSIS text qualifier results in text file containing "_x003C_none_x003E"

  

There is a bug in SSIS when exporting data from SQL to a comma delimited text file where you specified no text qualifier.

This bug apparently only occurs when you develop the SSIS on a x64 win7 PC and copy the .dtsx file (windows explorer copy/paste) to network path of a x86 SQL server and schedule the job to run from SQL Agent on the same x86 SQL server.

When the SSIS runs, the text file is written out containing text qualifier = "“_x003C_none_x003E".

If you look at " _x003C_none_x003E", it actually means <none>.  x003C = "<" and x003E = ">".

 

If you go into the SSIS package, double-click in the connection manager section to open the flat file connection manager editor and try to clear the text qualifier removing the <none>, the <none> value get added back in.

Problem in Date Format While Exporting To Excel Destination From Flat File Source In SSIS

  

Hi,

I have a flat file(.csv) which contains data(strings incuding date in yyyy-mm-dd format).This i am using in Flat File Source and want to export those data to an Excel Destination.Before exporting to Excel Destination i have created a Derived Column component where i am using an express to get day,month,year part and creating a date sting in dd/mm/yyyy format then conveting it to DT_DATE datatype and then exporting to Excel Component.But while exporting to Excel this date format is getting changed to mm/dd/yyyy format.

The expression i used in derived column is

(DT_DATE)(SUBSTRING(date,9,2) + "/" + SUBSTRING(date,6,2) + "/" + SUBSTRING(date,1,4))

where date is a column from Flat File Source.

Even I changed the Locale to English(United Kingdom) in Flat File Connection Manager for the above .csv file.But still it didnt work.

I searched a lot in google and failed to get any solution on this.

Can anyone help me in solving this issue ?


Inside Windows: An In-Depth Look into the Win32 Portable Executable File Format, Part 2

  

The Win32 Portable Executable File Format (PE) was designed to be a standard executable format for use on all versions of the operating systems on all supported processors. Since its introduction, the PE format has undergone incremental changes, and the introduction of 64-bit Windows has required a few more. Part 1 of this series presented an overview and covered RVAs, the data directory, and the headers. This month in Part 2 the various sections of the executable are explored. The discussion includes the exports section, export forwarding, binding, and delayloading. The debug directory, thread local storage, and the resources sections are also covered.

Matt Pietrek

MSDN Magazine March 2002


Inside Windows: An In-Depth Look into the Win32 Portable Executable File Format

  

A good understanding of the Portable Executable (PE) file format leads to a good understanding of the operating system. If you know what's in your DLLs and EXEs, you'll be a more knowledgeable programmer. This article, the first of a two-part series, looks at the changes to the PE format that have occurred over the last few years, along with an overview of the format itself. After this update, the author discusses how the PE format fits into applications written for .NET, PE file sections, RVAs, the DataDirectory, and the importing of functions. An appendix includes lists of the relevant image header structures and their descriptions.

Matt Pietrek

MSDN Magazine February 2002


Writing to a text file

  

Hello,

I have the following  subroutine in an asp.net application class. The subroutine writes data to a text file upon application login.   My question is that if this function was called at the same time by two different  users would it cause any kind of error. Is there a need for a try catch?

 Public Shared Sub writeToLogFile(ByVal UserName  As String)
        Dim strLogMessage As String = String.Empty
        Dim strLogFile As String = System.Web.HttpContext.Current.Server.MapPath("~/Log.txt") Dim swLog As StreamWriter
        strLogMessage = DateTime.Now.ToShortDateString().ToString()  " ==> " & UserName

        If Not File.Exists(strLogFile) Then
            swLog = New StreamWriter(strLogFile)
        Else
            swLog = File.AppendText(strLogFile)
        End If


        swLog.WriteLine(strLogMessage)
      &nbs

The file reached the maximum download limit. Check that the full text of the document can be meaning

  

Hi,

I'm facing an issue with the indexing.

I have 1 WFE+Index server+DB server.

Index server is not installed with MS FIlter pack 1.0

When crawling, the there will be document with warning in crawl log:
The file reached the maximum download limit. Check that the full text of the document can be meaningfully crawled.

Documents that with warning are such as doc, ppt, xls, docs, ppts and many others.
However, I view into the successful crawled document, there are doucments with ext doc and ppt.

For large file index, there are MaxGrowthFactor + MaxDownLoadSize required to be added into the index server.


As my understanding is, MS Filter Pack should installed into index server(already did, correct me if i'm wrong).

I looked into the Office SharePoint Seach(CA>Services in farm), if the server is appointed to "Use this server as indexing server", then MS Filter Pack is suppose to be installed into that particular server as well.

At the bottom, there also has another option is "Use all web front end for crawling".

The question here is, IF the option "Use all web front end for crawling" is selected.
Does the WED FRONT END Server required to installed the Ms Filte

Update Profile Database from text file using BDC

  

Hi,

I have a requirement to import the profile properties from a text file in sFTP to update the MOSS Profile Database. The objective is to create a single repository for the Staff Directory in MOSS.

Can I use BDC to connect and update the Profile database?

What is the alternate approach?

Regards
Sanra


Deleting Last Line of Text in a text file

  
Hi, I needed some help on the best way to delete the last line of  text in a text file. I have a subroutine that looks like this, but it is missing some code. Could anyone advise me on this? Dim strLogFile As String = System.Web.HttpContext.Current.Server.MapPath("~/Log.txt")         Dim fs As New FileStream(strLogFile, FileMode.Open, FileAccess.ReadWrite, FileShare.None)         Dim swLog As New StreamReader(fs)         Dim array As String()         Dim value As String = swLog.ReadToEnd ''Array that holds the lines of the text file.         array = value.Split(vbCrLf)        ''code for deleting last line: . . .  fs.Close()

Data loading from text file

  
Hi, I am loading the data from a flat file into a sql table. The values in one of the column have spaces at the end of the value. I want to remove those spaces and load into the table. I am thinking of using derived column and use rtrim on that column, how can I use that as an expression. Thanks.sqldev

Consistently running out of page file memory with full text indexer

  
Using MS SQL Server 2008 SP1 x64 Standard Edtion on Windows 2008 R2 Enterprise, I'm about to full-text index for the first time 1 table and 2 views. The table contains about 250'000 entries with a data space of 180 MB. As soon as I activate the full text indexing, the fdhost.exe task starts to consume slowly but surely all the available page file space (this can be easily watched using the Resource Monitor and the Commit Charge graph on the memory tab). Once all the virtual memory has been consumed, the server becomes unusable since it can't open any new windows any more, and RDP stops working. The machine specs are as follows: 12 GB of RAM 80 GB free on hard disk out of 136 GB 8 CPUs Custom size paging file with sizes between 24 GB - 60 GB (originally, this was system managed size, but then the server ran out of memory sooner) Max SQL server memory set to 6 GB (first 10 GB, then 8 GB) I've set the max fulltext crawl range to 8. During the indexing, the 8 CPUs are bit busy for a while, but not excessively. What is astonishing is that there is almost no use of physical memory during the indexing (I can see an increase from 2 GB to 3 GB which still leaves plenty of RAM available). Does anybody have an idea how I can convince fdhost.exe to consume physical memory and leave the paging memory alone? Or what else can I try?
Categories: 
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend