.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Register
 
Win Surprise Gifts!!!
Congratulations!!!


Top 5 Contributors of the Month
Kaviya Balasubramanian
Sgraph Infotech
Imran Ghani
Post New Web Links

Using Serialization/Deserialization versus flat file for Interprocess Communication?

Posted By:      Posted Date: October 19, 2010    Points: 0   Category :.NET Framework
 

I have a process A that gathers data and hands it over for processing to process B. The amount of data is anywhere near 650MB or more, the transport should be as fast as possible, yet painless (if possible). A and B are different processes, so the scenario could be considered Inter Process Communication.

Here is what I've done and found so far:

  1. Process A created an object filled with all information and serialized it into a file. B deserialized and processed the data.
    - the computational time on the serialization is OK.
    - the deserialization is many times bigger than serialization to the point that I consider it useless because it takes too long.
  2. I do not serialize/deserialize instead use a flat file for the communication.
    - it's quick and same speed for read and write
    - lacks flexibility and is not very elegant either.
  3. I used namedPipes to transfer the serialized object between processes A and B, which is elegant but ever slower because of the issue in #1 and the fairly poor performance of namedPipes and big data?

 

Questions:

  1. Can somebody please advise me on how to best deal with my requirements?
  2. Explain to me why deserialization is soooo slow (I understand that depending on type a lot of object have to be created) and what better approach one should take.


    View Complete Post


More Related Resource Links

WCF Tutorial - Basic Interprocess Communication

  
What the heck is WCF? For a long time, I couldn't tell you exactly what it was. And honestly, I still don't think I can. WCF is such a broad concept that it's hard to boil down into a concise explanation. When I finally started reading about it, and getting beyond the concepts, things started to clear up. That's what today's tutorial is about - cutting through all the concepts and just use WCF to build a basic server and a basic client.

Service Station: Serialization in Windows Communication Foundation

  

Windows Communication Foundation supports several serialization mechanisms and provides a simple, interoperable foundation for future service-oriented applications. Here Aaron Skonnard explains it all.

Aaron Skonnard

MSDN Magazine August 2006


Join 2 flat file data flows - retain unmatched rows

  
I have two data flows from two separate flat files. They may contain matching IDs (account number), in this case specific data from each flow should be used to create one row. When there is no match, the rows would stand on their own. At the end of the flow, I need both flows combined into one flow, with one record for each key record (account number). If I were able to use a look-up, I could easily union the no-match data flow back into the match data flow and have the desired result. I cannot use a look-up, since the source is flat files, but this is exactly the functionality I am trying to achieve. Solutions I want to avoid: staging tables, and cache transformations. Any ideas are appreciated.

Issue: Load {NUL} from flat file to "Flat File Source" faild.

  
Hi guys, I've met a strange issue when I was working to use SSIS to load data from flat file to database. It is a story about {NUL}. Sample Flat File: FIELD_SAMPLE|OTHERS ABC{NUL}DEFG|Any Others   I placed an Flat File Source to load these into package, and the configuration of the Connection Manager are as following: FIELD_SAMPLE: Unicode string [DT_WSTR], Length 8   Then I click the "Preview" button and I can see this data in the "Preview" window correctly (with the value "ABC{NUL}DEFG"). But when I tried to run this package, I've got an error that the "ABC{NUL}DEFG" is too long for this field. Then I changed the length to 20 and it was loaded into SSIS package. But I found the value in SSIS package is "ABC", and "{NUL}DEFG" was not there~ I have no ideal why it is ok in Preview but it is not ok when the package running~     Anyway, I hope to load the {NUL} from the flat file into target database. So that is there anybody could give me a hand?

Inserting a row on an extract flat file

  
Hi, I want to add in a header row to an extracted file. The file contains 5 columns of data but I only want to add in a 5chr id (stored in a variable) in the very first line of the extracted flat file.. please can you assit and tell me the best way of doing this..

Export db to flat file

  
I'm trying to export data from a database to a flat file using the tool import and export of SQL Server 2008, I is generating an error that says: Error 1 Validation error. Data Flow Task 1: Data Flow Task 1: The code page on input column "Tienda_Nombre" (201) is 1252 and is required to be 20 127. anaquel.dtsx 0 0 Error 2 Validation error. Data Flow Task 1: Data Flow Task 1: The code page on input column "Proveedor_Nombre" (203) is 1252 and is required to be 20 127. anaquel.dtsx 0 0 Error 3 Validation error. Data Flow Task 1: Data Flow Task 1: The code page on input column "Depto_Nombre" (205) is 1252 and is required to be 20 127. anaquel.dtsx 0 0 Error 4 Validation error. Data Flow Task 1: Data Flow Task 1: The code page on input column "Subdepto_Nombre (207) is 1252 and is required to be 20 127. anaquel.dtsx 0 0 Validation Error 5 error. Data Flow Task 1: Data Flow Task 1: The code page on input column "Clase_Nombre (209) is 1252 and is required to be 20 127. I saved the package and opened it, I saw that in the metadata columns that I got error I get the value of 1252, flat file connection says that the code page is 20127. What should I do to correct these errors?

How do you specify special characters as flat file delimiters?

  
I have a flat file that has fields delimited by tilde {~} and rows delimited by {CR}{LF}.  The problem is, my source is spitting out the {NULL} (&H00) at the end of the row. The data displays correctly in the pre-viewer (with a non-print symbol at the end of the last field), but gives an error when I try to run the package. Example: "data"~"data"~"data"{NULL}{CR}{LF} I can process the row in c# using: "\0\r\n" as the row delimiter.   What is the equivalent notation for SSIS?   Thanks,

Load and parse a file where the file path is located in a column of a flat file source

  
Hi, I have a situation where I have a CSV that contains a bunch of data that populates a bunch of related tables with FK constraints directly.  Thats no problem.  The problem is that one of the columns specifies a filepath to an XML file that is supposed to contain data to be populated in a table with an FK constraint.  How do I specify the file name to the XML source component at runtime? For example TableA has columns: TableAID GroupName TableB has columns TableBID TableAID (FK to Table A) MemberID (the data in question) TableA has a 1 to many relationship with Table B. The CSV has a bunch of columns including: TableAID File path to list of members for each group. I don't know how to solve this problem.  I tried using a ForEach ... but I couldn't get it to work. Thanks in advance for any assistance.

Flat file upload: Error when inserting into datetime field.

  
Hi All I am having an issue with uploading a txt file into an SQL table and one of the fields needs to go into a datetime column. The issue is with column3 I have been into the advanced editor of the flat file source and set the output column to decimal and it fails. The error I am getting is below: Error: 0xC02020A1 at Materials Transfer, Flat File Source [1]: Data conversion failed. The data conversion for column "Column 3" returned status value 2 and status text "The value could not be converted because of a potential loss of data.". Error: 0xC0209029 at Materials Transfer, Flat File Source [1]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "output column "Column 3" (48)" failed because error code 0xC0209084 occurred, and the error row disposition on "output column "Column 3" (48)" specifies failure on error. An error occurred on the specified object of the specified component.  There may be error messages posted before this with more information about the failure. Error: 0xC0202092 at Materials Transfer, Flat File Source [1]: An error occurred while processing file "D:\Development\r3_downloads\mat.txt" on data row 1. Any help would be great.   Thanks BigGopher

Adaptive Flat File Destination Columns

  
I'm sure it must possible to adjust the columns for a flat file output based on the input, but so far have not found the right tools. I have to convert quite a few tables to flat files, where the input columns have not been safely specified yet. The tables are still being changed.   e.g. I have a table with columns A,B,C. Columns D,E,F... could be added later, maybe B or C will even be removed or renamed. I have to transfer all data from the table except for column A to a flat file.   I find it annoying having to adjust the columns manually every time. Would be extreemly thankful for your advice.

Import a flat file with combined data into separate SQL tables using SSIS

  
I have a flat text file (comma delimited) that is essentially multiple files, each with its own format, combined into one file. The file is coming from an external software vendor so unfortunately we don't have much choice but to work with what we are receiving. Here is an example of what the file could look like: Customer Data CustID,FName,LName,PhNum,Email 12345,John,Smith,,jsmith@gmail.com 12346,Jane,Doe,8001111111,jdoe@hotmail.com Customer Plan CustID,PlanType,PlanName,PlanStart 12345,0,Plan1,01/01/2010 12345,2,PlanVis,01/01/2010 12346,3,PlanLf,04/01/2010 12346,0,Plan1,01/01/2010 Customer Payment CustID,LastPayment,Amount 12345,09/01/2010,100.00 12346,05/01/2010,50.00 There is an empty line between each 'section' of data. I adapted a VB script I found online that can take the incoming file and save off each section as its own file so that each one can be separately imported, but this seems inefficient. I'm really new to SSIS in general, but it seems like it shouldn't be that difficult to take the data, split it where there is an empty line, and then import each section into the appropriate SQL table. Any ideas would be most welcome. Thanks!  

\ in flat file forcing truncation

  
I am using a Flat File Source to import a csv file. The connection manager uses Delimted, Text Qualifier None. I am importing a row at a time then breaking it up. It is not importing a row because of the "/" char. Any way to fix? Thx in advance.   "September 11, 2010","article","51811 - government\ not loaded",1 "September 11, 2010","article","51811 - government gets loaded fine",1  

Do not create a flat file if no rows are read

  
Hi, I have a number of flows in a package that are reading to check for changes in data and then outputing those changes to flat files.  If there are no changes, that is, I read no rows in my oledb source, an empty flat file is created.  Consequently, the file is sent and the destination is empty for the next run time. This is not really the worst thing in the world, but I would rather not FTP off empty flat files, if only for the sake of efficiency.  Is there a way to prevent the creation of the flat file if no rows are going to be written? Mark

DataRows Coming cotinuoulsy in flat file

  
Hi, Iam using flat file destination using a flat file connection manager to write data into a file with .FILE extension. The source data is of a single column of width 40 characters. When setting up connection manager, I used Fixed Width mode and also used CR-LF as Header row terminator. The source is from a Stored procedure witha table select which returns data with 40 characters width. But when I execute the package and open the file using notepad I can see that all the rows are coming one after the other instead of one below the other. I have been trying to solve the issue, but am not able to. Can some one help where I would have gone wrong. Regards, Sreekanth

Data conversion failed --flat file connection manager

  

Hello All,

Can any one help me in this .

[Flat File Source [16]] Error: Data conversion failed. The data conversion for column "acct" returned status value 2 and status text "The value could not be converted because of a potential loss of data.".

[Flat File Source [16]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "output column "acct" (373)" failed because error code 0xC0209084 occurred, and the error row disposition on "output column "acct" (373)" specifies failure on error. An error occurred on the specified object of the specified component.  There may be error messages posted before this with more information about the failure.

[DTS.Pipeline] Error: SSIS Error Code DTS_E_THREADCANCELLED.  Thread "WorkThread1" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown.  There may be error messages posted before this with more information on why the thread was cancelled.

 

can any one help me in this error :

 

[Lookup [465]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "component "Lookup" (465)" failed because error code 0xC020901E occurred, and the error row disposit

Serialization and deserialization over network is very slow

  

I am using the built in C# serialization and deserialization functionality to pass objects over network streams and it works fine. The problem iswhen the client and server programs are running on different computers, sending even the simplest objects over the pipe using serialization is painfully slow (about 50x slower than using byte arrays). If I have the client running on the same box as the server, the serialization approach is nearly as fast as the byte array approach (even though network sockets and serialization are still used). It only slows down when the client and server are running on different machines. Clearly, Serialization is not the cause of the slow down. So why is it so much slower than the byte array approach when the server and client are not on the same computer? Thanks for any help. 

 

I have put together a simple client/server program to demonstrate this oddity:

namespace PingTest {
 class Server {
  public static void Start() {
   System.Runtime.Serialization.Formatters.Binary.BinaryFormatter binFormatter =
     new System.Runtime.Serialization.Formatters.Binary.Binar

Import flat file from the report to filtered data

  

Hi

Is there a way to use a flat file which could be imported from the report in order to filtered the data displayed ?

Thanks

Regards,


Categories: 
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend