.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Win Surprise Gifts!!!

Top 5 Contributors of the Month
Gaurav Pal
Post New Web Links

How to use Named Pipes for Inter Process Communication and transfer of lots of data?

Posted By:      Posted Date: September 25, 2010    Points: 0   Category :.NET Framework

I guess I have posted my question at the wrong place before:

So hopefully somebody in this forum can advise me on how to make my code work properly. Here's what I did:

I created a Windows service that opens a named pipe to receive data from an external application. Latter one connects and transfers up to 250 MB of data via a named pipe after it serialized a LIST of objects.

My windows service deserializes the data back into a LIST of objects, processes them and outputs the entire list into a file.

My issues are:

1.The data transfer is extremely slow! 200 MB take about 7 minutes via named pipe vs. 20 seconds if a file is used.
QUESTION: is it possible to speed up the data transfer somehow?

2.After my Windows service based processing is done I dispose the named pipe, clear the list and even call GC.Collect() but still my service is not releasing the memory.

3.Also I continue to have very high CPU load (98%) on all cores (2xQuad-core) although the service is not doing anything.
 It would be great if somebody could advise me on some of the p

View Complete Post

More Related Resource Links

Best practice: communication and data transfer between .net to another.



Hi All,

I developed a new GIS and data access library using the c#.net, and we want to change our main application's substructure to new library. (Old GIS application written with the Delphi).

Therefore, I wrote COM interfaces.

There is no problem with communication, but, we have two main problems that related with performance;

P.S.  The application is drawing many geometries per zoom, pan etc. operation, and records are retrieving from the application that written with Delphi.

1. Drawing takes too much time, because each record that belong to geometry is about 100-300Kb and COM data transfer is too poor.

2. Im drawing result map to main application canvas with Graphics.FromHdc(hdc).DrawImage(img, 0, 0); this take about 2-3 seconds.



What do you think about this?

Thanks in advance.

.NET 3.5 Adds Named Pipes Support

you want a named pipe server, all you have to do is create some instances of NamedPipeServerStream to handle each client connection. I stole the following code straight from the MSDN documentation.

Using Conditional Split data Transfer in SSIS 2008

This article uses the Integration Services Conditional Split Data Transformation element to filter and transfer data from a set of flat text files to SQL Server database table. The concept can be easily extended to apply to any other source or destination such as Microsoft Excel. This scenario is useful in creating denormalized database tables in a reporting and analysis situation.

Cutting Edge: Pros and Cons of Data Transfer Objects


After a brief refresher on procedural and object based patterns for organizing the business logic layer, the author focuses on data transfer objects and the impact they have on the development of the software project.

Dino Esposito

MSDN Magazine August 2009

Data Deployment: Streamline Your Database Setup Process with a Custom Installer


Database setup can be a tough and time-consuming process and sometimes fall victim to human error. Microsoft Installer or InstallShield can help, as can your own custom installer. In this article, the author tackles one approach to writing database installers and demonstrates the process with a working code sample.

Alek Davis

MSDN Magazine September 2004

Marshalling Your Data: Efficient Data Transfer Techniques Using COM and Windows 2000


The way you choose to transfer data is vitally important in a distributed application. Windows 2000 provides several new features that allow you to transfer data more efficiently. Lightweight handlers allow you to write smart proxies that can cache results and perform buffered reads and writes, minimizing the number of network calls. Windows 2000 also allows you to use pipe interfaces to transfer large amounts of data efficiently through a read-ahead facility. This article illustrates several ways to improve data transfer in Windows 2000 using these new features. It also reports the results of transfer time tests and provides recommendations for transferred buffer sizes.

Richard Grimes

MSDN Magazine September 2000

How to process data AFTER page is completely loaded for the user?


I have some logging that has to be done, which are some database update or inserts.

However this logging is of no importance to the user, so I only want to process the data/logs after the page is completed for the user.
As of now, if I process the data while the page is loading, I go from 1.4 to 2.0 speed (server side time processing of code), which is quite a bit comparing it is of no use to the user.

So I want this code only executed after that the page is complete.

I've been checking out Ajax, async pages etc, but the problem is it still gets executed before the page is rendered to the client, which means the client will have to wait longer then it is not being processed.

I've been checking out Threadpool.queueBackgroundWorker, however if I get it correct, this will still be executed before the page is shown to the user, as it has to complete before the prerender.

Then I thought about creating a new thread, and do the processing there, which would not have my page waiting to complete the data/logging.
However when I have 100 or 200 users loading pages at the same time, then that would mean I would be creating 200 threads on those loads, which I doubt will be good for performance. So if I want to solve this, I would have to create my own threadpool and only assign for example a max of 40 threads to it which can pr

Named Pipes code

We created a very simplistic wrapper for named pipes. If all you need is a basic messag epipe, this should work for you. Code & info at Named Pipes in .NET thanks - daveVery funny video - Reporting as a Metaphor

Named Pipes provider error - backup script

I have a couple of installs of SQL Server 2005 Express.  One thing it doesn't provide is an agent to schedule jobs so I read where you can use Windows job scheduler to execute a batch job to in turn run a sqlcmd file.  I have that working on one server but on the other server I get the dreaded Name Pipes provider error and it goes on to talk about it may not be setup to allow remote connections.  If I check using Surface Configurator it is set to allow remote connections using both tcpip and named pipes.  To my knowledge all I want it tcpip but I enabled named pipes in an effort to get things working. About the only difference I can see is on the working system the database is a named instance (ie. Servername) versus default instance on the non-working system which shows up as (Servername\SQLEXPRESS).  Hopefully I have my terminology right here. Any tips on how to troubleshoot?  Does it depend on the username I am using?  Domain admin or local server admin or what have you? I should mention I can execute the backup script and it does work.  It is when I run the batch job that it fails. The batch job just says "sqlcmd -i d:\MSSQL_Jobs\backup.sql -o d:\Logs\sqlbackup_output.txt"

IS There is any way out to process only one partition of cube and data of other partition remain ava

HI ALL. Please let me know is there is any way out to process one partition only,while data of other partition remain available in the cube with the help of SSIS

Process Data VS Incremental Process

HI All. what is the difference between Process Data VS Incremental Process.   Regards Amit

Configurable SSIS data transfer package

I have to create SSIS Pakage for importing data form a sql server to  another. And this SSIS data transfer package has to be configurable at run time for: - Database source connection - Database destination connection - Query to call from the source database - Table to insert in destination database And Also need to call this package from a stored procedure...

Data list with page transfer

I built a data list list in the follow. It seems that href is not asp language. How can I transfer it to asp language? <ItemTemplate>  <a href="~/Team.aspx?Id=<%#Eval("TeamId")%>"> <%#Eval("TeamName") %> </a>  </ItemTemplate>

how to transfer data from list column to another column of the same list programmatically?

I have a list and i want to transfer the data from column "Test Display Name" to "Test Display Name Dummy" at renamed it back to "Test Display Name" however i'm getting problem transferring the data. pls help. thanks actually the root problem is my list is not exporting to spreadsheet. i'm getting an error "An operation that uses the database driver could not be completed If the driver is a Microsoft driver, make sure the driver file isn't damaged, and if it is, reinstall the driver by reinstalling Microsoft Query. For other drivers, contact your database administrator or driver vendor." any idea how can i export this Private Sub UpdateTestName() Using oSite As SPSite = New SPSite("http://test_baseline:13838/") Using oWeb As SPWeb = oSite.OpenWeb("SharedArea") Try Dim oList As SPList = oSite.RootWeb.GetList("/SharedArea/Lists/Test Names") oList.Fields.Add("Test Display Name Dummy", SPFieldType.Text, True) oList.Update() For Each oListItem As SPListItem In oList.Items If oListItem.Fields.ContainsField("Test Display Name") Then oListItem("Test Display Name Dummy") = oListItem("Test Display Na

Can not transfer data usisng SSIS

Database Server is SQL2008 with SP1. I created ODBC that connects to one table in the database in the server. It works fine. Then I created a SSIS package that tranfer this table's data into another table using the ODBC. When executing the SSIS , it works fine. No problem transfering data using ODBC via SSIS. Requirement is to use SSIS package in SQL Server job and run it from there. When the SQL job is run , it fails.  Error thrown is Message Executed as user: CORE\svc_prodsql1. Microsoft (R) SQL Server Execute Package Utility  Version 10.0.2531.0 for 64-bit  Copyright (C) Microsoft Corp 1984-2005. All rights reserved.    Started:  12:42:08 PM  Error: 2010-08-26 12:42:08.39     Code: 0xC0016016     Source:       Description: Failed to decrypt protected XML node "DTS:Password" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information. This error occurs when there is a cryptographic error. Verify that the correct key is available.  End Error  Error: 2010-08-26 12:42:08.75     Code: 0xC0047062     Source: Data Flow Task ADO NET Source [16]     Description: System.Data.Odbc.OdbcException: ERROR [IM002] [Microsoft]

Best Practices for WPF Single Instance with named pipes?

Hi, I've developed a WPF server-hub-typed application, and a bunch of client-typed applications. The hub app is always a single instance (using mutex), and implements named pipes.   However, I'm having problem with the clients. The clients are applications that send data to the hub. They need to start the hub if it is not already running.   The problem with these clients is, they are optional and portable. Hence, the user might store them in different locations on their system. I really do not wish to deploy the Hub application with every client. What's the best way for me to implement this?   Another thing, do I *have* to use Process.Start(path, args) to start my hub application?    I've attached a sample project that I've done. But there are several problems with it, which I've not figured out how to fix. http://www.sendspace.com/file/ykhdg0   Thanks Min Chew   [References I've used:] http://blogs.microsoft.co.il/blogs/maxim/archive/2010/02/13/single-instance-application-manager.aspx <-- This is the one that seems the most promising (single instance + IPC), but I want a bare-minimum example so I can get a better hang on things. Besides, it crashes when I click on any of the 3 Run buttons http://www.switchonthecode.com/tutorials/wcf-tutorial-basic-interprocess-communication <-- Named pipe IPC using wcf http://www.switchonthec

automatic transfer data from one table to another table in mysql.

Hi all,         i done one web application in this application i need to transfer the data from one table to another automatically when the time is 10.00PM. i need to write this function sepearte time function not in page load function.the user select any of page the data should be transfer from one to another without page load only using time control function.. if anybody know this ,please reply me.Thank You,Arunabathan.G
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend