.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Win Surprise Gifts!!!

Top 5 Contributors of the Month
david stephan
Gaurav Pal
Post New Web Links

Data transfer between processes for large data

Posted By:      Posted Date: October 12, 2010    Points: 0   Category :.NET Framework


Does anyone know the most efficient data transfer technique between processes for LARGE DATA apart from Shared memory and message queuing techniques.

Thanks in advance

View Complete Post

More Related Resource Links

Data transfer between processes for large data



Does anyone know the most efficient data transfer technique between processes for LARGE DATA apart from Shared memory and message queuing techniques.

Thanks in advance

Using Conditional Split data Transfer in SSIS 2008

This article uses the Integration Services Conditional Split Data Transformation element to filter and transfer data from a set of flat text files to SQL Server database table. The concept can be easily extended to apply to any other source or destination such as Microsoft Excel. This scenario is useful in creating denormalized database tables in a reporting and analysis situation.

Cutting Edge: Pros and Cons of Data Transfer Objects


After a brief refresher on procedural and object based patterns for organizing the business logic layer, the author focuses on data transfer objects and the impact they have on the development of the software project.

Dino Esposito

MSDN Magazine August 2009

Data Points: Using XQuery, New Large DataTypes, and More


SQL Server 2005 introduces a lot of new features, but it also enhances the popular and oft-used Transact-SQL (T-SQL) language. Changes include the introduction of new datatypes to store large values using the MAX indicator, the integration of enhanced XML querying and data modification with XQuery, and the new XML datatype.

John Papa

MSDN Magazine March 2006

Multiprocessor Optimizations: Fine-Tuning Concurrent Access to Large Data Collections


Application performance involves more than just speed. In a Web server environment, top performance also means ensuring that the maximum numbers of users can be served concurrently. This can be accomplished through efficient use of multiprocessor machines and thread management. This article presents techniques that can solve a number of concurrency problems. One approach, using thread management, controls access to a database on a per-thread basis, which protects the integrity of the data. In the article, reusable thread classes are built and presented. The classes are then tested and their performance in a live environment is examined.

Ian Emmons

MSDN Magazine August 2001

Marshalling Your Data: Efficient Data Transfer Techniques Using COM and Windows 2000


The way you choose to transfer data is vitally important in a distributed application. Windows 2000 provides several new features that allow you to transfer data more efficiently. Lightweight handlers allow you to write smart proxies that can cache results and perform buffered reads and writes, minimizing the number of network calls. Windows 2000 also allows you to use pipe interfaces to transfer large amounts of data efficiently through a read-ahead facility. This article illustrates several ways to improve data transfer in Windows 2000 using these new features. It also reports the results of transfer time tests and provides recommendations for transferred buffer sizes.

Richard Grimes

MSDN Magazine September 2000

WCF issues sending large data - "An existing connection was forcibly closed by the remote host"

Hi Guys I have posted the following on www.asp.net but thought it might be productive posting here too.I have been pulling my hair out trying to fix an issue with sending data over WCF, I have read pretty much every THREAD on this forum regarding or similar to this issue without any successful solution.  I have a simple data object which has a [DataMember] with a data type of byte[]. I use this to send binary data from my web app as follows: Asp.net MVC website --> WCF Client (Has my Service References) --> My Services (MyServices.svc) --> Business/Data Tier Everything is being executed from within VS 2008 Pro.   Now I have successfully sent data up to 2.2MB, however it fails if i try and send a file such as 2.3MB or larger i get the following exceptions: General Exception Exception rethrown at [0]:    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)   at SoftApp.WCFClient.MyServiceReference.IMyService.Document_Save(DtoDocument dtoDocument, Int32 usercode)   at SoftApp.WCFClient.MyServiceReference.MyServiceClient.Document_Save(DtoDocument dtoDocument, Int32 usercode) in c:\dev\softapp\softapp.wcfclient\service references\myservicereference\refe

SSRS 2008 Export to PDF fails for large data with System.Exception: Parameter is not valid

Hi, We are working on Range Bar Chart using SSRS 2008. The report has huge data. It is properly displaying the data, but when we tried to export to pdf it is failing with the following exception. Exporting to excel works fine. Also if the report has small/medium data it is property exporting to PDF. It is failing only in case of huge data (we have custom page size of 28" X 14"). Of course the report has lot of expressions we used to customize the colors/text as per our requirements.   Server Error in '/Reports_SQLDEV2008' Application. Parameter is not valid. Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.Exception: Parameter is not valid. Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. Stack Trace:   [Exception: Parameter is not valid.]   [Exception: An error occurred during rendering of the report.]   [Exception: An error occurred during rendering of the report.]    Microsoft.Reporting.WebForms.ServerReport.ServerUrlRequest(Boolean isAbortable, String url, Stream ou

data contract code generation for large/complex schema (HR-XML/OAGIS) - is there an alternative?

Hello, and thank you for reading.I am implementing a Service based on a predefined specification (HR-XML 3.0). As such, I am starting with the schema, and working my way back to code. There are a number of large Schema documents (which import yet more Schema documents) related to my implementation, provided by this specification.I am able to generate code using xsd.exe, by supplying the "main" and "supporting" xsd files as arguments. But there are several issues, and I am wondering if this is the right approach.- there are litterally hundreds of classes - the code file is half a meg in size- duplicate classes (ex. Type, Type1 - which both represent the same type)- there are classes declared as inheriting from a base class, but that base class is not generated/definedI understand that there are limitations to the types of Schema supported by svcutil.exe/xsd.exe when targeting the DataContractSerializer and even XmlSerializer. My question is two-fold:1. Are code generation "issues" fairly common when dealing with larger, modular xsd files? Has anyone had success with generating data contracts from OAGIS or HR-XML schema?2. Given the above issues, are there better approaches to this task, avoiding generating code and working with concrete objects? Does it make better sence to read and compose a SOAP message directly, while sti

Configurable SSIS data transfer package

I have to create SSIS Pakage for importing data form a sql server to  another. And this SSIS data transfer package has to be configurable at run time for: - Database source connection - Database destination connection - Query to call from the source database - Table to insert in destination database And Also need to call this package from a stored procedure...

Data list with page transfer

I built a data list list in the follow. It seems that href is not asp language. How can I transfer it to asp language? <ItemTemplate>  <a href="~/Team.aspx?Id=<%#Eval("TeamId")%>"> <%#Eval("TeamName") %> </a>  </ItemTemplate>

how to transfer data from list column to another column of the same list programmatically?

I have a list and i want to transfer the data from column "Test Display Name" to "Test Display Name Dummy" at renamed it back to "Test Display Name" however i'm getting problem transferring the data. pls help. thanks actually the root problem is my list is not exporting to spreadsheet. i'm getting an error "An operation that uses the database driver could not be completed If the driver is a Microsoft driver, make sure the driver file isn't damaged, and if it is, reinstall the driver by reinstalling Microsoft Query. For other drivers, contact your database administrator or driver vendor." any idea how can i export this Private Sub UpdateTestName() Using oSite As SPSite = New SPSite("http://test_baseline:13838/") Using oWeb As SPWeb = oSite.OpenWeb("SharedArea") Try Dim oList As SPList = oSite.RootWeb.GetList("/SharedArea/Lists/Test Names") oList.Fields.Add("Test Display Name Dummy", SPFieldType.Text, True) oList.Update() For Each oListItem As SPListItem In oList.Items If oListItem.Fields.ContainsField("Test Display Name") Then oListItem("Test Display Name Dummy") = oListItem("Test Display Na

For a Data-driven Subscription, If it Processes 20 but 2 error , how do i just run the ones that err

 I'm trying to figure out if there is a way to just run the errored out ones.  Sometimes I have up to 100 being processed and need to only rerun the ones that error out. Is there a way to do this? and how?

Can not transfer data usisng SSIS

Database Server is SQL2008 with SP1. I created ODBC that connects to one table in the database in the server. It works fine. Then I created a SSIS package that tranfer this table's data into another table using the ODBC. When executing the SSIS , it works fine. No problem transfering data using ODBC via SSIS. Requirement is to use SSIS package in SQL Server job and run it from there. When the SQL job is run , it fails.  Error thrown is Message Executed as user: CORE\svc_prodsql1. Microsoft (R) SQL Server Execute Package Utility  Version 10.0.2531.0 for 64-bit  Copyright (C) Microsoft Corp 1984-2005. All rights reserved.    Started:  12:42:08 PM  Error: 2010-08-26 12:42:08.39     Code: 0xC0016016     Source:       Description: Failed to decrypt protected XML node "DTS:Password" with error 0x8009000B "Key not valid for use in specified state.". You may not be authorized to access this information. This error occurs when there is a cryptographic error. Verify that the correct key is available.  End Error  Error: 2010-08-26 12:42:08.75     Code: 0xC0047062     Source: Data Flow Task ADO NET Source [16]     Description: System.Data.Odbc.OdbcException: ERROR [IM002] [Microsoft]

ASP.NET Events and Pageload not firing when Datagrid containing large number of rows of data

Hi All, I have a datagrid in aspx page.Inside of datagrid i am using around 15 controls such as Button,dropdownlist and text box controls.Once the datagrid binds then the events in the aspx page not firing.Issue occured when the number of rowsgreater than 500.The number of rows less then its works fine.If anybody knows the solution please let me know."Platform i am working on ASP.NET1.1"Regards Hareesh

automatic transfer data from one table to another table in mysql.

Hi all,         i done one web application in this application i need to transfer the data from one table to another automatically when the time is 10.00PM. i need to write this function sepearte time function not in page load function.the user select any of page the data should be transfer from one to another without page load only using time control function.. if anybody know this ,please reply me.Thank You,Arunabathan.G

Hi.How I can to transfer data from object to parameter?

Hi to All. I have class with object wich populated with data from Users Interface  now I need to transfer the data to Store Procedure SQL but i can to transfer the data only using Parameters how I can to get the data from object and to put it in @parameter? Thanks/
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend