.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Win Surprise Gifts!!!

Top 5 Contributors of the Month
david stephan
Gaurav Pal
Post New Web Links

Multithreaded UoW to Bulk Multithtreaded UoW

Posted By:      Posted Date: October 13, 2010    Points: 0   Category :.NET Framework


I have a multithreaded application (using ThreadPool threads) that is perfoming a number of similar tasks on different threads. At the moment each task is done independently from every other task. I need to somehow pass these tasks from each calling thread to another woker thread that will bulk these tasks together into batches (either defined by the amount of time that has passed since the last request was received or by the total number of requests received). I then need to process these batches (in the worker thread) and send the correct result back to the correct calling thread. I think I might need to use something like the producer/consumer pattern and somehow marshal the responses back to the calling thread. Does anyone have any pointers/examples of how I might do this?

Thanks very much.

View Complete Post

More Related Resource Links

Bulk Copy From Datatable To SQL Server Table


 I have read the data of a excle file and captured the data into the dataset to a datatable, now that data is to be inserted into a SQL Server table using bulk copy option. I am using -

 public bool BulkEnterData(DataTable dt, string tblName)
        SqlBulkCopy bulk = new SqlBulkCopy(con);
        bulk.DestinationTableName = tblName;

        return true;

ERROR Getting-
A transport-level error has occurred when receiving results from the server. (provider: Shared Memory Provider, error: 0 - The pipe has been ended.) 

How to resolve the above problem or any other solution is available.


How to send bulk sms from asp.net


Hi experts,

I want to send and receive sms in my asp.net website.I am currently targetting users in India.I may need to send the same sms to thousands of users at the same time.How can I achieve this?

Thanks in advance

Concurrency Hazards: Solving 11 Likely Problems In Your Multithreaded Code


Here we describe some of the more common challenges to concurrent programming and present advice for coping with them in your software.

Joe Duffy

MSDN Magazine October 2008

Memory Models: Understand the Impact of Low-Lock Techniques in Multithreaded Apps


Because the use of low-lock techniques in your application significantly increases the likelihood of introducing hard-to-find bugs, it is best to use them only when absolutely necessary. Here Vance Morrison demonstrates the limitations and subtleties low-lock techniques so that if you are forced to use them you have a better chance of using them correctly.

Vance Morrison

MSDN Magazine October 2005

Concurrency: What Every Dev Must Know About Multithreaded Apps


Multithreading offers lots of benefits, but it's not a technique you can employ willy nilly without first learning about all the potential pitfalls. To get you started, Vance Morrison covers multithreading and the shared memory threading model, race conditions and how concurrent access can break invariants. He also discusses how to use locks so you can understand the costs involved.

Vance Morrison

MSDN Magazine August 2005

Bulk Copy Operations in ADO.NET 2.0


Bulk copying of data from one data source to another data source is a new feature added to ADO.NET 2.0. Bulk copy classes provides the fastest way to transfer set of data from once source to the other.

Each ADO.NET data provider provides bulk copy classes. For example, in SQL .NET data provider, the bulk copy operation is handled by SqlBulkCopy class, which is described in Figure 1. As you can see from Figure 1, data from a data source can be copied to one of the four types - DataReader, DataSet, DataTable, or XML.


Can I bulk insert to specific columns when bulk inserting a text file or do I have to bulk insert into all columns of a table? The extra column in the table is an identity integer. So I had to create a view without the identity integer to make it work. Maybe there is a more direct way?? If so what would be the syntax? I am a beginner to intermediate SQL guy.   strSQL = "BULK " & _        "INSERT CustomersUpload " & _        " FROM 'C:\MydataFile.txt' " & _        " WITH " & _        " ( " & _        " FIELDTERMINATOR = '|', " & _        " ROWTERMINATOR = '\n' " & _        " )" Thanks

System.Data.SqlClient.SqlException: Bulk load data was expected but not sent

Hi everyone We've had this error twice now from our C# code. Nothing is changing the columns, but I've noticed though that it appears to happen when two threads are attempting to bulk insert into the same table, generally millions of rows. The docs say that SqlBulkCopy.WriteToServer is not guaranteed to be thread-safe, but is this the problem here? It feels like it. I've seen some posts where people appeared to have experienced problems in similar circumstances (http://social.msdn.microsoft.com/forums/en-US/sqldatabaseengine/thread/85d51484-6381-46bc-bcce-63b58b362d2d). There is no identity column just a non-unique clustered index that is (spid) process keyed to each thread. Do I need to synchronise access to the table for WriteToServer - is this a known problem? Many thanks Reg --- System.Data.SqlClient. SqlException: Insert bulk failed due to a schema change of the target table.   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)   at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternal()   at System.Data.SqlClient.SqlBulkCo

Bulk amount of data in List

Hi All, I got a requirement that to display bulk amount data in a List (Around 50000 items). Can any one of you suggest me best performance way?(Displaying this amount data would definitely leads to performance issue). Let me know if you have any Queries.   Regards, Kumar.

doing bulk update

Hello Everyone, I have a gridview. users can update the database using this gridview. There are about 10 columns and 60 rows in this gridview and once the user makes the changes to a particular row, there is an update button on that row and user can click that button to update the changes that he/she made. My boss wants to remove the update button from each row and put the button at the bottom of the gridview. he says that I will make the changes to whatever rows, I want to make the changes to and then hit the update button that is at the bottom of the grid and it should make the changes to all the rows in the database. I can accomplish this by opening the database coonetion and making a call to the database several times and updating row one by one or I can do all bulk insert. how can I acheive this bulk insert because the other option is quite expensive. Please let me know if anyone has any ideas. Thanks in advance  

Errors during bulk insert

Hi, While writing a stored procedure in SQL 2008, for a bulk insert, I am getting several errors. Part of the SP is written below for your reference. Need to know where am I going wrong. Any help will be thankfully acknowledged.-Learn24x7----------CREATE PROCEDURE Sp_BulkInsert(@Directory nvarchar(256), @TblName nvarchar(50), @Collation nvarchar(50), @FieldSeparator nvarchar(50),    @RowTerminator nvarchar(50))ASBEGIN    SET NOCOUNT ON;                BULK INSERT [@TblName]    FROM ''+@Directory+'' --Error1: see below    WITH --Error2: see below     (     FIRSTROW = 2,     CODEPAGE = (SELECT CAST(@Collation AS INT)), --Error3: see below     FIELDTERMINATOR = ''+@FieldSeparator+'',     ROWTERMINATOR = ''+@RowTerminator+''     )  ENDGO----------FROM ''+@Directory+'' --Error1: An object or column name is missing or empty. For SELECT INTO statements, verify each column has a name. For other statements, look for empty alias names. Aliases defined as "" or [] are not allowed. Change the alias to a valid name.WITH --Error2: Incorrect syntax near the keyword 'with'. If this statement is a commo

Unable to prepare the SSIS bulk insert for data insertion

Hi,   I am using SQL Server Destinations in my data flow tasks. I'm running this package in the server until i encountered this error:   OnError,,,LOAD AND UPDATE Dimension Tables,,,10/24/2007 1:22:23 PM,10/24/2007 1:22:23 PM,-1071636367,0x,Unable to prepare the SSIS bulk insert for data insertion. OnError,,,Load Dimensions,,,10/24/2007 1:22:23 PM,10/24/2007 1:22:23 PM,-1071636367,0x,Unable to prepare the SSIS bulk insert for data insertion. OnError,,,Discount Reason, ISIS Condition, ISIS Defect, ISIS Repair, ISIS Section, ISIS Symptom, Job Status, Parts, Purchase SubOrder Type, Service Contract, Service Reason, Service Type, TechServiceGrp, WarrantyType, Branch, Wastage Reason,,,10/24/2007 1:22:23 PM,10/24/2007 1:22:23 PM,-1073450974,0x,SSIS Error Code DTS_E_PROCESSINPUTFAILED.  The ProcessInput method on component "Dim_T_ISISDefect" (56280) failed with error code 0xC0202071. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.  There may be error messages posted before this with more information about the failure. What could be the reason for this? I don't usually have an error.   cherriesh

SQL ServerDestination error - Unable to prepare the SSIS bulk insert for data insertion.

Having searched the forum, this one clearly has form... However beyond assisting those who have fallen at the first hurdle (i.e. forgetting/not knowing that they cannot execute the package remotely to the instance of SQL Server into which they are inserting), the issues raised by others have not been addressed. Thus I am bringing nothing new to the table here - just providing an executive summary of problems which others have run into, written about, but not received answers for.   First the complete error: Description: Unable to prepare the SSIS bulk insert for data insertion.  End Error  Error: 2008-01-15 04:55:27.58     Code: 0xC004701A     Source: <xxx> DTS.Pipeline     Description: component "<xxx> failed the pre-execute phase and returned error code 0xC0202071.  End Error  DTExec: The package execution returned DTSER_FAILURE (1).  Started:  4:53:34 AM  Finished: 5:00:00 AM  Elapsed:  385.384 seconds.  The package execution failed.  The step failed.   Important points It mostly works - It produces no error more than 9 times out of 10. It fails on random dataflows - My package has several dataflows, (mostly) executing concurrently. Where the error occurs it does not do so on the same dataflow each time: on one run it'll fail on dataflow A whilst B,C,D and E succeed, then A-E will all succeed (and continue doing so for the next ten runs thereafter), and

Strange lockup in multithreaded application - Visual Studio debugger unable to break execution

Multithreaded C# application, Visual Studio 2008, Windows XP, Framework 3.5 SP1. This application is a little crawling utility to download pages from our website, extract some information and upload it into a database. It runs for a while, then somewhere between about 10 minutes and an hour in, it locks up totally, user interface and all. When I try to "break all" in the VS debugger, I get a message "cannot break execution". Now the confusing thing about this is that since the UI was locking up, I thought that some deadlock must be happening on the main UI thread. So I started stripping out any code and locks that were being acquired on that thread that might cause problems ... until there was nothing left. Even with the window simply spawning a bunch of worker threads on create, then totally forgetting about them and giving no reverse access, the lockup still happens, the UI still freezes and VS still can't break the execution.   There are 10 threads to do the crawling, and one each to do the processing and the uploading. The next thing I did was take out the uploading entirely to remove the possiblity of a deadlock in the SqlClient libraries. No difference. So now, the only things that could be really deadlocking the main thread that I can see are the centrally created HTTPWebRequest objects I'm using. WebRequest.create is supposedly thread safe

Get error message from bulk insert task.

Hello, I've created a dtsx package that bulk inserts some data from a csv file into a sql table (using the BULK INSERT TASK). Because this package is scheduled to run automatically, I also added a EXECUTE SQL TASK that will insert a row in a 'log table'. It is linked to the BULK INSERT task with a 'Succes constraint (green arrow)', so I know about all the successes. But I am more interested in the failures, so I also added another EXECUTE SQL TASK and linked it to the BULK INSERT task with the 'failure constraint (red arrow)'. Question: How can I get the error message from the BULK INSERT TASK (like 'file does not exist' or 'error converting .....') and store it in a field in the 'log table'? Thanks in advance, Peter

How to do a parameterized bulk insert into SQL Server 2008 R2 from C# (.NET 4.0)

Hi, I'm trying to execute a BULK INSERT with a parameterized SqlCommand from a C# .NET 4.0 project. This is the code: private System.Data.SqlClient.SqlCommand sqlCmd2; private string sqlCmd2String = "BULK INSERT ExchangeTrackingLogs.dbo.ParsedLogs FROM @name WITH (FIELDTERMINATOR=',', ROWTERMINATOR='\n', FIRSTROW=2)"; private System.Data.SqlClient.SqlParameter[] sqlCmd2Params = new System.Data.SqlClient.SqlParameter[] { new System.Data.SqlClient.SqlParameter("@name", System.Data.SqlDbType.VarChar) }; (...) sqlCmd2Params[0].Value = source; sqlCmd2.Parameters.Clear(); sqlCmd2.Parameters.AddRange(sqlCmd2Params); sqlCmd2.ExecuteNonQuery(); where source is a string containing the full path to a file, i.e. a value "C:\\TEMP\\MSGTRK20100701-1.LOG_EXHUB01". When I execute this code, I get an Exception with Message "Incorrect syntax near '@name'." In the database trace I see an RPC:Completed event with TextData equal to exec sp_executesql N'BULK INSERT ExchangeTrackingLogs.dbo.ParsedLogs FROM @name WITH (FIELDTERMINATOR='','', ROWTERMINATOR='' '', FIRSTROW=2)',N'@name varchar(36)',@name='C:\TEMP\MSGTRK20100701-1.LOG_EXHUB01' I tried using verbatim string literals, but doesn't seem to help. Any idea?

Bulk Insert Problem

I am trying to perform a bulk insert on pipe delimited file (approx 25 columns,5000+ rows of data).  Normally this is no problem however this file has a header and trailer record with no delimiters (1 column).  I could use a utility like grep to remove the header/trailer record but I would like to capture the entire file inside sql server. I can bulk insert the entire file into a single column (wide) that would include the delimiters.  My problem from there is to process the single column data into the appropriate rows /columns.  I am looking for a set solution as opposed to a cursor type solution. HTML?  Seems slow.  Cross Apply?   Changing the file format is not a viable solution. I’m likely to have more than a few of these oddball formats with a variable number of columns.  Any ideas?   I thank you for your time on this puzzle.
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend