.NET Tutorials, Forums, Interview Questions And Answers
Welcome :Guest
Sign In
Win Surprise Gifts!!!

Top 5 Contributors of the Month
Gaurav Pal
Post New Web Links

ETL in distributed environment (Scale out architecture)

Posted By:      Posted Date: September 22, 2010    Points: 0   Category :Sql Server


Currently I'm using SSIS packages, C#,Powershell, BulkInserts technique for ETL and I’ve one database with different schemas.
for example core.table1, core.table2, trans.table1, trans.table2 in SQL-2008.

I would like to create 3 different SQL physical nodes for load balancing, performance and to handle large scale of data.

Two simple questions:
1) create same table structure in all 3 node, and load data parallel,
 for example if I’m receiving 1000 records- distribute 1000 records in 3 nodes (300 + 300 + 400),
 To retrieve the data I should have same algorithms, three separate thread process parallels and get me the result form 3 nodes.

2) Create one schema on one node, another in second node and dump the data accordingly.
       for example all core. tables are under node1, trans. tables are under node2 and so on..

please let me know the pros & cons of above approaches?

Is that Store procedure adviseable in distributed SQL environmnet?
I feel LINQ-SQL is more convineant as application get the data by using DBML Schema? Please advise


Which would be preferable approch on ETL,
1) Extract > Transform > Load
2) Extract > Load (Flat file Load) > Trnasform > Load

 I'm conf

View Complete Post

More Related Resource Links

Scale Out: Distributed Caching On The Path To Scalability


Developers frequently face issues related to scalability bottlenecks when they have applications that need to handle a lot of load. This article describes the role of a distributed cache in building scalable applications.

Iqbal Khan

MSDN Magazine July 2009

What are Best practice for setting up databases in a distributed load balancing environment



What is the best practice out of the following in a distributed load balancing environment:

keep application database and persistent/tracking database seperate

Keep application , persistence , tracking data all in one database.



SQL 2005 Express to work with C# in a distributed architecture


Hi, I am super new using SQL Srv 2005, I have been using for years Access 2003 and everyone is pointing me to better use SQL Server 2005 for many reasons. I haven't had any problem until now using Access2003 and I am using a distributed application developed in C# that basically uses the Access2003 file that is located in a shared folder in the LAN. However, I think it is worth to learn SQL Srv 2005 and more if there is a free version (Express).


I already have a deployed C# application that is using an MSAccess2003 that is located in a shared folder in the LAN (around 30 computers already have this application connected to the same LAN, of course), I am planning to install in the server the SQL Srv 2005 and use it.


Could I have more than one application connected to SQL Server 2005 Express edition at the same time? (<- since it is free, I have my questions about this scenario)




Architecture Tools in VSTS 2010

Designing new functionality on existing applications can be daunting. There are always differences between the original design and the current implementation.

The new Architecture tools within Visual Studio Team System 2010 help you understand the application you have, design new functionality you need, and validate that your design and your implementation do not deviate.

SharePoint Tutorial - SharePoint Logical Architecture

The SharePoint logical architecure consists of nested and isolated levels. These levels provide benefits for security, navigation, search, branding and ect. depending on the level

SharePoint Farm
The base level for any SharePoint implementation is the SharePoint farm. Physically a farm can consist of one server or many servers. An organization may implement one or more farms. This usually depends on security and performance needs.

SharePoint Tutorial -SharePoint Physical Architecture

The SharePoint physical architecture consists of services running on one or more servers.

There are three types of services that together run the SharePoint farm: web, application and database services. Web services through Internet Information Server is what processes the ASP.Net and sends back the html to a user's browser. Application services is where the "brains" of SharePoint are located. All of the SharePoint specific logic and services are processed there. Database services are run by SQL Server and it's where all the data/content for SharePoint is stored.

MVC Architecture

The main aim of the MVC architecture is to separate the business logic and application data from the presentation data to the user.

Here are the reasons why we should use the MVC design pattern.

They are resuable : When the problems recurs, there is no need to invent a new solution, we just have to follow the pattern and adapt it as necessary.
They are expressive: By using the MVC design pattern our application becomes more expressive.

MVC architecture in ASP.Net using C# and Microsoft Data Access Application block

The Model-View-Controller (MVC) pattern separates the modeling of the domain, the presentation, and the actions based on user input into three separate classes [Burbeck92]:

Model. The model manages the behavior and data of the application domain, responds to requests for information about its state (usually from the view), and responds to instructions to change state (usually from the controller).

View. The view manages the display of information.

Controller. The controller interprets the mouse and keyboard inputs from the user, informing the model and/or the view to change as appropriate.

Project Example: ASP.NET MVC + SubSonic Architecture

use of SubSonic to provide the DAL in association with some ASP.NET MVC sites and was waylaid in my original intent to throw up a post summarizing the architecture we're using for the effort. Well I've finally gotten around to it. Recall that in his fake it till you need it post, Dave provided a window on our architecture that I'll build upon here

Three Tier Architecture with ASP.NET

ASP.NET supports layered architecture, and this article will illustrate how it can work with the ASP.NET presentation controls. This article focuses mainly on the GridView control, mostly because it is prevalently used in .NET development. However, the concepts work for other controls as well.

ASP.NET 2.0 (III) Architecture and Tools

In this third article we'll take a look at the changes to the architecture of ASP.NET v2.0. In particular we'll look at the changes to the compilation and deployment model.

High-Performance .NET Application Development & Architecture

It has always been a goal of project architects to plan an effective strategy from the ground up in regards to an new application. All relevant factors are taken into consideration with respect to the application, from its design and layout to a functional website infrastructure. Pre-.NET strategies and design guidelines still effective now were developed with Microsoft's DNA (Distributed interNet Application) platform. This model successfully served the purpose of architecting N(any number of)-Tier (levels) applications. In its basic sense, as in most robust, distributed applications, you'll architect 3 main layers or Tiers: presentation, business rules and data access.

Application architecture in asp .net

Application Architecture is the most important process of developing a good application. Some people tends to jump into coding without any architecture laid. Later we see those people changing the architecture and finally the application had to be developed from the scratch with the correct architecture.

In this article I will describe some details about how you can improve the architecture of you application. This article is targeted to the beginner's audience.

A Low Level Look at ASP.NET Architecture

ASP.NET is a powerful platform for building Web applications, that provides a tremendous amount of flexibility and power for building just about any kind of Web application. Most people are familiar only with the high level frameworks like WebForms and WebServices which sit at the very top level of the ASP.NET hierarchy. In this article I'll describe the lower level aspects of ASP.NET and explain how requests move from Web Server to the ASP.NET runtime and then through the ASP.NET HTTP pipeline to process requests.

Realizing a Service-Oriented Architecture with .NET

This article is meant to be a practical discussion guide to building a .NET application in a service-oriented architecture. We will consider real-world goals, real-world obstacles, and experience-based solutions. I quickly concede the approaches discussed here are not exhaustive or infallible. This paper is focused on application development, not application integration. We will specifically consider architectural issues and component design issues.

The Potential of Web Services

3-tier architecture in C# sample

This article discusses and implements 3-tier architecture using C# and a dummy customer using MS Access database. In this article I am trying to implement a small reusable component that maintains customers in 3-tier architecture. It shows how to add, update and find customer details.

ASP.NET Architecture - ASP.NET Worker Process - HTTP Pipleline - Http Modules and Handlers

"ASP.NET is a powerful platform for building Web applications, that provides a tremendous amount of flexibility and power for building just about any kind of Web application. Most people are familiar only with the high level frameworks like WebForms and WebServices which sit at the very top level of the ASP.NET hierarchy. In this article I'll describe the lower level aspects of ASP.NET and explain how requests move from Web Server to the ASP.NET runtime and then through the ASP.NET Http Pipeline to process requests.
ASP.NetWindows Application  .NET Framework  C#  VB.Net  ADO.Net  
Sql Server  SharePoint  Silverlight  Others  All   

Hall of Fame    Twitter   Terms of Service    Privacy Policy    Contact Us    Archives   Tell A Friend