I have been tasked with importing a large number of old log files into a database. These files came from a Unix type server and therefore have scattered Cr and Lf's throughout. Of course anyone who has ever tried to use a streamreader.ReadLine method knows this plays havoc.
My First attempt was just to do a ReadToEnd, and then use the Replace method to filter out everything but the CrLf's... this worked outstanding.... until my file sizes got too large, and then I started getting Out of Memory Issues.
According to Google, the solution was to use a BinaryReader along with it's Peek method to look Read in a Character at a time until a CrLf was found... This also worked great .... However it was extremely slow.
For example, I did a test on a 1mb file. The ReadLine method processed the file (even though incorrectly) in 8 seconds... the Binary Reader (though Correct) took 48 seconds. Doesn't sound like much but when you are looking at Gigs worth of file, it was taking hours to process just 1 CD.
... So finally to the good stuff..... I knew ReadLine was the way to go...but h
View Complete Post