Anyone here ever hear of "baton files"? We're supposed to use them in an ETL project. It's apparently an old mainframe trick - you transfer the big, fat data file first foo.csv, and when that's done you transfer a tiny baton file, foo.baton (or
other suffix). The baton file's mere existence tells you the big file has finished transfering in, which might otherwise be hard to tell. For extra credit the baton file can contain useful information like a rowcount and/or some aggregates or checksums
of the accompanying data.
OK, I 've seen that before, but apparently there's a second part of this kind of protocol, where they also send along a table schema in SQL. We have one project suggesting that this be checked against our table formats every time, in production, in
case the sender unexpectedly changes format (and yet is good enough to send us the changed and accurate schema!).
Anyone ever hear of this second part, and can you say what the origin might be?
And do you think it's a good idea?
View Complete Post