From: Philo, John <jphilo@amgen.com>
  To  : 'rasmb' <rasmb@bbri.harvard.edu>
  Date: Mon, 14 Apr 1997 10:13:02 -0700

binary: consensus?; move to new group?

Since the file format/organization discussion has been quiet for several
days, it seems that it is time to try to summarize where we are, and to
decide how to proceed from here.  I also think it is time to move any
further discussion of details to a separate group (more on that below).

Before doing that, however, I note that there has been little discussion
about the best way to organize the data.  In addition to his database
idea, Tom Laue has proposed eliminating the current multiple
subdirectory "tree" scheme and putting all scans for each experiment
into a single directory (an idea which I believe would also require
another major change in the ASCII data file naming convention).  

I would like to encourage everyone to submit their comments and ideas
about data/file organization issues now, to the entire RASMB, so all
viewpoints can be heard.  Some issues that people should think about:
(1) do we want to change the current naming convention for the ASCII
data files for scanner and interference data, or to avoid such changes
for software compatibility? (2) If naming is changed, should we stick
with the DOS 8.3 naming restrictions, or go to Win95 long file names
(which would help organize the files, but would pose problems for all
existing DOS/Win3 programs)?

To summarize the discussion of binary file formats:

(1) One point on which there is wide agreement is that the present ASCII
format should remain available.  There seems to be no major objection to
the idea of a binary format becoming the "primary" format as long as the
current ASCII format can be quickly and easily generated, and as long as
the details of the binary format are made available to everyone.  (If I
am wrong and there are indeed strong objections, speak NOW!)

(2) As regards reasons for switching to binary, probably the most
persuasive is that it should increase the maximum rate of data
acquisition and thus increase the range of experiments that can be done.
 

(3) The actual extent to which binary data files would increase the
speed of acquisition and data analysis, and the actual savings in disk
storage space, appear to depend strongly on the programming language,
specific compiler, and disk operating system.  Nonetheless, it is
certainly true that overall, and over the long run, a binary format
should significantly improve speed and lower disk storage.  

(4) Due to the inefficiencies of disk operating systems, the disk
storage problem exacerbated by the large NUMBER of data files, and there
is potentially a large savings by placing many scans into a single file.

Where do we go from here?  I suggest that we ask Walt Stafford to create
a second e-mail group address list that is specifically for AU software
developers (perhaps we could call it AUSDF - Analytical
Ultracentrifugation Software Developers' Forum).  Those who wish to can
then use this group to thrash out details of file formats (and those who
don't care won't have to deal with it!)  If there is indeed further
general discussion of file organization, that can remain on RASMB for a
few days until it too, runs its course, and then details of
implementation can be discussed within the new list.

Assuming that idea is acceptable to everyone, I will ask Walt Stafford
to create a list including the people listed below who had earlier
expressed interest and/or have participated in the discussion.  In
addition, perhaps Walt could post a message to RASMB telling people how
to add or remove themselves from the new list.  Once the new list is
operational, Walt can post a message to it to let its members know they
can "have at it".  

'Best to all,

John Philo

-----------------------------
Behlke, Joachim
Bloom, Jim
Brasswell, Emory
Butler, Jo
Coelfen, Helmut
Cole, Jim 
Correia, Jack 
Demeler, Borries 
Furst, Allen 
Hensley, Preston 
Holladay, Les 
Johnson, Michael
Langhorst, Brad 
Lary, Jeff 
Laue, Tom 
Lewis, Marc 
McRorie, Don
Minton, Allen
Philo, John
Ralston, Greg
Rossmanith, Peter
Rowe, Arthur
Stafford, Walt
Wu, Jia-Wen
Yphantis, David