Need For File Deployment

A contemporary sourcecode site consists of a very large number of files, some of which are text files and some of which have binary formats. It is the mandate of this system that it be able to process an existing sourcecode site and to then transform it into a new sourcecode site with different files, different structure, and even with different internal file format. In addition, this system itself has its own sourcecode site which must be organized and often reconfigured, so that it can be moved to new sites. The term "Deployment" is used to refer to the various operations required to gather large numbers of files into a few bundles and then to redistribute those bundles in a new location.

Though there are many binary compaction formats that could be used to gather files into a single file, this system uses a simple text-based format that is compatible with the UNIX shell system. The details of this format will be discussed later. The advantages of using a text-based format are two-fold. First, different operating systems use different internal file formats for text files, which form the bulk of the files in the site. This system knows these formats and can both recognize and produce them as it performs the various deployment operations. Binary zip formats cannot change the formats as they gather and then distribute files; thus, the user is required to use a separate process to do the text-line format conversions. Second, the authoring facilities of this system produce outputs which contain many files. It makes this operation easier to manage if the authors can produce single files using this text-based format that can then be deployed once they have been produced and accepted.

The Deployment Command Line

This page describes how to start the Deployment tool on the command line. The syntax of the command line is as follows

Deploy filename[.bnd | .dir] [switches]

The actual task performed by the tool depends upon the filename extension. The following different file types are recognized:

The following command line switches are recognized:

A Simple Example

To better understand the need for this deployment capability and the basic bundling and unbundling operations that it performs, the following is a simple example. This work is being performed in a Microsoft command window. Keeping things simple, the editor being used is notepad. A user of this system has submitted a zip-file that was produced on a Linux machine that contains a large number of individual source files along with a copy of the script used to do a build and the log of that build. That log, called buildads.log, shows errors which need to be corrected.

After unzipping the files, there are 182 of them, the first step is to use notepad to look at buildads.log to see what the problems are. Unfortunately, notepad reports that this file contains 15 lines, each 1022 characters long that contain a large number of undisplayable characters. In fact, all of the files show this sort of thing. Though some Microsoft tools can work with them, most cannot. The problem of course is that Linux uses simple line-feeds to mark the end of text-lines; while Microsoft expects carriage-return, line-feed pairs at the end of each line (except the last one sometimes -- really!). Before this set of files can be used, they must be converted into Microsoft form. This can be done by bundling them and then unbundling them using the deployment capability. Of course, if the user had used this tool to do the bundling in the first place, then this second step would not be necessary. The tools were not available on the Linux machine, which is often the case.

To bundle a set of files, a list of those files is needed in a separate file whose extension is dir. The easiest way to do this in a Microsoft command window is with the dir command. For example

Code Block
language none theme Eclipse linenumbers true

> dir/b *.log,*.sh buildads.log buildads.sh

the above is a "brief" listing that produces the names of the .log and .sh files in simple form -- without additional size, type, or date information. Similar commands exist on all operating systems. Saying instead

Code Block
language none theme Eclipse linenumbers true

> dir/b *.log,*.sh >test.dir

produces a file test.dir which contains the names of the two files. Obviously the same can be done for large numbers of files. Once the .dir file exists, the simple command

Code Block
language none theme Eclipse linenumbers true

> Deploy test.dir

produces a text file bundle test.bnd that contains the two files. There are two very important points. First, the deployment logic automatically recognized that the input files were in UNIX format and converted them as it read them. There was no need to tell the tool to do this. It always converts to the format of the platform that it is on when it reads a file to be bundled. Second, the bundled file is now a simple text file in the local Microsoft format and can be viewed/edited with notepad. It looks generally as follows, showing the start of each file

Code Block
language none theme Eclipse linenumbers true
cat >buildads.log <<'!)(!' mkdir -p ./lnx echo Making imslib.a ... Making imslib.a ... cd imslib rm -r -f *.o cc -c -DAN6PLAT -I./ -I../include *.c rdqlocal.c: In function `RdqPushSymbol': ..... !)(! cat >buildads.sh <<'!)(!'

mkdir -p ./lnx

echo Making imslib.a ...

cd imslib ..... !)(!

Note that each file is introduced by a cat command and is terminated by a !)(! marker. The tool understands these commands but so do the various shell programs originally developed for UNIX, but now available for all platforms. This deployment capability is NOT needed to deploy these bundled files.

The bundled file could now be worked with directly, but to show the entire process, the tool can now be told to unbundle the test.bnd file

Code Block
language none theme Eclipse linenumbers true

> Deploy test.bnd verbose deploy V9.82 (BETA.001) System Build(12/08/09 16:05:15) The file already exists: buildads.log The file already exists: buildads.sh

The bnd extension tells the tool that an unbundling operation is to be performed. One of the problems with the deployment process is that it is easy to unintentionally overwrite existing files. The tool will not overwrite an existing file unless it is explicitly told to do so, using the REPLACE option,

Code Block
language none theme Eclipse linenumbers true

> Deploy test.bnd verbose REPLACE deploy V9.82 (BETA.001) System Build(12/08/09 16:05:15) Unbundling: buildads.log Unbundling: buildads.sh

Now, the files that were originally in UNIX format have been replaced by files that are in native format and can be worked on with all the native tools.

For deployment on UNIX platforms, text files must use line-feed (LF) end-of-line conventions. If you use the UNIX command-line option, the deploy utility will do this automatically on the Windows text files listed in test.dir

Code Block
language none theme Eclipse linenumbers true

> deploy test.dir UNIX VERBOSE deploy V9.82 (BETA.001) System Build(12/08/09 16:05:15) Processing: buildads.log Processing: buildads.sh

The test.bnd bundle file produced now uses UNIX end-of-line conventions, and it can be shelled directly on a UNIX Linux machine, using for example the UNIX command sh

Code Block
language none theme Eclipse linenumbers true

> sh test.bnd

On Windows, though it is now in UNIX format, test.bnd will unbundle into files that use Windows end-of-line format conventions

Code Block
language none theme Eclipse linenumbers true
> deploy test.bnd verbose REPLACE deploy V9.82 (BETA.001) System Build(12/08/09 16:05:15) Unbundling: buildads.log Unbundling: buildads.sh