Need For File Deployment

A contempory sourcecode site consists of a very large number of files, some of which are text files and some of which have binary formats. It is the mandate of this system that it be able to process an existing sourcecode site and to then transform it into a new sourcecode site with different files, different structure, and even with different internal file format. In addition this, system itself has its own sourcecode site which must be organized and often reconfigured, so that it can be moved to new sites. The term "Deployment" is used to refer to the various operations required to gather large numbers of files into a few bundles and then to redistribute those bundles in a new location.

Though there are many binary compaction formats that could be used to gather files into a single file, this system uses a simple text-based format that is compatible with the UNIX shell system. The details of this format will be discussed later. The advantages of using a text-based format are two-fold. First, different operating systems use different internal file formats for text files, which form the bulk of the files in the site. This system knows these formats and can both recognize and produce them as it performs the various deployment operations. Binary zip formats cannot change the formats as they gather and then distribute files; thus, the user is required to use a separate process to do the text-line format conversions. Second, the authoring facilities of this system produce outputs which contain many files. It makes this operation easier to manage if the authors can produce single files using this text-based format that can then deployed once they have been produced and accepted.

The Deployment Command Line

This page describes how to start the Deployment tool in on the command line. The syntax of the command line is as follows

Deploy filename[.bnd | .dir] [switches]

The actual task performed by the tool depends upon the filename extension. The following different file types are recognized:

Ext	Description of task
bnd	The specified file contains an bundled multi-file output of the tool. At this point those files are unbundled and deployed to their final target destinations.
dir	The specified file contains a list of filenames that are to be bundled into a single file for easy backup, later redeployment, or transfer to another environment.

The following command line switches are recognized:

Switch	Description
ES	Echos the input to the tool as it reads it
UNIX	Requests that text files use UNIX linefeed conventions
QUIET	Requests that no process messages be displayed
VERBOSE	Requests that progess messages br displayed
REPLACE	Requests that existing files be replaced
DEPLOY	Requests that file management commands be executed.
NOHEADER	Requests that no header be displayed
Xname	Specifies an extraction location
Mname	Specifies the name of the file to be made
Pname	Specifies an input filename prefix

A Simple Example

To better understand the need for this deployment capability and the basic bundling and unbundling operations that it performs, the following is a simple example. This work is being performed in a MicroSoft command window. Keeping things simple, the editor being used is notepad. A user of this system has submitted a zip-file that was produced on a Linux machine that contains a large number of individual source files along with a copy of the script he used to do a build and the log of that build. That log, called buildads.log, shows errors which need to be corrected.

After unzipping the files, there are 182 of them, the first step is to use notepad to look at buildads.log to see what the problems are. Unfortuneately notepad reports that this file contains 15 lines, each 1022 characters long that contain a large number of undisplayable characters. In fact all of the files show this sort of thing. Though some MicroSoft tools can work with them, most can not. The problem of course is that Linux uses simple line-feeds to mark the end of text-lines; while MicroSoft expects carriage-return, line-feed pairs at the end of each line (except the last one sometimes -- really). Before this set of files can be used they must be converted into MicroSoft form. This can be done by bundling them and then unbundling them using the deployment capability. Of course, if the user had used this tool to do the bundling in the first place, then this second step would not be necessary. He did not have the tools on the Linux machine, which is often the case.

To bundle a set of files, a list of those files is needed in a separate file whose extension is dir. The easiest way to do this in a MicroSoft command window is with the dir command. For example


buildads.log
buildads.sh

the above is a listing that produces the names of the log and sh files in simple form -- without additional size, type, or date information. Similar commands exist on all operationing systems. Saying instead


 C:\test>dir/b *.log,*.sh >test.dir

produces a file test.dir which contains the names of the two files. Obviously the same can be done for large numbers of files. Once the dir-file exists, the simple command

produces a text file bundle test.bnd that contains the two files. There are two very important points. First, the deployment logic automatically recognized that the input files were in UNIX format and converted them as it read them. There was no need to tell the tool to do this. It always converts to the format of the platform that it is on when it reads a file to be bundled. Second, the bundled file is now a simple text file in the local MicroSoft format and can be viewed/edited with notepad. It looks generally as follows, showing the start of each file


cat >buildads.log <<'!)(!'
  mkdir -p ./lnx

  echo Making imslib.a ...
Making imslib.a ...
  cd imslib
    rm -r -f *.o
    cc -c -DAN6PLAT -I./ -I../include *.c
rdqlocal.c: In function `RdqPushSymbol':

.....

cat >buildads.sh <<'!)(!'
  mkdir -p ./lnx

  echo Making imslib.a ...
  cd imslib
.....

Note that each file is introduced by a cat command and is terminated by a !)(! marker. The tool understands these commands but so do the various shell programs originally developed for UNIX, but now available for all platforms. This deployment capability is NOT needed to deploy these bundled files.

The bundled file could now be worked with directly, but to show the entire process, the tool can now be told to unbundle the test.bnd file.


deploy V9.82 (BETA.001) System Build(12/08/09 16:05:15)
The file already exists: buildads.log
The file already exists: buildads.sh

The bnd extension tells the tool that an unbundling operation is to be performed. One of the problems with the deployment process is that it is easy to unintentionally overwrite existing files. The tool will not overwrite an existing file unless it is explicitely told to do so.


deploy V9.82 (BETA.001) System Build(12/08/09 16:05:15)
Unbundling: buildads.log
Unbundling: buildads.sh

Now the files that were originally in UNIX format have been replaced by files that are in native format and can be worked on with all the native tools.

Assuming now that what corrections were needed have been made and the files are to be sent back to the Linux user who does not have this tool set on that site. He does, however, have a UNIX shell. All that is needed is a bundled file that uses UNIX line-feed conventions. This is done with the UNIX command-line option


deploy V9.82 (BETA.001) System Build(12/08/09 16:05:15)
Processing: buildads.log
Processing: buildads.sh

The test.bnd file produced now does not use MicroSoft conventions, it uses UNIX conventions. It can be shelled directly on the Linux machine. Alternatively, though it is now in UNIX format, if its unbundled back to the MicroSoft platform, the files would be in MicroSoft form.

Table of Contents

gmCLDeployTool

Need For File Deployment

The Deployment Command Line

A Simple Example