The LongMemory Service Class

The service class LongMemory manages long memory areas. In the area of information management program storage must be viewed as containing a wide variety of different structures, indexes, lists, fixed records, long unformatted records, etc.. In addition it may be purely memory bound or it may be stored in a persistent file system. The role of this class is to provide a single interface for storing and retrieving information. It is sufficiently robust to allow for the management of large files in the multiple gigabyte size range; while it is still efficient for use by purely memory bound applications.

Physically, a long memory storage area consists of a series or relatively large fixed sized blocks. The manner in which these blocks are created, maintained, and accessed is determined when the storage area is created. See the discussion within the LongMemory_Create method for a description of the different approaches used. Within these large fixed sized blocks there are four fundamentally different record types stored within the storage area:

1. Control Tables -- These are relatively short tables that are always stored entirely within a single block. The method LongMemory_Allocate is used to enter new control tables into the storage area.
2. Data Tables -- These are arbitrarily long vectors that may cross block boundaries, but whose entire expanse consists of a logically contiguous area. The method LongMemory_Expand is used to enter new data tables into the storage area.
3. Blocked Tables -- These are arbitrary collections of information that use entire storage blocks that need not be contiguous. The manner in which the blocks are used is dependent on the type of information collections. Most complex entities like indexes, storage based property sequences, short volatile records, etc. are blocked tables. The method LongMemory_EmptyBlock is used to allocate blocked tables in the storage area.
4. Posting blocks -- These are fixed sized linked smaller blocks used to contain posting, collections of integer value such as selection vectors, inverted indexes, and sorting control tables.

At a logical level the storage area is viewed as consisting of a simple sequence of words. In this implementation a word is a 32-bit signed integer. All information is stored on a word boundary and all offsets are expressed in terms of words.

The method LongMemory_Access

Prototype

The LongMemory_Access method is the lowest level logical access method in the storage system. The caller specifies a word offset into the storage area that he wishes to access along with his intent to change it. This method returns a semipermanent pointer to that word in the storage area. This method has the logical view of the storage area -- i.e., a long linear sequence of words numbered from zero. Its parameters are:

If all goes well this method returns a semipermanent pointer to the word; else it takes a fatal error branch and does not return.

The method LongMemory_Allocate

Prototype

The LongMemory_Allocate method allocates a contiguous area of storage. Though normal "data" storage in the persistent storage area makes no assumptions about the occurrence of block boundaries, the basic entry tables around which the object identifiers and property vectors are built assume that they do not contain any internal boundaries. It is this assumption that makes it possible to use semipermanent pointers to access these areas. This method allocates a stretch of continuous storage in the storage area that does not cross a block boundary. Obviously the length of that area must be less than the pagesize being used in the underlying physical storage of the blocks. Its parameters are:

This method returns the actual starting offset in the persistent storage area of the allocated area.

The method LongMemory_AllocateBytes

Prototype

The LongMemory_AllocateBytes method allocates sufficient storage to accommodate a specified number of bytes that crosses no block boundaries. The storage starts on a word boundary. Though normal "data" storage in the persistent storage area makes no assumptions about the occurrence of block boundaries, the basic entry tables around which the object identifiers and property vectors are built assume that they do not contain any internal boundaries. It is this assumption that makes it possible to use semipermanent pointers to access these areas. This method allocates a stretch of continuous storage in the storage area that does not cross a block boundary. Obviously the length of that area must be less than the pagesize being used in the underlying physical storage of the blocks. Its parameters are:

This method returns the actual starting offset in the persistent storage area of the allocated area.

The method LongMemory_Close

Prototype

The LongMemory_Close method closes the specified storage area and returns all of its memory resources to the operating system. Its parameter is:

This method has no return value.

The method LongMemory_Create

Prototype

The LongMemory_Create method creates a storage area. Once the information about how the blocks of the persistent storage area are to be managed in memory is known, this method creates that area along with the machinery needed to manage its fixed-sized blocks and block boundaries. Note that this is a very low level method. Typically the methods LongMemory_OpenArea or LongMemory_OpenRaw should be used, as these select the best approach to be used given the context of use. Its parameters are:

If all goes well, this method returns the storage area handle to be used for the created area; else it returns NULL.

The method LongMemory_DeleteBlock

Prototype

The LongMemory_DeleteBlock method logically deletes a block from blocked-table storage. The block is not physically deleted, rather it is entered into the deleted block chain. From there it can be reused via the LongMemory_EmptyBlock or the LongMemory_Allocate methods. Its parameters are:

This method has no return value.

The method LongMemory_DeleteRecord

Prototype

The LongMemory_DeleteRecord method logically deletes a contiguous record from long memory. The record is not physically deleted, rather it is posted into a deleted record chain. From there it can be reused via the LongMemory_Allocate method. Its parameters are:

This method has no return value.

The method LongMemory_EmptyBlock

Prototype

The LongMemory_EmptyBlock method gets an empty block from a storage area. It is used whenever a block is needed by a blocked-table storage user. The block may be at the end of the storage area or it may be a block somewhere inside of the area that was marked as being deleted by the method LongMemory_DeleteBlock. To emphasize, blocks returned by this method are not guaranteed to be contiguous. If contiguous long storage is needed use the LongMemory_Expand method. Its parameters are:

If all goes well this method returns a semipermanent pointer to the start of the block; else it returns a NULL and stores a zero in the BlockNumber parameter.

The method LongMemory_Extend

Prototype

The LongMemory_Extend method extends a storage area. Information can only be put or gotten from areas that have been previously allocated either via the LongMemory_Allocate or the LongMemory_Extend methods. This method extends the persistent storage area by the indicated number of words without regard to any block boundaries that might be spanned by the new area. This area then can be used for data storage but not for control tables and property vector storage. Its parameters are:

This method returns the actual offset in the persistent storage area of the start of the new area.

The method LongMemory_GetBlock

Prototype

The LongMemory_GetBlock method is the lowest level physical access method for the persistent storage area. The caller specifies a physical block number that he wishes to access along with his intent to change it. This method returns a semipermanent pointer to the start of the block in memory. Its parameters are:

If all goes well this method returns a semipermanent pointer to the start of the block, else it takes a fatal error branch and does not return.

The method LongMemory_GetDeleted

Prototype

The LongMemory_GetDeleted method determines if a deleted contiguous record exists that is at least a specified size long. If so it removes it from that deleted record chain and returns its offset in the long memory area. This method is used by the contiguous record allocation methods before they attempt to allocate new memory to satisfy a request. Its parameters are:

If a deleted segment is found that contains sufficient room to satisfy the request, then its offset in the long memory area is returned. Else a zero is returned.

The method LongMemory_GetNext

Prototype

The LongMemory_GetNext method supplies the retrieval offset associated with the next available word of long memory. The user may then begin writing to this storage. Note that the user is guaranteed that he can perform offset arithmetic with this retrieval value however, he is not guaranteed that the memory associated with these values is contiguous. Thus information may only be copied into (written) and copied from (read) the storage area. Its parameter is:

This method returns the retrieval offset for the next available word of storage.

The method LongMemory_GetPageSize

Prototype

The LongMemory_GetPageSize method is for use by those storage applications that use their own internal block layouts. Its parameter is:

This method returns the page size of the storage area.

The method LongMemory_GetPostingSize

Prototype

The LongMemory_GetPostingSize method returns the posting size of the storage area for use by those storage applications that use their own internal block layouts. Its parameter is:

This method returns the posting size of the storage area.

The method LongMemory_HasChanged

Prototype

The LongMemory_HasChanged method specifies if any changes have been made to the storage area. If not, then the closing of various components can either be skipped or simplified. Its parameter is:

If the storage area has not been changed, then a zero is returned. If it has been changed, then a nonzero value is returned.

The method LongMemory_Open

Prototype

The LongMemory_Open method opens a storage area. Once the information about how the blocks of a previously created persistent storage area are to be managed in memory is known, this method opens that storage area and creates the machinery needed to manage its fixed-sized blocks and block boundaries. Note that this is a very low level method. Typically the method LongMemory_OpenArea or LongMemory_OpenRaw should be used, as these select the best approach to be used given the context of use. Its parameters are:

If all goes well,this method returns the storage area handle to be used to reference the storage area. A NULL return value means that the file could not be opened.

The method LongMemory_NewBlock

Prototype

The LongMemory_NewBlock method adds a new block to the back of the storage area. It is used in those cases where long sequential records are being added to the memory area which must span multiple sequential blocks. It is of course also used by the various allocation methods that require storage space when no deleted blocks are available. Its parameter is:

If no new blocks can be allocated to the long memory area then a NULL is returned; else a semipermanent pointer to the start of the block is returned.

The method LongMemory_OpenArea

Prototype

The LongMemory_OpenArea method creates or opens a long storage area. It selects the best paging method depending upon the context of use. Its parameters are as follows:

If all goes well,this method returns the storage area handle to be used to reference the storage area else it returns a NULL.

The method LongMemory_OpenRaw

Prototype

The LongMemory_OpenRaw method creates or opens an existing storage area that has it own internal structure. No boot record is maintained and no deleted block or segment information is retained. The only methods that should be used with raw storage areas are: LongMemory_Access, LongMemory_Close, LongMemory_Read, LongMemory_Rewrite, or LongMemory_Write. Its parameters are:

If all goes well this method returns the handle for the controlling storage structure; else it returns a NULL.

The method LongMemory_ReadBytes

Prototype

The LongMemory_ReadBytes method reads bytes of information from persistent storage by copying them into a buffer supplied to this method. A byte-stream is retrieved using this method. Doing the copy ensures that the ultimate user of these bytes need not be concerned with boundary problems and need not be concerned over the duration of use of those values. This method has the logical view of the storage area -- i.e., a long linear sequence of words numbered from zero. Its parameters are as follows:

The method LongMemory_RewriteBytes

Prototype

The LongMemory_RewriteBytes method rewrites bytes of information into persistent storage by copying them from a buffer supplied to this method. This method is used for storing byte-streams as opposed to word streams. Its parameters are as follows:

The method LongMemory_ZeroBytes

Prototype

The LongMemory_ZeroBytes method zeros bytes of information in persistent storage. This method is used for initializing byte-streams as opposed to word streams. Its parameters are:

The method LongMemory_WriteBytes

Prototype

The LongMemory_WriteBytes method writes byte-stream information to the back of the storage area. Its parameters are as follows:

The return value from the method is the actual offset in persistent storage of the start of the new area. Note that the offset is word-oriented, not byte oriented.

The method LongMemory_Read

Prototype

The LongMemory_Read method reads words of information from persistent storage by copying them into a buffer supplied to this method. Most "data" is retrieved using this method. Doing the copy ensures that the ultimate user of these words need not be concerned with boundary problems and need not be concerned over the duration of use of those values. This method has the logical view of the storage area -- i.e., a long linear sequence of words numbered from zero. Its parameters are as follows:

The method LongMemory_Rewrite

Prototype

The LongMemory_Rewrite method rewrites words of information in persistent storage by copying them from a buffer supplied to this method. Most "data" is stored using this method. This is because the normal process of adding records to the back of storage involves extending it first and then doing a rewrite into the extension. Doing the copy ensures that the ultimate creator of these words need not be concerned with boundary problems. Its parameters are as follows:

The method LongMemory_Write

Prototype

The LongMemory_Write method writes words to back of a storage area. Its parameters are as follows:

The method returns the actual offset in storage of the start of the record. The record can be later read using this offset.