Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Data File Conversion

Applies to: WG3.x

Editor's note: this chapter is new in the WG3 layer. It is from section 4 ("Data File Conversion") of the Developer's Addendum for Worldgroup 3.12 (wg312dev; the earlier 3.10 edition, devad3, is the same text apart from "3.10" for "3.12", BC5 for BC50, InstallShield V5.0 for V5.1 and spelling). The addendum has no figures. Prose is unrewritten; "Header check" notes record where the printed text disagrees with the WG3.2 headers. It extends chapter 9 (Databases), which only advises keeping spare bytes in records; the chapter documents WGSDFCVT.EXE, the .CVT file format and the dfc*() API (dfcapi2.h).

The previous data file conversion API’s and .CVT file structures had many shortcomings. Most notably, it became harder to create .CVT files that could properly track and upgrade multiple versions of one data file between both DOS and NT/95. Certain situations would cause data files to be converted more than once, thereby destroying the data contained in the file. These and many other issues forced us to create a new data file conversion API.

Backward compatibility

The new data file conversion API does not remove old functions. Existing .CVT files and custom conversions will continued to be supported through the new API, with a few exceptions:

...

2. Custom conversion programs will only be passed one of 2 strings as the conversion type: “2.0 to NT”, “3.0 to NT”. No modifications are being made to pass “2.0 to 3.1 NT” or any equivalent. If you need to do data file conversions that may require this information, you need to change to the new style custom conversions.

WGSDFCVT.EXE

This utility was totally re-written. The old interface that asked you which files you wanted to convert no longer exists. Instead, the new interface scans all .CVT files and decides if a conversion needs to take place. If a data conversion is required, the operator is asked to specify the directory he would like to back up files to. Leaving this option blank forces no backup. No other user intervention exists, and a screen of information regarding the files wgsdfcvt is converting appears. If no datafile conversion needs to take place, nothing is displayed to the user.

Conversion file theory

The old conversion file API’s worked on the premise that if the “From” file existed and the .CVT file was in the directory, then it needed conversion. Because of this, installation programs needed to have some smarts built into them, to decide which copies of the .CVT files should be installed.

...

.CVT files are used mainly to convert data files from one platform to another. If your conversions require more than simple re-aligning of fields (adding fields for instance), then you can create a custom conversion program and supply your own set of conversion functions.

.CVT file formats

The .CVT file formats are completely different from the older version. There is no longer a need to have multiple .CVT files for one data file. Each .CVT file contains all information pertaining to the life of its associated data file. Numerous examples of the new .CVT file structure can be found in \WGSERV\WGSDFCVT. The new .CVT file format is as follows:

[;]
Description: <description>

Developer: <developer name>

[CustomConversion: <conversion file>]

[
BeginFile: <platform>,<file>[,<platform>,<file>...]
<tagname 1>     TYPE=<field type> [ELEMS=<length>]  [VARIABLE]
...
<fldtagN> TYPE
EndFile

[
BeginFile: <platform>,<file>[,<platform>,<file>,...]
...
]
]

The ; operator

The semi-colon “;” is used as a remark. All text after the semi-colon is ignored.

Description:

This is a required field. Include a description of the data file that this .CVT file is associated with. Text is truncated past 30 characters of the description.

Developer:

This is a required field. Include the name of your company. This field is used to determine the difference between the new and old .CVT files. Text is truncated past 30 characters of the developer name.

CustomConversion:

Use this field to specify a custom conversion program for a data file. No other fields should be included if a custom conversion is present. Custom conversion programs are described later in this document.

BeginFile: and EndFile

These are required fields for any NON-Custom Conversion .CVT file. For every BeginFile:, there is exactly one EndFile. Data Fields are required between each occurrence of BeginFile: and EndFile, and are discussed later in this document. The BeginFile: Line contains a list of platforms and data file names separated by commas. Only one platform/data file is required, however, you may include as many as you want. Each BeginFile:/EndFile combination is a list of data fields contained within a data file. It is quite possible that a data file has the same set of fields for both DOS and WNT, the only difference being the data alignment (DOS being packed, and WNT being unpacked). The following is an example taken from WGSUSR.CVT:

...

The only valid platform identifiers are: DOS for DOS Btrieve 5.x datafiles, and WNT for Btrieve 6.x data files. The data file conversion API assumes the following: DOS platforms are packed/little endian, and WNT platforms are unpacked/little endian. If your conversion requires that these defaults not be followed, you must create a custom conversion and bypass these defaults.

Data Fields

Data Fields are similar to their old versions. You must have at least one data field for every BeginFile:/EndFile combination. Data Fields are defined as follows:

...

The keyword VARIABLE is used to designate a variable length field. Variable length fields only work on the last Data Field in your list. You must include the ELEMS field when you use the VARIABLE keyword, however, the ELEMS field should specify the maximum number elements that could be present in this variable field.

TYPE=STRUCT and EndStruct

The previous version of the data file API had two serious flaws: You could only nest structures one deep, and you could not remove data fields that were within structures. This is no longer the case. Structures are now defined in the same way that BeginFile:/EndFile are used. The following is WGSMENU.CVT. It has an example of variable length structures:

...

All field names are unique. Field names within substructs must have different names than any other fields in your .CVT files. You can not have a field named “KEY” inside a structure and outside a structure.

New Data File Conversion API

All previous data file conversion API’s still exist and are supported, however, you should migrate your software to the new API functions. The new API’s and structures are located in the file DFCAPI2.H.

DFCLIST structure

typedef struct tagDFCLIST {        /*  Platform Specific list of files     */
     INT Platform;                 /*  Operating system for file list      */
     pDFCFILE file;                /*  Pointer to the file information     */
} DFCLIST,*pDFCLIST;

...

pointer to an array of DFCFILE structures that contain a list of files specific to the given platform. This can be NULL when the Platform is DFC_END.

DFCFILE structure

typedef struct tagDFCFILE {        /* Datafile information                 */
     CHAR Name[GCMAXPTH];          /*  Name of the datafile                */
     pDFCFUNCTION cnvfunc;         /*  Conversion function                 */
     pDFCDATAFIELDS cnvfields;     /*  Conversion fields                   */
} DFCFILE,*pDFCFILE;

...

Pointer to an array of DFCDATAFIELDS structures. This array contains information on each data field within the data file similar to those used in .CVT files.

DFCDATAFIELDS structure

typedef struct tagDFCDATAFIELDS {  /* Datafile fields definition           */
     CHAR Name[DFC_NAMSIZ];        /*  Name of the field                   */
     UINT Type;                    /*  Type of the field                   */
     INT nelems;                   /*  Number of elements                  */
     struct tagDFCDATAFIELDS *substruct;/*  Pointer to sub structure       */
} DFCDATAFIELDS,*pDFCDATAFIELDS;

...

If the field type is DFCFLD_STRUCT or DFCFLD_VSTRUCT, you must supply another array of DFCDATAFIELDS for that structure, otherwise this parameter is NULL.

DFCOFFSETS structure

typedef struct tagDFCOFFSETS {     /* Offset information structure         */
     CHAR Name[DFC_NAMSIZ];        /*  Name of the variable                */
     UINT Type;                    /*  Type of record it is                */
     INT nelems;                   /*  Number of elements                  */
     UINT Offset;                  /*  Offset within data record           */
} DFCOFFSETS,*pDFCOFFSETS;

...

Applies to: WG3.x

Header check: no dfcFieldOffst() exists; the function is dfcFindOffset() (dfcapi2.h:127; section "dfcFindOffset()" below).

DFCFILEINFO structure

typedef struct tagDFCFILEINFO {    /*  File info to pass to conversions    */
     INT Platform;                 /*  Current system Platform             */
     CHAR Name[GCMAXPTH];          /*  Conversion file                     */
     INT Type;                     /*  Conversion macros (CVTBIGEND...)    */
     pDFCDATAFIELDS Fields;        /*  Pointer to the fields in data       */
     pDFCOFFSETS Offsets;          /*  Pointers to offset information      */
     UINT Offsetsn;                /*  Number of offsets                   */
     struct flddef *fdef;          /*  Field definition array (set up)     */
     CHAR *Data;                   /*  Pointer to actual data              */
     size_t DataLength;            /*  Length of the data                  */
     struct flddef** fdefs;        /*  Used to allocate memory for fdef    */
     UINT fdefsn;                  /*  Number of field defs                */
} DFCFILEINFO,*pDFCFILEINFO;

...

The number of fdefs in the array. This field should not be used or modified.

DFCPFCONVERT structure

typedef struct tagDFCPFCONVERT {   /*  Platform conversion                 */
     INT PlatformFrom;             /*  Platform converting from            */
     INT PlatformTo;               /*  Platform converting to              */
     pDFCFUNCTION cnvfunc;         /*  Pointer to conversion function      */
} DFCPFCONVERT,*pDFCPFCONVERT;

...

Pointer to a conversion function used to convert data from PlatformFrom to PlatformTo.

dfcAskBackup()

VOID
dfcAskBackup(VOID);                /*  Ask the backup directory            */

Ask the user what directory they wish to back files up to. If you are using dfcRegister to convert your data files, this function is automatically called by the internal API’s.

dfcBackupDirectory()

Applies to: WG3.x

const CHAR * vs CHAR * hazard. CHAR *dfcBackupDirectory(VOID) returns a writable CHAR * (dfcapi2.h); treat it as read-only. The dfc* string parameters (descrp, FileFrom, FileTo, stg, file, FieldName, ModuleName) are const CHAR *.

...

Pointer to the directory the user has asked to back files up.

dfcBackupRequired()

GBOOL
dfcBackupRequired(VOID);           /*  Is a backup required?               */

...

TRUE if the user has requested to back up their data files,

FALSE if they have not.

dfcConvertDefault()

DFCFUNCTION                        /*  Pointer to new data                 */
dfcConvertDefault(                 /* Default conversion function          */
pDFCFILEINFO from,                 /*  File information converting from    */
pDFCFILEINFO to);                  /*  File information converting to      */

...

A pointer to the converted data.

dfcConvertPFDefault()

DFCFUNCTION                        /*  Pointer to new data                 */
dfcConvertPFDefault(               /* Default Platform conversion          */
pDFCFILEINFO from,                 /*  File information converting from    */
pDFCFILEINFO to);                  /*  File information converting to      */

...

A pointer to the converted data.

dfcDisplayBackup()

VOID
dfcDisplayBackup(VOID);            /*  Display backup notice               */

Displays a notice that says “Backing up data files”. If you are using dfcRegister to convert your data files, this function is automatically called by the internal API’s.

dfcDisplayInfo()

VOID
dfcDisplayInfo(                    /* Display DFC information              */
const CHAR *Description,           /*  Description to display              */
const CHAR *FileFrom,              /*  File converting from                */
const CHAR *FileTo,                /*  File converting to                  */
ULONG Records);                    /*  Number of records to display        */

...

Total number of records that are in the data file.

dfcDisplayStats()

VOID
dfcDisplayStats(                   /*  Display the stats bar               */
ULONG TotalRecords,                /*  Total records in conversion         */
ULONG DoneRecords);                /*  Number of records completed         */

...

Total number already converted.

dfcDisplayShutdown()

VOID
dfcDisplayShutdown(VOID);          /*  Release captured video screen       */

Closes all windows that may have been opened while using dfcDisplay... functions. If you are using dfcRegister to convert your data files, this function is automatically called by the internal API’s.

dfcError()

VOID
dfcError(                          /*  Print an error to screen            */
const CHAR *stg,                   /*  String to print                     */
...);                              /*  Extra arguments                     */

...

A message to display upon the termination of your program.

dfcFieldSize()

UINT                               /*  Number of bytes of field size       */
dfcFieldSize(                      /* Gets field size of data type         */
UINT Type);                        /*  Type of datafield                   */

...

The size in bytes of the data type passed, or 0 if the data type is not found.

dfcFilePlatform()

INT                                /*  DFC_DOS, DFC_WNT, DFC_END           */
dfcFilePlatform(                   /* Files platform                       */
const CHAR *file);                 /*  Pointer to file to check            */

...

DFC_WNT for a btrieve 6.x data file.

DFC_END on error.

dfcFindOffset()

pDFCOFFSETS                        /*   returns ptr to field offset info   */
dfcFindOffset(                     /* find offset of a particular field    */
pDFCFILEINFO info,                 /*   file information converting from   */
const CHAR *FieldName);            /*   name of field to find              */

...

A pointer to a DFCOFFSETS structure, or NULL if FieldName does not exist.

dfcGetFDA()

struct flddef *                    /*  Pointer to FDA structure            */
dfcGetFDA(                         /* Get an FDA from type of convert      */
UINT Type);                        /*  Type of option                      */

...

A pointer to a valid flddef structure or NULL. This structure could be used as part of a call to cvtData().

dfcRegister()

GBOOL                              /*  TRUE if conversion took place       */
dfcRegister(                       /*  Register and search for files       */
const CHAR *descrp,                /*  Description name                    */
const DFCLIST *dfc,                /*  pointer to all DFCFILES             */
const DFCPFCONVERT *pfconv,        /*  Platform Converter                  */
INT Key,                           /*  Key number to use (-1 for physical) */
INT Direction);                    /*  Direction (DFC_FORWARD/DFC_BACKWARD)*/

...

FALSE if a conversion did not take place.

dfcRegisterGDB()

GBOOL                              /*  TRUE if conversion took place       */
dfcRegisterGDB(                    /*  Register conversion for GDB         */
const CHAR *descrp,                /*  Description of conversion           */
const DFCLIST *dfc,                /*  Pointer to list of information      */
const DFCPFCONVERT *pfconv,        /*  Pointer to OS conversions           */
const CHAR *ModuleName);           /*  Module name registering             */

...

FALSE if a conversion did not take place.

DFCCONVERT (Conversion functions)

Applies to: WG3.x

Header check: there is no type DFCCONVERT in WGDEV/INC. The conversion-function type is DFCFUNCTION (typedef VOID* DFCFUNCTION;) and pDFCFUNCTION in dfcapi2.h; dfcConvertDefault() returns DFCFUNCTION (dfcapi2.h:117). The heading and the sample in "Adding another data file version" use the name DFCCONVERT.

...

This function must return a pointer to the converted data. You can use the to->Data field to store your converted data. If you are converting a record that is variable length, the from->DataLength reflects the size of the converting from record. You must specify the length of the converting to record by modifying to->DataLength. You may return NULL to NOT convert this record.

Creating a custom conversion program

Every custom conversion program requires that you include the file DFCAPI2.H. This header file contains all function prototypes and structure definitions. The following is a step by step procedure for creating WGSVBCVT.EXE (The file that converts WGSVBL2.DAT).

Create the DFCLIST structure

The Worldgroup Variables data file exists on both DOS and NT/95 platforms. The following structure was created:

...

VariableDOS and VariableWNT are pointers to DFCFILE structures.

Create the DFCFILE structures

The DOS versions of the Worldgroup Variables data file are BBSVBL.DAT and WGSVBL2.DAT. Thus, the following was created:

...

The reason wgsvbl2 is contained within both file lists is because each file has the exact same data field layout. The only difference is the PACKED vs. UNPACKED.

Create the DFCDATAFIELDS structures

As seen in the creation of the DFCFILE structures, that are two different DFCDATAFIELDS structures that need to be created:

...

Notice the DFCFLD_VCHAR fields (Variable CHAR) specify the largest record size that “VARSTUFF” could possibly be.

Creating the platform conversion function list

On most custom conversion programs there is no need for a special function to convert data from the DOS platform to the WNT platform. However, for WGSVBCVT.EXE, there was a need to modify the data manually before saving the WNT version. When this need arises, it is necessary to create a DFCPFCONVERT structure that contains a list of platform conversion functions:

...

The function PlatformConvert will be called when converting data from DOS to WNT.

Create the custom conversion functions

As part of the DFCFILE structures, we specified one conversion function: ConversionDOS1(), and as part of the DFCPFCONVERT structure, we also specified on conversion function: PlatformConvert(). Custom conversion functions are created when you need to transform one version of a datafile to another version, and the transformation requires that you add new fields, or re-adjust somethings value. Custom conversion functions are passed two pointers to DFCFILEINFO structures which provide a wealth of information about both the file you are converting from and the file you are converting to. Here are the two custom conversion functions for WGSVBCVT.EXE:

...

The number of groups we track for changed from Worldgroup 2.00 to Worldgroup 3.00.

Putting it all together

Now that every structure required for the conversion is complete, there is only one thing left: registering all of this information with the data file conversion API. Your programs main() routine should look like this:

...

Your conversion programs no longer receive command line arguments to specify what is being converted. Instead, it is the conversion APIs job to figure out what needs to be converted based on the structures you have created. Calling dfcRegister() notifies the conversion API of the different files, datatypes and conversion functions, and everything else is handled for you. Your conversion functions are only called if a data file conversion takes place.

Adding another data file version

This is the easy part. If the need ever arises to create another version of a data file that already has a custom conversion, it is simple to add it to your conversion program. For instance, say we created a new version of Worldgroup Variables that contained a LONG variable in between the “key” and “varstuff” variables. We would have to do the following (additions are in bold and italicized):

...