Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Reliability

Debug Versions

Applies to: v10

Debugging a v10 add-on: to debug in Release mode, start the BBS, then in Visual Studio choose Debug, Attach to Process, and select wgserver.exe (MBBS10.txt section 3). Some bugs appear only in Release because, according to MBBS10.txt, Debug builds zero out pointers (so garbage memory does not always show up), and likewise for string-literal misuse (chapter 29, section 29.4). The Debug-pointer-zeroing statement is the author's observation: unverified.

...

#ifdef DEBUG
    for (i=0 ; i < MAXLST ; i++) {
        if (list[i] == NULL) {
            catastro("NULL list entry!");
        }
    }
#endif
Assertions

An easy to use debugging tool is the assertion. Assertions are failures triggered by the ASSERT() macro. ASSERT() is used to assert that a condition is always true. When DEBUG is not defined, ASSERT() calls are not evaluated. When DEBUG is defined, ASSERT() will fail if the passed condition is not true. In the failure message will be the filename, file version, line number, and specific condition that failed. By default, assertion failures take the form of catastro() crashes. But, assertions can be altered to fail in different ways, as described below.

...

In this example, the debug version will increment i, the non-debug version won’t. The following code may rely on i being incremented, and work fine as long as you’re running the debug version. Once you build final, non-debug shipping disks, you’ll be surprised to find a new bug. Bugs caused by debugging code are often the most painful.

Phase Transitions

Another helpful method of debugging involves using phase transitions. Phase transitions are used to track the basic flow of code by recording major events. The BEG_PHASE() and END_PHASE() macros are used to record the beginning and end of phases or events. The last x phase transitions are reported in catastro() and GP crashes to help understand what led to the crash. BEG_PHASE() and END_PHASE() both take a string and a long for a brief description of the phase and any relevant data. Generally, they are used to bracket some well-defined set of code:

...

In the above case, the address of the global command just executed is included as the extra parameter to END_PHASE(). This will report the address of the global command just executed along with the Global command end transition message.

Debug Options in CNF

If you create any debug-only CNF options, put them in CNF’s level 95. That’s the level where all debugging options go by default. A Sysop can edit those options by running CNF directly from DOS with a special command-line parameter:

CNF -L95
The Baseline's Debugging Module

The Debugging in baseline module is installed automatically as part of the Client/Server Developer’s Kit. By default, it’s disabled. To enable it, go to Basic Utilities (choice 7 on the Introductory Menu), run WGSDMOD (the Enable/Disable Modules utility), and move Debugging in baseline from the Expressly Disabled column to the Available column.

...

Check out CNF level 95 once you’ve enabled the debugging module. All of its features can be turned on or off from there. The available options are briefly described in the option help messages. Two of the most notable are:

    the ability to debug memory allocations in order to detect memory under/overflow bugs; andthe ability to alter assertion failures to be quietly ignored or logged to disk rather than catastro(). Some Philosophy on Debugging

    Your best debugging work takes place long before you test-run any of your software. A programming task that’s well thought through and meticulously coded has the best chance of long term success.

    ...

    You’re probably trying to increase your programming speed just like we are. Nothing will help your overall productivity more than an ingrained habit of completing highly reliable foundations before you proceed to use them. There are many techniques for making your foundations highly reliable: code walk-throughs, rigorous testing by the developer, line-by-line testing of source code, branch-by-branch testing, routinely retesting “repaired” code, alpha-testing (in-house), beta-testing (by motivated customers). The goal is to get software done fast, put it to use, and move on to other projects, without your customers having to find the problems and you solving them over the phone.

    Programming Tips for

    ...

    the Major BBS

    There are several unique things about the programming environment of Worldgroupthe Major BBS. The multi-user single-tasking aspects mean that your code can’t wait very long for any one thing to happen.

    Once you become proficient in programming for Worldgroupthe Major BBS, there are a few classes of problems you’ll want to be on guard against. If you review these problems and develop an eagle eye for them, then your code can be much more reliable and free from frustrating bugs.

    ...

    Wrong wayRight way
    prfmsg(GREETING);prfmsg(GREETING);
    if (problem) {if (problem) {
    prfmsg(WARNING);prfmsg(WARNING);
    outprf(usrnum);}
    }outprf(usrnum);
    outprf(usrnum);

    General Protection Faults

    The best thing and the worst thing about protected mode operation is the “General Protection” fault, or GP. Protected mode is designed to halt at the slightest sign of a program gone awry. The first time you try to unravel and debug a GP will be an adventure. But after all the work of tracking it down, it will probably lead you directly to a bug in your code.

    What is a GP?

    A General Protection fault is the computer CPU’s way of saying “I give up” or “I can’t do this”. Usually it has to do with an invalid or restricted memory address. There are other GP causes, like privilege violation, executing the wrong code, or “switching to a busy task”, whatever that is. If you’re getting one of these, your program has probably gotten pretty out of control.

    GP’s don’t catch every bug, not by a long shot. But they can catch certain thorny problems way before they get out of hand. The worst kinds of bugs to find are intermittent with inconsistent symptoms: you can’t make them happen when you want to, and it’s always something different when they go wrong. Or this variation: you make what should be an irrelevant modification, perhaps to add some debugging or probing code, and the symptoms change. This is exactly what can happen from random unintentional memory reads or writes, and this is exactly the kind of thing that a GP is likely to catch in the act, with a bull’s-eye on the instruction that made the memory reference. With some patience, you can sift through the clues at the scene of the crime and find out exactly what went wrong. This chapter is all about helping you get started with GP detective work.

    Kinds of GPs

    Illegal memory accesses can happen in hundreds of unusual ways, some of which may even be completely harmless in a program running in real mode. In protected mode, however, they cause a GP fault. Here are three common programming errors that can result in GP’s:

    ...

    If your control string has more %s place-holders in it than there are pointers in your argument list, the machine will find itself picking up random stack contents as a pointer, and accessing memory there. Actually, the mere loading of a random stack value into a selector register is likely to generate a GP right off. If not, then accessing memory through a random offset is likely to exceed the length bounds of the selected segment.

    Sysop GP Handling

    When a Worldgroup Major BBS Sysop wants maximum up-time for his server, he sets CNF option GPHDLR in Hardware Setup to YES:

    ...

    This also helps with hacker protection if a user discovers a way to generate an otherwise harmless GP: with GPHDLR set to YES, a GP simply logs him off.

    Developer GP Handling

    But developers should use GP’s on their demo or support Worldgroup the Major BBS server to maximum advantage for tracking down bugs.

    ...

    In this mode, extensive information on the GP is captured in a text file \WGSERV\GP.OUT.

    Reading a GP.OUT Report

    A GP is a software interrupt 13. You could also get other software interrupts for other disastrous conditions like interrupt 12 for a stack fault. One possible cause of interrupt 12 is an infinitely recursive routine. But only a GP can provide you with a lengthy report.

    ...

            WORLDGROUP GP @ 2d4f:031a EC 0000 (recorded 12/16/94 17:11:17)
            GP'ed between 2d4f:0000 _INIT__TELECON and 0000:0000 Unknown
            AX=0000 BX=0000 CX=0000 DX=0787 SI=0005 DI=0054 BP=2a88 ES=0000 DS=2d57 Flags=3246
            Current CS:IP==>26 83 0f 01 b8 f7 04 8e c0 26 c4 1e 87 08 26 c7
    
            YOUR SYSTEM NAME HERE
            Server Version: 1.00
            User 3  channel 00  User-ID "Sysop"  status 3
            Online level 6, state 5, substate 0
            MSG:galtlc.mcv/-1
            Module "Teleconference"
            Input "T"
    
            0167=C:\WGSERV\MAJORBBS.EXE
            0717=C:\WGSERV\GALGSBL.DLL
            073f=C:\WGSERV\BBSBTU.DLL
            075f=C:\WGSERV\DOSCALLS
            2a5f=C:\WGSERV\GALMSG.DLL
            2cff=C:\WGSERV\GALRSY.DLL
            2d3f=C:\WGSERV\GALTLC.DLL
            2d6f=C:\WGSERV\GALTXV.DLL
            2d97=C:\WGSERV\GALUIE.DLL
            2dbf=C:\WGSERV\GALXIT.DLL
            2de7=C:\WGSERV\GALTST.DLL
            Stack:
            0707:2a78  04ff  0746  04ff  0054  2f47  04ff  0054  0005               0707:2a88 >2aaa< 0ba3  0197  04ff  0000  098a  0197  04f7               0707:2a98  0000  4f54  0050  2aaa  0029  039f  0074  0065               0707:2aa8  04f7 >2ac0< 529a  018f  0039  28ef  0001  0000               0707:2ab8  04f7  1b32  0003  0000 >2ace< 3579  018f  04f7               0707:2ac8  000e  4f84  018f >2b30< 3099  018f  04f7  022f               0707:2ad8  018f  0707  1b14  0000  0000  0000  0000  0000
            0707:2ae8  0000   0000   0000   011f   96c7  0017   0006  0000          0707:2af8  0707   0000   011f   966b   0017  0e29   0000  0e29          0707:2b08  0000   2b36   0707   0707   0001  1330   26cc  0147          0707:2b18  1b32   0707   1b32   1b14   2b36  4c48   016f  0578          0707:2b28  0707   0587   0707   000f  >0000< 014f   016f  0000          0707:2b38  0000   0000   fdcc   077f  ----  ----  ----  ---             0707:2b48  ----  ----  ----  ----  ----  ----  ----  ---                        0707:2b58  ----  ----  ----  ----  ----  ----  ----  ---                0707:2b68  ----  ----  ----  ----  ----  ----  ----  ---                0707:2b78  ----  ----  ----  ----  ----  ----  ----  ---                0707:2b88  ----  ----  ----  ----  ----  ----  ----  ---                        Routines:
            0197:07c5 _GOPAGE         <     0197:0ba3       <   0197:0e3d _DSPMNU
            018f:4e23 _NXTLOF         <     018f:529a       <   018f:53be _CURUSR           018f:3487 _HDLINP         <     018f:3579       <   018f:35e4 _CLRXRF           018f:2502 _KILETC         <     018f:3099       <   018f:3145 _CONDEX           0000:0000 Unknown         <     016f:014f       <   016f:0155 __CLEANUP
    GP Location

    The first line of GP.OUT has the CPU instruction pointer CS:IP (or CS:EIP depending on the CPU) contents, which reflect the location (address) of the fault. Usually this points to an instruction that the CPU refused to execute for some reason. This line also tells you when the GP occurred. If you see GPCNT here, then the Sysop has GPHDLR set to YES and the GP reported is not the first (and perhaps not the most informative — have them set GPHDLR to NO and get you another GP.OUT).

    GP Vicinity

    The second line of GP.OUT attempts to place the CS:IP between two public symbols that you can search for in your source files. Only symbols declared EXPORT are included here though. The only EXPORT routine in your whole .DLL is probably your init__xxx() routine (page 42). If the GP occurred in a source file with no EXPORT routines, you’re likely to see some irrelevant addresses. You can still get an idea what file the GP is in from the .DLL list. More on that below.

    For a reproducible GP (one you can make happen any time you want), you might declare some routines EXPORT just to get more information here. Often, narrowing the problem down to one routine is enough to start studying the routine and to find the problem.

    Registers

    This section shows the contents of the assembly language registers right when the GP occurred.

    Code at the GP Location

    Here on the fifth line of GP.OUT are 16 bytes of executable code starting from the GP location. This includes the actual instruction that caused the GP. This object code can come in handy when searching for the exact locations in an assembly listing (more below).

    User Conditions

    The next section of GP.OUT shows information on the most recent server activity on a user’s channel.

    Server Version: 1.00 Worldgroup the Major BBS server software version

    User 3 User number (usrnum, see page 56)

    ...

    To put more debugging information into GP.OUT, you can carefully add some code to the lower part of appgprept() in PLBBS.C. You’ll see the fprintf() calls that formatted all of this information, along with careful checking in case pointers are not usable. Using a bad pointer inside the GP report routines is like accidentally exploding a bomb at the crime scene — you’ll lose the evidence from the original crime due to the new one on your hands.

    .DLL Selectors

    This is a list of the starting selectors for each of the .DLL’s you have loaded on the server. You can find out in which .DLL the GP occurred by finding the highest .DLL starting selector that is less than the GP selector.

    ...

    Once you get a feel for these things you’ll know that 031a should be pretty near the beginning of a file as big as MJRTLC.C.

    Stack Dump

    The stack dump is very relevant to “the crime” of the GP, and it’s rich with data — too rich. Let’s jump ahead to the Routines section with the return address chain at the bottom. That’s where the most meaningful information is pulled from the stack dump and given symbols from the source code where possible. We’ll come back to the stack dump later.

    Routines in the Return Address Chain

    At the bottom of GP.OUT are the chain of return addresses pulled from the stack. These reflect the nesting of subroutines that the CPU was executing at the moment of the GP. The first one in the list, 0197:0ba3, is pulled from the stack at 0707:2A8A or at SS:BP+2. If you know how C language subroutines are implemented in assembly language, it’s not hard to realize that this is the return address to some ancestor routine that called (perhaps indirectly) the routine in MJRTLC.C where the GP was triggered.

    ...

    The next ancestor of gomodl() shown in the return address chain is between nxtlof() and curusr(). This has a different selector than gomodl() (018f versus 0197) so it must come from a different file. In fact it comes from MAJORBBS.C. You’ll notice that there’s no call to gomodl() in all of MAJORBBS.C, but there’s one in MENUING.C inside of gopage(). Hmm, this is making sense. There are reams of calls to gopage() in MAJORBBS.C for menu tree pages in general. And gopage() calls gomodl() for module pages in particular. gopage() was skipped in the return address chain. That’s probably because gomodl(), after various compiler optimizations and shenanigans, didn’t need to save and reload BP.

    Stack Dump, Revisited

    The notations like >2aaa< in the stack dump are BP save locations. When traversing the chain of saved and loaded BPs to generate the return address chain at the bottom of GP.OUT, the BP save locations are flagged with little "> <" symbols. The return addresses shown at the bottom immediately follow each of the "> <" symbols. For example, the >2aaa< is followed by 0ba3 and 0197. 0197:0ba3 is the first return address in the list.

    Close scrutiny of the stack dump can help us determine the values of parameters passed between routines, and the values of automatic stack variables. In some cases, the values of pushed registers are also of some help in investigating the circumstances of the crime. But to do any of this, we have to look at the assembly language listing of the code where the GP happened and where the ancestor routines did their subroutine calling. And an assembly language listing lets us find out the exact instruction that caused the GP. Let’s start with the routine where the GP occurred, probably telecn(), definitely in MJRTLC.C.

    Assembly Listing

    To get the assembly language listing of MJRTLC.C, we have to generate assembly source code and then assemble it and make a listing:

    ...