Reliability
Debug Versions
Applies to: v10
Debugging a v10 add-on: to debug in Release mode, start the BBS, then in Visual Studio choose Debug, Attach to Process, and select
wgserver.exe(MBBS10.txtsection 3). Some bugs appear only in Release because, according toMBBS10.txt, Debug builds zero out pointers (so garbage memory does not always show up), and likewise for string-literal misuse (chapter 29, section 29.4). The Debug-pointer-zeroing statement is the author's observation: unverified.
...
#ifdef DEBUG
for (i=0 ; i < MAXLST ; i++) {
if (list[i] == NULL) {
catastro("NULL list entry!");
}
}
#endifAssertions
An easy to use debugging tool is the assertion. Assertions are failures triggered by the ASSERT() macro. ASSERT() is used to assert that a condition is always true. When DEBUG is not defined, ASSERT() calls are not evaluated. When DEBUG is defined, ASSERT() will fail if the passed condition is not true. In the failure message will be the filename, file version, line number, and specific condition that failed. By default, assertion failures take the form of catastro() crashes. But, assertions can be altered to fail in different ways, as described below.
...
In this example, the debug version will increment i, the non-debug version won’t. The following code may rely on i being incremented, and work fine as long as you’re running the debug version. Once you build final, non-debug shipping disks, you’ll be surprised to find a new bug. Bugs caused by debugging code are often the most painful.
Phase Transitions
Another helpful method of debugging involves using phase transitions. Phase transitions are used to track the basic flow of code by recording major events. The BEG_PHASE() and END_PHASE() macros are used to record the beginning and end of phases or events. The last x phase transitions are reported in catastro() and GP crashes to help understand what led to the crash. BEG_PHASE() and END_PHASE() both take a string and a long for a brief description of the phase and any relevant data. Generally, they are used to bracket some well-defined set of code:
...
In the above case, the address of the global command just executed is included as the extra parameter to END_PHASE(). This will report the address of the global command just executed along with the Global command end transition message.
Debug Options in CNF
If you create any debug-only CNF options, put them in CNF’s level 95. That’s the level where all debugging options go by default. A Sysop can edit those options by running CNF directly from DOS with a special command-line parameter:
CNF -L95The Baseline's Debugging Module
The Debugging in baseline module is installed automatically as part of the Client/Server Developer’s Kit. By default, it’s disabled. To enable it, go to Basic Utilities (choice 7 on the Introductory Menu), run WGSDMOD (the Enable/Disable Modules utility), and move Debugging in baseline from the Expressly Disabled column to the Available column.
...
Check out CNF level 95 once you’ve enabled the debugging module. All of its features can be turned on or off from there. The available options are briefly described in the option help messages. Two of the most notable are:
the ability to debug memory allocations in order to detect memory under/overflow bugs; andthe ability to alter assertion failures to be quietly ignored or logged to disk rather than catastro(). Some Philosophy on Debugging
Your best debugging work takes place long before you test-run any of your software. A programming task that’s well thought through and meticulously coded has the best chance of long term success.
...
You’re probably trying to increase your programming speed just like we are. Nothing will help your overall productivity more than an ingrained habit of completing highly reliable foundations before you proceed to use them. There are many techniques for making your foundations highly reliable: code walk-throughs, rigorous testing by the developer, line-by-line testing of source code, branch-by-branch testing, routinely retesting “repaired” code, alpha-testing (in-house), beta-testing (by motivated customers). The goal is to get software done fast, put it to use, and move on to other projects, without your customers having to find the problems and you solving them over the phone.
Programming Tips for
...
the Major BBS
There are several unique things about the programming environment of Worldgroupthe Major BBS. The multi-user single-tasking aspects mean that your code can’t wait very long for any one thing to happen.
Once you become proficient in programming for Worldgroupthe Major BBS, there are a few classes of problems you’ll want to be on guard against. If you review these problems and develop an eagle eye for them, then your code can be much more reliable and free from frustrating bugs.
...
| Wrong way | Right way |
| prfmsg(GREETING); | prfmsg(GREETING); |
| if (problem) { | if (problem) { |
| prfmsg(WARNING); | prfmsg(WARNING); |
| outprf(usrnum); | } |
| } | outprf(usrnum); |
| outprf(usrnum); |
General Protection Faults
The best thing and the worst thing about protected mode operation is the “General Protection” fault, or GP. Protected mode is designed to halt at the slightest sign of a program gone awry. The first time you try to unravel and debug a GP will be an adventure. But after all the work of tracking it down, it will probably lead you directly to a bug in your code.
What is a GP?
A General Protection fault is the computer CPU’s way of saying “I give up” or “I can’t do this”. Usually it has to do with an invalid or restricted memory address. There are other GP causes, like privilege violation, executing the wrong code, or “switching to a busy task”, whatever that is. If you’re getting one of these, your program has probably gotten pretty out of control.
GP’s don’t catch every bug, not by a long shot. But they can catch certain thorny problems way before they get out of hand. The worst kinds of bugs to find are intermittent with inconsistent symptoms: you can’t make them happen when you want to, and it’s always something different when they go wrong. Or this variation: you make what should be an irrelevant modification, perhaps to add some debugging or probing code, and the symptoms change. This is exactly what can happen from random unintentional memory reads or writes, and this is exactly the kind of thing that a GP is likely to catch in the act, with a bull’s-eye on the instruction that made the memory reference. With some patience, you can sift through the clues at the scene of the crime and find out exactly what went wrong. This chapter is all about helping you get started with GP detective work.
Kinds of GPs
Illegal memory accesses can happen in hundreds of unusual ways, some of which may even be completely harmless in a program running in real mode. In protected mode, however, they cause a GP fault. Here are three common programming errors that can result in GP’s:
...
If your control string has more %s place-holders in it than there are pointers in your argument list, the machine will find itself picking up random stack contents as a pointer, and accessing memory there. Actually, the mere loading of a random stack value into a selector register is likely to generate a GP right off. If not, then accessing memory through a random offset is likely to exceed the length bounds of the selected segment.
Sysop GP Handling
When a Worldgroup Major BBS Sysop wants maximum up-time for his server, he sets CNF option GPHDLR in Hardware Setup to YES:
...
This also helps with hacker protection if a user discovers a way to generate an otherwise harmless GP: with GPHDLR set to YES, a GP simply logs him off.
Developer GP Handling
But developers should use GP’s on their demo or support Worldgroup the Major BBS server to maximum advantage for tracking down bugs.
...
In this mode, extensive information on the GP is captured in a text file \WGSERV\GP.OUT.
Reading a GP.OUT Report
A GP is a software interrupt 13. You could also get other software interrupts for other disastrous conditions like interrupt 12 for a stack fault. One possible cause of interrupt 12 is an infinitely recursive routine. But only a GP can provide you with a lengthy report.
...
WORLDGROUP GP @ 2d4f:031a EC 0000 (recorded 12/16/94 17:11:17)
GP'ed between 2d4f:0000 _INIT__TELECON and 0000:0000 Unknown
AX=0000 BX=0000 CX=0000 DX=0787 SI=0005 DI=0054 BP=2a88 ES=0000 DS=2d57 Flags=3246
Current CS:IP==>26 83 0f 01 b8 f7 04 8e c0 26 c4 1e 87 08 26 c7
YOUR SYSTEM NAME HERE
Server Version: 1.00
User 3 channel 00 User-ID "Sysop" status 3
Online level 6, state 5, substate 0
MSG:galtlc.mcv/-1
Module "Teleconference"
Input "T"
0167=C:\WGSERV\MAJORBBS.EXE
0717=C:\WGSERV\GALGSBL.DLL
073f=C:\WGSERV\BBSBTU.DLL
075f=C:\WGSERV\DOSCALLS
2a5f=C:\WGSERV\GALMSG.DLL
2cff=C:\WGSERV\GALRSY.DLL
2d3f=C:\WGSERV\GALTLC.DLL
2d6f=C:\WGSERV\GALTXV.DLL
2d97=C:\WGSERV\GALUIE.DLL
2dbf=C:\WGSERV\GALXIT.DLL
2de7=C:\WGSERV\GALTST.DLL
Stack:
0707:2a78 04ff 0746 04ff 0054 2f47 04ff 0054 0005 0707:2a88 >2aaa< 0ba3 0197 04ff 0000 098a 0197 04f7 0707:2a98 0000 4f54 0050 2aaa 0029 039f 0074 0065 0707:2aa8 04f7 >2ac0< 529a 018f 0039 28ef 0001 0000 0707:2ab8 04f7 1b32 0003 0000 >2ace< 3579 018f 04f7 0707:2ac8 000e 4f84 018f >2b30< 3099 018f 04f7 022f 0707:2ad8 018f 0707 1b14 0000 0000 0000 0000 0000
0707:2ae8 0000 0000 0000 011f 96c7 0017 0006 0000 0707:2af8 0707 0000 011f 966b 0017 0e29 0000 0e29 0707:2b08 0000 2b36 0707 0707 0001 1330 26cc 0147 0707:2b18 1b32 0707 1b32 1b14 2b36 4c48 016f 0578 0707:2b28 0707 0587 0707 000f >0000< 014f 016f 0000 0707:2b38 0000 0000 fdcc 077f ---- ---- ---- --- 0707:2b48 ---- ---- ---- ---- ---- ---- ---- --- 0707:2b58 ---- ---- ---- ---- ---- ---- ---- --- 0707:2b68 ---- ---- ---- ---- ---- ---- ---- --- 0707:2b78 ---- ---- ---- ---- ---- ---- ---- --- 0707:2b88 ---- ---- ---- ---- ---- ---- ---- --- Routines:
0197:07c5 _GOPAGE < 0197:0ba3 < 0197:0e3d _DSPMNU
018f:4e23 _NXTLOF < 018f:529a < 018f:53be _CURUSR 018f:3487 _HDLINP < 018f:3579 < 018f:35e4 _CLRXRF 018f:2502 _KILETC < 018f:3099 < 018f:3145 _CONDEX 0000:0000 Unknown < 016f:014f < 016f:0155 __CLEANUPGP Location
The first line of GP.OUT has the CPU instruction pointer CS:IP (or CS:EIP depending on the CPU) contents, which reflect the location (address) of the fault. Usually this points to an instruction that the CPU refused to execute for some reason. This line also tells you when the GP occurred. If you see GPCNT here, then the Sysop has GPHDLR set to YES and the GP reported is not the first (and perhaps not the most informative — have them set GPHDLR to NO and get you another GP.OUT).
GP Vicinity
The second line of GP.OUT attempts to place the CS:IP between two public symbols that you can search for in your source files. Only symbols declared EXPORT are included here though. The only EXPORT routine in your whole .DLL is probably your init__xxx() routine (page 42). If the GP occurred in a source file with no EXPORT routines, you’re likely to see some irrelevant addresses. You can still get an idea what file the GP is in from the .DLL list. More on that below.
For a reproducible GP (one you can make happen any time you want), you might declare some routines EXPORT just to get more information here. Often, narrowing the problem down to one routine is enough to start studying the routine and to find the problem.
Registers
This section shows the contents of the assembly language registers right when the GP occurred.
Code at the GP Location
Here on the fifth line of GP.OUT are 16 bytes of executable code starting from the GP location. This includes the actual instruction that caused the GP. This object code can come in handy when searching for the exact locations in an assembly listing (more below).
User Conditions
The next section of GP.OUT shows information on the most recent server activity on a user’s channel.
Server Version: 1.00 Worldgroup the Major BBS server software version
User 3 User number (usrnum, see page 56)
...
To put more debugging information into GP.OUT, you can carefully add some code to the lower part of appgprept() in PLBBS.C. You’ll see the fprintf() calls that formatted all of this information, along with careful checking in case pointers are not usable. Using a bad pointer inside the GP report routines is like accidentally exploding a bomb at the crime scene — you’ll lose the evidence from the original crime due to the new one on your hands.
.DLL Selectors
This is a list of the starting selectors for each of the .DLL’s you have loaded on the server. You can find out in which .DLL the GP occurred by finding the highest .DLL starting selector that is less than the GP selector.
...
Once you get a feel for these things you’ll know that 031a should be pretty near the beginning of a file as big as MJRTLC.C.
Stack Dump
The stack dump is very relevant to “the crime” of the GP, and it’s rich with data — too rich. Let’s jump ahead to the Routines section with the return address chain at the bottom. That’s where the most meaningful information is pulled from the stack dump and given symbols from the source code where possible. We’ll come back to the stack dump later.
Routines in the Return Address Chain
At the bottom of GP.OUT are the chain of return addresses pulled from the stack. These reflect the nesting of subroutines that the CPU was executing at the moment of the GP. The first one in the list, 0197:0ba3, is pulled from the stack at 0707:2A8A or at SS:BP+2. If you know how C language subroutines are implemented in assembly language, it’s not hard to realize that this is the return address to some ancestor routine that called (perhaps indirectly) the routine in MJRTLC.C where the GP was triggered.
...
The next ancestor of gomodl() shown in the return address chain is between nxtlof() and curusr(). This has a different selector than gomodl() (018f versus 0197) so it must come from a different file. In fact it comes from MAJORBBS.C. You’ll notice that there’s no call to gomodl() in all of MAJORBBS.C, but there’s one in MENUING.C inside of gopage(). Hmm, this is making sense. There are reams of calls to gopage() in MAJORBBS.C for menu tree pages in general. And gopage() calls gomodl() for module pages in particular. gopage() was skipped in the return address chain. That’s probably because gomodl(), after various compiler optimizations and shenanigans, didn’t need to save and reload BP.
Stack Dump, Revisited
The notations like >2aaa< in the stack dump are BP save locations. When traversing the chain of saved and loaded BPs to generate the return address chain at the bottom of GP.OUT, the BP save locations are flagged with little "> <" symbols. The return addresses shown at the bottom immediately follow each of the "> <" symbols. For example, the >2aaa< is followed by 0ba3 and 0197. 0197:0ba3 is the first return address in the list.
Close scrutiny of the stack dump can help us determine the values of parameters passed between routines, and the values of automatic stack variables. In some cases, the values of pushed registers are also of some help in investigating the circumstances of the crime. But to do any of this, we have to look at the assembly language listing of the code where the GP happened and where the ancestor routines did their subroutine calling. And an assembly language listing lets us find out the exact instruction that caused the GP. Let’s start with the routine where the GP occurred, probably telecn(), definitely in MJRTLC.C.
Assembly Listing
To get the assembly language listing of MJRTLC.C, we have to generate assembly source code and then assemble it and make a listing:
...