Subject: FYI: Access violation in IDPDOX32.DLL Date: Mon, 17 Mar 1997 11:39:36 +0100 From: Hallvard Vassbotn Organization: Falcon AS To: 76711.666@compuserve.com, lisch@tempest-sw.com, Ralf.Quint@bon.netsurf.de, r.p.sterkenburg@dataweb.nl, 70630.717@compuserve.com Bug in : BDE Paradox driver BDE versions : 3.12, 3.5 and 4.0 (from Delphi 3.0 beta) Also in : Paradox Engine for Windows, Paradox Engine for DOS Description : Access violation in IDPDOX32.DLL after adding about 100 MB to Blob file! Found by : Hallvard Vassbotn, hallvard@falcon.no, Reuters Norge AS Severity : Extremly serious Background: As the research and development department of Reuters in the Nordic contries, we develop real-time and historical data systems for the financial markets. One of the key components in this is our historical database of news items, known as HistNews. HistNews stores all stories from all news agencies covering the nordical financial markets. It has news stories from 1989 until today and consists currently of about 400.000 news items and about 800 new items are added every day. These HistNews databases are replicated at each client site (about 300) and are stored using the Paradox 4.0 database format. The size of the database is about 385 MB (84 MB for the DB file, 300 MB for the Blob file and 1 MB for the primary index). The program that updates the Paradox tables run each night and is a Dos-based protected mode program written using BP 7.0 and Paradox Engine for DOS 3.01. Lately we have had problems in that the updating program crash with a GP from time to time inside the Paradox Engine library. This is a major headache as we must travel to the client site to replace his faulty tables. After spending days trying to re-create the problems in-house we finally came to a sceniaro where we could constantly recreate the problem. This happened on a local disk with no other potential conflicts with other applications or users. The sceniaro is to let the updating program create a new table on the local disk, then run in a loop in a batch file updating the same set of news over and over again. The only difference is that we set the date for wich the news should be added. This way we simulated adding one years worth of data in a few hours. Each day was updated using the exact same datafile and it added all the days just fine until the middle of desember of the simulated year. Then the blob file was about 130 MB and the program crashed with a GP giving an address inside the PXISAM1 module of Paradox Engine for Dos 3.01 (pmode). The next step was to actually step through the assembly code of Paradox Engine using the Turbo Debugger. This quickly revealed that the crash happened inside a call to the PXBlobOpenWrite function. The dump from Turbo Debugger around the crashing instruction looked like this: Turbo Debugger Log PROTECTED CPU 80486 0B57:128E 45 inc bp 0B57:128F 55 push bp 0B57:1290 8BEC mov bp,sp 0B57:1292 83EC04 sub sp,0004 0B57:1295 8B4604 mov ax,[bp+04] 0B57:1298 BA0900 mov dx,0009 0B57:129B F7EA imul dx 0B57:129D 8BD8 mov bx,ax 0B57:129F 8B87AA29 mov ax,[bx+29AA] 0B57:12A3 8B97A829 mov dx,[bx+29A8] 0B57:12A7 8946FE mov [bp-02],ax 0B57:12AA 8956FC mov [bp-04],dx 0B57:12AD C45EFC les bx,[bp-04] 0B57:12B0 26803F01 cmp es:byte ptr [bx],01 { <- GP Here! } 0B57:12B4 750B jne 12C1 0B57:12B6 8B56FE mov dx,[bp-02] 0B57:12B9 8B46FC mov ax,[bp-04] 0B57:12BC 050300 add ax,0003 0B57:12BF EB0D jmp 12CE 0B57:12C1 C45EFC les bx,[bp-04] 0B57:12C4 8B56FE mov dx,[bp-02] 0B57:12C7 8B46FC mov ax,[bp-04] 0B57:12CA 26034709 add ax,es:[bx+09] 0B57:12CE EB00 jmp 12D0 0B57:12D0 8BE5 mov sp,bp 0B57:12D2 5D pop bp 0B57:12D3 4D dec bp 0B57:12D4 C20200 ret 0002 In addition to this simple localization of the problem, we also tried to find out _why_ the crash happened here. We did not find the exact solution, but it seemed to have to do with Paradox Engine checking the wrong flag before using a structure (i.e. it assumes the structure is valid when it is infact not). One routine invalidates one flag, but another routine checks another flag. Knowing that Paradox Engine for DOS is a product that Borland does not longer support, we decided to port the updating program to a 32-bit Delphi console application that uses the BDE32. We reckoned that these bugs in Paradox Engine probably (hopefully) was removed from the Paradox driver in BDE. After spending almost a week converting the updating program to use the TTable component to update the database, we found that it did work according to the specs (i.e. the old 16-bit windows client applications could read the database written by the new 32-bit update program). Then we started the heavy testing again with excactly the same sceniaro and datafiles that had crashed the 16-bit DOS protected mode update program. We found that the new update program ran faster than the old one - but it did still crash after updating about 9 months of news items. Back in the debugger we found that the Access Violation is inside the BDE Paradox driver, IDPDOX32.DLL. The address is 4CE0615D and the assmebly code look like this: Turbo Debugger Log CPU Pentium Thread #215 :4CE06146 C20C00 ret 000C :4CE06149 55 push ebp :4CE0614A 8BEC mov ebp,esp :4CE0614C 0FBF450C movsx eax,word ptr [ebp+0C] :4CE06150 8D14C0 lea edx,[eax+8*eax] :4CE06153 8B4D08 mov ecx,[ebp+08] :4CE06156 8B8411B9000000 mov eax,[ecx+edx+000000B9] :4CE0615D 803801 cmp byte ptr [eax],01 {<- AV Here! } :4CE06160 7505 jne IDPDX32.4CE06167 (4CE06167) :4CE06162 8D5003 lea edx,[eax+03] :4CE06165 EB06 jmp IDPDX32.4CE0616D (4CE0616D) :4CE06167 0FB75009 movzx edx,word ptr [eax+09] :4CE0616B 03D0 add edx,eax :4CE0616D 8BC2 mov eax,edx :4CE0616F 5D pop ebp :4CE06170 C20800 ret 0008 :4CE06173 55 push ebp :4CE06174 8BEC mov ebp,esp :4CE06176 53 push ebx :4CE06177 56 push esi :4CE06178 57 push edi :4CE06179 8B7508 mov esi,[ebp+08] :4CE0617C 8B5D14 mov ebx,[ebp+14] :4CE0617F 66837D1000 cmp word ptr [ebp+10],0000 :4CE06184 7415 je IDPDX32.4CE0619B (4CE0619B) :4CE06186 668B03 mov ax,[ebx] :4CE06189 50 push eax :4CE0618A 56 push esi :4CE0618B E8A7FAFFFF call IDPDX32.4CE05C37 The observant will see that this is exactly the same logic as the 16-bit code above - the only difference is that this is compiled as 32-bit code. The conclusion is that the old bug from the Paradox Engine code is also present in the newest BDE version. It has been a long struggle to get where we are today and my attemts to make a simplified program that exhibits this bug has been in vain. The biggest problem in doing this is that the bug does not surface until you have run the update program for 2-3 hours. Because of this I'm forced to send you the executable version of my update program together with the datafiles it needs for the updating and some batch files to make it run in a loop. I also submit the key units in the program so that you can see how I do it. There also a simple program (PDOXBUG) that I wrote to try to recreate the bug in a small example, but I have not made this version crash. The bug probably only surfaces under certain circumstances that has to do with the changing sizes of the blob fields. So it seems that there is a set of conditions that must be true before this bug is triggered. The Blob file must be of a certain size, the blob fields added must be of a certain size and added in a certain sequence etc. But this could occur in any product that uses BDE to update Paradox tables containing Blob fields. It is very urgent for Reuters that this bug is fixed as soon as possible - we have several hundred clients with databases that could corrupt any day, and they do corrupt at the rate of 3-4 clients every week. If this cannot be resolved in a reasonable amount of time, we will have to consider our relationship with Borland and be forced to look elsewhere for software development products. Sincerly yours Hallvard Vassbotn Software Developer Falcon R&D, Reuters Norge AS hallvard@falcon.no Phone: +47 22 00 57 33 Fax : +47 22 42 50 32 Available upon request: HNTEST.ZIP - The 32-bit update program with datafiles and batch files. Unpack all files to a single directory. Run TTHN.BAT to test updating an entire year. You should have at least 150 MB free diskspace. In our testing this has crashed the BDE32 with a Access Violation at 4Ce0615D trying to read at 00000000. The batch file directs all output to file called HNTEST32.LOG. HNUPDSRC.ZIP - The source code for the updating program. Copyright Falcon AS and Reuters Norge AS. PDOXBUG.ZIP - A simplified application that will update a database with a blob field. Run it from the PDOXLOOP.BAT file to let it run "forever". We have _not_ been able to crash BDE32 with this program. -- Hallvard Vassbotn Falcon R&D Reuters Norge AS Document5, version 05 April 1997 page 4 of 5