XML Editor
Sign up for a WebBoard account Sign Up Keyword Search Search More Options... Options
Chat Rooms Chat Help Help News News Log in to WebBoard Log in Not Logged in
Show tree view Topic
Topic Page 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Go to previous topicPrev TopicGo to next topicNext Topic
Postnext
Ramez GhazzaouiSubject: StylusDiff.exe crashes when comparing two huge XML files
Author: Ramez Ghazzaoui
Date: 22 Dec 2008 05:01 PM
Hi,
I am comparing two XML files of size 276 Mb each. (Actually, for my initial test, I am comparing the same file to itself). The program runs for a couple of minutes on the Windows Command Line then it gives me the error:

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

If I load it into the MS .Net debugger it says:

'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\StylusDiff.exe'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\ntdll.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\kernel32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\wxvault.dll', Binary was not built with debug information.
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\psapi.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\mpr.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\advapi32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\rpcrt4.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\secur32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\user32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\gdi32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\version.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\shlwapi.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\msvcrt.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\detoured.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\shell32.dll'
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\StCmn.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\shfolder.dll'
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\icuuc40.dll', Binary was not built with debug information.
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\icudt40.dll', Binary was not built with debug information.
'StylusDiff.exe': Loaded 'C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.1433_x-ww_5cf844d2\msvcr80.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\wininet.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\crypt32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\msasn1.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\oleaut32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\ole32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\iphlpapi.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\ws2_32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\ws2help.dll'
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\xmlengine.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\WinSxS\x86_Microsoft.VC80.MFC_1fc8b3b9a1e18e3b_8.0.50727.762_x-ww_3bf8fa05\mfc80u.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.5512_x-ww_35d4ce83\comctl32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.1433_x-ww_5cf844d2\msvcp80.dll'
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\RWUXThemeSU80.dll'
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\sfl500asu.dll', Binary was not built with debug information.
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\ot1100asu.dll', Binary was not built with debug information.
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\vc8-re200l.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\imm32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\lpk.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\usp10.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\comctl32.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\WinSxS\x86_Microsoft.VC80.MFCLOC_1fc8b3b9a1e18e3b_8.0.50727.762_x-ww_91481303\mfc80ENU.dll', Binary was not built with debug information.
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\Plugins\XMLDiff.dll'
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\StXmlCmn.dll'
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\StylusHlp.dll'
'StylusDiff.exe': Loaded 'C:\Program Files\Stylus Studio 2009 XML Enterprise Suite\bin\SmartEdit2.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\msctf.dll'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\msctfime.ime'
'StylusDiff.exe': Loaded 'C:\WINDOWS\system32\apphelp.dll'
The thread 'Win32 Thread' (0xf10) has exited with code 0 (0x0).
The program '[5524] StylusDiff.exe: Native' has exited with code 3 (0x3).


Is this a scalability limitation in the program? My next test was going to involve XML files 544 Mg large...

-Ramez

Postnext
Tony LavinioSubject: StylusDiff.exe crashes when comparing two huge XML files
Author: Tony Lavinio
Date: 23 Dec 2008 03:11 AM
You're probably straining the limits of a 32bit address space now.
You could try the -fast option to see if you can eke out a little
bit more.

Remember that comparing two 250meg+ files means 1/2gig of RAM just
to load the raw text; building the list of pointers for cross-referencing
them also takes space.

Why do you need to diff files? Is there something specific you are
looking for?

Postnext
Ramez GhazzaouiSubject: StylusDiff.exe crashes when comparing two huge XML files
Author: Ramez Ghazzaoui
Date: 23 Dec 2008 01:47 PM
Thank you for the response.

The question as to why I need to compare such large XML files is one that I cannot answer. That's the size of the files I am given for comparing. I wish I could split them, but that would assume that there is a clean middle-point where I know the top halves compare to each other and the bottom halves compare to each other. In reality, the need for a tree-based comparison arises from the fact that the XML elements aren't ordered in the same fashion across the two files.

About stretching the limits of memory, I really throught that since Windows 2000 there has been no limit to memory because Windows allocates all the swap file virtual memory that it needs, with no limit other than the size of your hard disk (or the swap file file size, if such size is specified in Control Panel).

Should I conclude from your answer that there is no solution to my problem?

-Ramez

Postnext
Tony LavinioSubject: StylusDiff.exe crashes when comparing two huge XML files
Author: Tony Lavinio
Date: 23 Dec 2008 02:34 PM
Windows 2000 is a 32bit operating system. That means that the memory
for any given process must be addressable by a 32bit number. The
operating system takes 1/2 for itself, leaving 2gig at most for
applications (there are ways to trick some versions of Windows to give
3gb, but that's out of scope for this discussion).

Other programs running, and operating system utilities, take up some
of that space also, and of course Stylus Studio needs some.

And XML files, when they are rendered in memory, have some overhead.

There are a few XML diff products which use different architectures to
get around some of these limitations; you might try DeltaXML.

Assuming you're not comparing two randomly-selected XML files, there
are usually ways to determine what sorts of changes you are looking
for, and then strategies for diffing them can be made. But for general
diffing, that's a hard problem.

Posttop
Ramez GhazzaouiSubject: StylusDiff.exe crashes when comparing two huge XML files
Author: Ramez Ghazzaoui
Date: 23 Dec 2008 02:46 PM
Yes, I guess there is no way around the memory address spage limit. I wonder how long before the need for more than 64 bits arises...

The XML files that I am comparing aren't random, in fact for my test here I am comparing one file to itself. But my actual need is for comparing two XML files that are supposed to have the same elements except their order is mixed.

I'll try -fast but at tis point in time I am discouraged. I wish it were possible to lead the trees level by level, pass by pass. I'm not sure how such an algorithm would work but it might be worth considering.

Cheers, and happy [your favorite holiday].

-Ramez

 
Topic Page 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Go to previous topicPrev TopicGo to next topicNext Topic
Download A Free Trial of Stylus Studio 6 XML Professional Edition Today! Powered by Stylus Studio, the world's leading XML IDE for XML, XSLT, XQuery, XML Schema, DTD, XPath, WSDL, XHTML, SQL/XML, and XML Mapping!  
go

Log In Options

Site Map | Privacy Policy | Terms of Use | Trademarks
Stylus Scoop XML Newsletter:
W3C Member
Stylus Studio® and DataDirect XQuery ™are from DataDirect Technologies, is a registered trademark of Progress Software Corporation, in the U.S. and other countries. © 2004-2016 All Rights Reserved.