Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: Out of Memory

  1. #1
    Join Date
    Feb 2009
    Location
    UK
    Posts
    826

    Default Out of Memory

    Hi

    Not played with V20 yet, but may have to put it into production now!

    Processing xml files via Chilkat in 19.1 (so 32 bit) and getting out of memory.

    LoadXmlFile:
    ChilkatVersion: 9.5.0.78
    path: G:\Dev\DF_191\Pat_Val_Sales\TransDat\MDR_Data\MCR-GBP-101-SalesData-00000001-201810271010591-src.xml
    loadFileX:
    Out of memory
    fileSize: 457079828
    --loadFileX
    Failed.
    --LoadXmlFile
    --ChilkatLog

    Should switching to 64 bit resolve this? (Biggest file that needs to be processed is 986,052,366 bytes.)

    Any advice would be welcome.
    Ian Smith
    (Member of the SigCj project)

  2. #2
    Join Date
    Feb 2009
    Location
    Stuart, FL
    Posts
    5,321

    Default Re: Out of Memory

    do you get this all the time with this call?
    Michael Salzlechner
    StarZen Technologies, Inc
    http.://www.starzen.com

    IT Director at Balloons Everywhere

    Development Blog
    http://www.salzlechner.com/dev

    DataFlex Package Manager (aka Nuget for DataFlex)
    http://windowsdeveloper.com/dfPackage

  3. #3
    Join Date
    Feb 2009
    Location
    Stuart, FL
    Posts
    5,321

    Default Re: Out of Memory

    reason i am asking is to determine if this is a general issue in chilkat or maybe an issue in your program of compounding memory/resource loss

    as far as 64bit and fixing this

    if it is a compounding memory loss issue because things are not freed properly it may delay it a bit. if it is a limitation in chilkat it depends on their 64bit code and limitations
    Michael Salzlechner
    StarZen Technologies, Inc
    http.://www.starzen.com

    IT Director at Balloons Everywhere

    Development Blog
    http://www.salzlechner.com/dev

    DataFlex Package Manager (aka Nuget for DataFlex)
    http://windowsdeveloper.com/dfPackage

  4. #4
    Join Date
    Feb 2009
    Posts
    5,468

    Default Re: Out of Memory

    What is your memory footprint in Task Manager ?

    Your application, resources, dlls and all the memory they use all share the same 32bit address space

    457mb does not seem that big but maybe the MSXML6 or whatever the underlying Windows XML parser is called these days expands it out (like a JPEG expands into memory bigger than its size on disk)

    Is there any reason your using Chilkat over FlexML ?

    If you need more than 2 gb in total then yes you will have to go to 64 bit
    Success consists of going from failure to failure without loss of enthusiasm - Winston Churchill

  5. #5
    Join Date
    Feb 2009
    Location
    UK
    Posts
    826

    Default Re: Out of Memory

    Smaller files process Ok

    App currently using approx 196MB, 11GB free on server, SQL Express has 1.8GB allocated.

    Looks like a Chilkat limitation, I have emailed Chilkat support what they expct the max file sizes to be.

    Just had a response form Chilkat

    Remember.. Chilkat XML is a DOM parser (not a SAX parser). It's building an in-memory document object model. DOM parsers are not memory efficient for huge XML files with millions of nodes/elements/attributes. I don't think you'll be able to use Chilkat XML with a file that large. You'll need to find a SAX based parsing solution, or just write custom string parsing code directly..
    Ian Smith
    (Member of the SigCj project)

  6. #6
    Join Date
    Feb 2009
    Posts
    5,468

    Default Re: Out of Memory

    I think FlexML will have the same problem. I remember having a forum conversation with Sonny a very long time ago and he did say at the time they were not going to write their own SAX parser
    Success consists of going from failure to failure without loss of enthusiasm - Winston Churchill

  7. #7
    Join Date
    Feb 2009
    Location
    UK
    Posts
    826

    Default Re: Out of Memory

    Is there any reason your using Chilkat over FlexML ?
    No, using Chilkat to unzip the archives, so used Chilkat to process the XML as well.

    Do you know if FlexML uses DOM? If so then the same limiations will probably apply.

    The XML is flat, all the data items are attributes of the node. No data in the node and no child elements.
    Ian Smith
    (Member of the SigCj project)

  8. #8
    Join Date
    Feb 2009
    Location
    Hengelo, Netherlands
    Posts
    10,869

    Default Re: Out of Memory

    Ian,

    FleXML is a wrapper around MSXML which is a DOM parser.
    Regards,
    Data Access Worldwide
    Vincent Oorsprong

  9. #9
    Join Date
    Feb 2009
    Location
    Brazil
    Posts
    5,446

    Default Re: Out of Memory

    by the way .

    Is FleXML somehow updated to support unicode xmls (generating and reading) on 2020 preview ?

    No more OEM/ANSI conversion under the covers ?

    If yes, I guess it will be unicode by default ? and if one still want to generate a ANSI xml, what should be done ?

    Regards
    Samuel Pizarro

  10. #10
    Join Date
    Feb 2009
    Location
    UK
    Posts
    826

    Default Re: Out of Memory

    Hi Vincent

    Thanks for confirming.

    FYI Wrote a class to handle this via standard seqential IO.
    Ian Smith
    (Member of the SigCj project)

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •