Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: OOM Error on 32 Bit before hitting 2GB boundary?

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1

    Default OOM Error on 32 Bit before hitting 2GB boundary?

    Hi,

    I had reported this to Harm before, but as I got no reply... sharing this with the community.
    Perhaps I'm overlooking something silly (wouldn''t be the first time)

    This is on DataFlex 20.1 (latest), 32 bit.

    The following code works on 64 bit and the project now also works fine under 64 bit, so one could argue that this is all hypothetical.
    However I'm wondering why I'm getting a Out Of Memory error (OOM Error) when using this.

    Demo Code:
    Code:
    Use Windows.pkg
    Use Dfpanel.pkg
    Use Batchdd.pkg
    Use File_dlg.pkg
    
    Object oMain is a Panel
      Set Size to 100 300
      Set Label to "Base64 OOM Error demo"
      
      Use seq_chnl.pkg
      Use cCharTranslate.pkg
      
      Object oCharTranslate is a cCharTranslate
      End_Object
      
      Object oBase64 is a cObject
    
        Function BinaryEncodeFileToUChar String sFileSource Returns UChar[]
          UChar[] ucFile
          UChar[] ucEncoded
          
          Boolean bExists
          Integer iChIn 
          
          If (sFileSource <> "") Begin
              Get Seq_New_Channel to iChIn
              If (iChIn <> DF_SEQ_CHANNEL_NOT_AVAILABLE) Begin
             
                Direct_Input channel iChIn ("BINARY:" + sFileSource)      
                  Read_Block channel iChIn ucFile -1
                Close_Input channel iChIn
                Send Seq_Release_Channel iChIn 
                  
                Get Base64EncodeUCharArray of oCharTranslate ucFile to ucEncoded   /// <--- crashing here
              End 
          End
          
          Function_Return ucEncoded
        End_Function 
      End_Object
      
      Object oXmlUpload_bp is a BusinessProcess
        Function UploadFile String sFileName Returns Boolean
          UChar[] ucEncoded
          Get BinaryEncodeFileToUChar  of oBase64 sFileName to ucEncoded 
          // more code here... for the actual chunked upload
        End_Function
    
      End_Object
    
      Object oOpenDialog is a OpenDialog
        Set Dialog_Caption to "Select a file"
    
        Set CreatePrompt_State  to False
        Set FileMustExist_State to True
        Set NoChangeDir_State   to True
        Set Filter_String to 'All Files|*.*'
      End_Object
      
      Object oFileForm is a Form
        Set Label to "Select File:"
        Set Location to 10 75
        Set Size to 13 150
        Set peAnchors to anBottomLeftRight
        Set Prompt_Button_Mode to pb_PromptOn
        Set Form_Button_Bitmap Item 0 to "GlobePlug.ico"
        
        Procedure Prompt
          Boolean bOk
          String  sFileName
        
          Get Show_Dialog of oOpenDialog to bOk
          If (bOk) Begin
            Get File_Name of oOpenDialog to sFileName
            Set Value              to sFileName    // as we do not want to save that
            Set Item_Changed_State to True         // in here.
          End
        End_Procedure
      End_Object
    
      Object oUploadButton is a Button
        Set Label to "Upload file"
        Set Size to 14 62
        Set Location to 45 122
        Set peAnchors to anBottomRight
    
        Procedure OnClick
          String  sFileName
          Boolean bExists
    
          Get Value of oFileForm to sFileName
          Send doUploadFile sFileName
        End_Procedure
      End_Object
      
      Procedure doUploadFile String sFileName
        Boolean bUploadOk
        String  sMsg
        
        Send Cursor_Wait of Cursor_Control
    
        Get UploadFile of oXmlUpload_Bp sFileName to bUploadOk
        
        Send Cursor_Ready of Cursor_Control
    
        If (bUploadOk) Move (sFileName*"upload to the server successfully.\n" ) to sMsg
        Else Move (sFileName*"upload to the server FAILED. Please try again\n" ) to sMsg
    
        Send Info_box sMsg "Upload status"
      End_Procedure
      
    End_Object
    
    Start_UI
    The code above comes from a heavily trimmed down webservice client to upload any kind of file to a DataFlex webservice.

    One of the things I tested it with was the 230MB sized DataFlex installer.. Not because we need to upload that kind of file, but because it was available and I wanted to see the performance and/or see if my code had an obvious issue in the webservice somewhere.

    So it's not exactly a small file to test with.

    However.. it isn't close to a 2GB limit either.

    As pointed out above it crashes on the line:
    Code:
                Get Base64EncodeUCharArray of oCharTranslate ucFile to ucEncoded   /// <--- crashing here
    If you put a breakpoint in that function itself then the crash is on the ResizeArray line below:
    Click image for larger version. 

Name:	OOM-Error.jpg 
Views:	26 
Size:	102.9 KB 
ID:	15616

    So this is just the code above, no webservice in sight or needed to reproduce this.

    I though I remembered that arrays are passed as pass by reference when used as a parameter to a function and then via "copy on write" expanded when changed. (see discussions from Sonny https://support.dataaccess.com/Forum...n-depth-Part-V ) but I might misremember and I guess I have to reread that blog series again.

    So if what I remember is true, it shouldn't -in theory- use more than 500MB in memory.
    If OTOH all of the data goes onto the stack on every call then I suppose it makes more sense.. but I'm still not counting 2GB.
    To be frank (Frank help?) it would make sense if the data goes on the stack.

    --
    Wil
    Last edited by wila; 23-Aug-2022 at 01:49 PM.

  2. #2
    Join Date
    Mar 2009
    Posts
    1,163

    Default Re: OOM Error on 32 Bit before hitting 2GB boundary?

    Did someone say my name???

    I have a better computer than you do. It takes resizing 300MB (as opposed to 230MB) of an array for me to get OOM with the latest 20.1.
    Obviously, you just need a better computer, lol

    Frank Cheng

  3. #3

    Default Re: OOM Error on 32 Bit before hitting 2GB boundary?

    Quote Originally Posted by Frank Cheng View Post
    Did someone say my name???

    I have a better computer than you do. It takes resizing 300MB (as opposed to 230MB) of an array for me to get OOM with the latest 20.1.
    Obviously, you just need a better computer, lol

    Frank Cheng


    Stupid Windows Server 2019 must be what is wrong here.

    --
    Wil

  4. #4
    Join Date
    Feb 2009
    Location
    Brazil
    Posts
    4,691

    Default Re: OOM Error on 32 Bit before hitting 2GB boundary?

    So if what I remember is true, it shouldn't -in theory- use more than 500MB in memory.
    Before getting to that line, there is at least 3 copies of that array in memory..

    The array you have in your procedure.

    The copy received by the Base64EncodedUCharArray parameter function. (uCharData)
    and
    3rd the Encoded buffer created. pBase64 (which is alwyas larger than the original one)

    so for sure it will use more than ~500MB . It may be near to the edge limit (2GB) before reaching the resizearray.

    have you checked the proces mem consumption using task manager just before reaching that line ?
    Samuel Pizarro

  5. #5

    Default Re: OOM Error on 32 Bit before hitting 2GB boundary?

    Samuel,

    Which is why I said "If what I remember is true" .. and that might be wrong.

    OK.. task manager says this:

    - immediately after reading in the file ==> 270MB in the Working Set column, so that's before calling Base64EncodeUCharArray
    - On the first and 2nd line in Base64EncodeUCharArray ==> 270 MB (looks like no 2 copies yet )
    - After Base64 encode ==> 806MB (which is about 200MB higher than I expected, but OK)
    - On the ResizeArray line with the OOM error ==> 1032MB

    So.. I'm seeing the OOM error on 1GB and not 2GB and I still find that surprising.

    --
    Wil
    Last edited by wila; 24-Aug-2022 at 04:17 AM. Reason: make more clear

  6. #6
    Join Date
    Feb 2009
    Location
    Sweden
    Posts
    1,788

    Default Re: OOM Error on 32 Bit before hitting 2GB boundary?

    Wil,

    Quote Originally Posted by wila View Post
    in the Working Set column
    I think you need to enable some other column, IIRC it's called either VMSize or Commit size. Enable any memory-related column you can find, it's the one with the largest value you want. =)
    // Anders

  7. #7

    Default Re: OOM Error on 32 Bit before hitting 2GB boundary?

    Quote Originally Posted by Anders Ohrt View Post
    Wil,



    I think you need to enable some other column, IIRC it's called either VMSize or Commit size. Enable any memory-related column you can find, it's the one with the largest value you want. =)
    Anders, I already have all columns selected
    Commit size says the same thing.

    Why don't you try yourself?
    --
    Wil

  8. #8
    Join Date
    Mar 2009
    Posts
    1,163

    Default Re: OOM Error on 32 Bit before hitting 2GB boundary?

    Why don't you try yourself?
    Is that why you posted code for reproducible steps in the original post?

    For me, the memory usage was not close to 1GB. Windows 10 Pro 64 bit with latest Windows update.

    Frank Cheng

  9. #9

    Default Re: OOM Error on 32 Bit before hitting 2GB boundary?

    Frank,

    Higher or lower?

    I just tried here on Win10 and it happened at 500MB somewhere. It also crashed in a different place (during the base64 encoding instead of during the uchar array resize)
    Time for me to reboot this Win10 machine as it has been running for a long time.

    --
    Wil

  10. #10
    Join Date
    Mar 2009
    Posts
    1,163

    Default Re: OOM Error on 32 Bit before hitting 2GB boundary?

    Here is a simpler example you can play with

    Code:
    Use UI
    
    Define CI_FILESIZE For 300000000
    
    Procedure Test 
        UChar[] fileContent
        Integer iBase64Size
        Move (CI_FILESIZE * 4 / 3) to iBase64Size
        Move (ResizeArray(fileContent, CI_FILESIZE)) to fileContent
        Move (ResizeArray(fileContent, iBase64Size)) to fileContent
        InKey FieldIndex
    End_Procedure
    
    Send Test
    Frank Cheng

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •