DF 19.0 - BLOB and TEXT allocations on table global buffer
Hi DAW.
On help, we have this documented for Binary and TEXT columns ...
[code]
DF_TEXT and DF_BINARY column length in DataFlex SQL Drivers
The maximum size of text and binary columns varies greatly between different back ends. For the Embedded Database it is 16kb, for SQL Server it is 2Gb.
When opening a table in such a back –end, it usually reports the column length as 2Gb.
When using the DataFlex SQL Drivers, the size of such columns will be truncated at 16kb by default. You can increase this size by editing the FIELD_LENGTH intermediate file keyword or setting the DF_FIELD_LENGTH attribute to the desired length when creating the column. [B]
[COLOR=#ff0000]Note that these columns are part of a table’s buffer and the size you specify for it will be allocated in memory when the table is opened[/COLOR][/B][COLOR=#ff0000].[/COLOR]
[/code]
Is this still true on DF 19.0? No enhancement on that area to just allocate what is really necessary dynamically using uChar arrays ?
If I have a 500 MB blob column, will it really allocate that much for the global buffer , even when the buffer is cleared, or most os the blobs, are smaller than that ?
I am working in a VIDEO/audio project here, and if this is still true, I need to change the table access method.
Regards
Re: DF 19.0 - BLOB and TEXT allocations on table global buffer
Ok.
I decided to run a small test and got surprised and upset.
In the back end, I have this blob column which is around 156 MB
I let the default 16k setting on the int for it. Compiled a simple program just opening this table. Went to Windows task list and saw the test.exe process had allocated around 32MB. Ok with that..
Then I changed the FIELD_LENGHT int attribute for this column to its real size.. (minus 2 bytes, per previous issues reported on WEbserverProps PAGING truncating 2 bytes on each page) .
[code]
FIELD_NUMBER 7
FIELD_LENGTH 157286398
[/code]
Compiled the program.. executed it.. And for my surprise, the test.exe process was now allocating 322 MB. :confused:
I went back to DF Studio, and noticed that when I select this column in Studio, the properties panel shows this in the [B]Size[/B] section:
[code]
length : 157286398
native length : 314572795
native size : 157286400
[/code]
"Native Size" is the real size defined on back-end. What is [I]"Native Length"[/I] and why its twice the [I]length[/I] [I](df_field_length)[/I]?
Reading the help for this attribute, the range should be
[B]Values[/B]
1 (minimum) ~ max field size (maximum)
Is this really expected ?
Regards
Re: DF 19.0 - BLOB and TEXT allocations on table global buffer
Hi Samuel,
IIRC the uChar functionality was supposed to help get around the memory issue. At least I seem to remember it that way and I'm sure the question was asked at some point. Can't remember the details.
I had the same issue a while ago with documents stored in an MSSQL blob field. It allocated memory according to what was in the INT file but did it for every DD object instance that referred to that table. Ended up with program using 2GB memory and had to remove that functionality.
Cheers
Richard
Re: DF 19.0 - BLOB and TEXT allocations on table global buffer
Hi Richard
Thanks for your feedback..
I had the same thinking, but it does not looks like that. Look at my 2nd post.
It looks the uChar usage will only save you on variables allocations.. When you need to move from global buffer to some variable.. But the global buffer itself, its looks it still pre-allocating the full size. Maybe for reservation purpose.. I can kind of understand that.
What I cant understand, is why its actually allocating the double of it.
Regards
Re: DF 19.0 - BLOB and TEXT allocations on table global buffer
Ouch !!! I'm seeing the same thing on my system !!!
I believe the uChar enhancements were targeted at the DataDictionary class - looks like it never made it to the file buffers.
My test:
1. Created a new table with a BLOB of 16KB, compiled and executed at 8548KB
2. Modified the BLOB to 10MB, compiled and executed at 28,460KB
That's a difference of 19MB !?? Double allocation !?
[Code]
Use Windows.pkg
Use cHtmlHelp.pkg
Use cApplication.pkg
Use cConnection.pkg
Object oHtmlHelp is a cHtmlHelp
End_Object
Object oApplication is a cApplication
Object oConnection is a cConnection
Set pbLoginOnOpen to False
Use LoginEncryption.pkg
Use DatabaseLoginDialog.dg
End_Object
End_Object
Open DOCSTORE // Testing BLOB
Send Info_Box "Hello World" "DataFlex Basic Project"
[/Code]
Re: DF 19.0 - BLOB and TEXT allocations on table global buffer
[QUOTE=Samuel Pizarro;333829]Hi Richard
What I cant understand, is why its actually allocating the double of it.
[/QUOTE]
File buffer and SQL fetch struct !!??
Re: DF 19.0 - BLOB and TEXT allocations on table global buffer
I guess it's "double" because it may keep a copy of the as read from disk data in memory so it knows what has changed
Re: DF 19.0 - BLOB and TEXT allocations on table global buffer
Too much to be wasted in memory just for that!
Imagine an webapp that has 20 process in the pool with 10 tables having some big blobs ( like MB range)
This behavior will consume a lot of RAM.
DAW, any comment?
Re: DF 19.0 - BLOB and TEXT allocations on table global buffer
Re: DF 19.0 - BLOB and TEXT allocations on table global buffer
I did not receive anything in private or personal email.