View RSS Feed

Development Team Blog

Transferring large strings with web services

Rate this Entry
The other day I received an interesting question from a developer about argument_size and web services. Specifically he wanted to create a web service in VDF that accepts a file attachment along with some other metadata for the file. It would look something like this (actual code modified for brevity):

Code:
Struct tAttachment
    String fileName
    String comment
    ...
    String fileContent
End_Struct

...

{ Published = True }
Function SubmitFile tAttachment fileAttachment ...
The file data would be transferred in base64 encoded form in fileContent. So far this all looks very good, except there's a potential problem with argument_size if the file is over 64K for example. Now there are several solutions I can think of here. First, note that you can't just change argument_size in the web service method, as the data would would already have been truncated. Before we get to my recommended solution, let's look at some other possible ways to deal with this problem.

The first solution - xmlHandle
When creating a web service in VDF, you can choose between two general strategies, either passing in and out plain XML with xmlHandle and working with cXmlDomDocument and friends, or you can use regular VDF types including struct and arrays. By the time the data goes across the wire, it doesn't really matter that much, it's all XML anyway. The real difference is in ease of use, where struct and array types are much easier to deal with and far less error prone, and it's also automatically described in the WSDL which makes it easier to use for the client side as well.

The xmlHandle parameter type for web services can be very useful when the structure of the data needs to be flexible. More importantly to our case, it's also not affected by argument_size. This means that if you use xmlHandle as a parameter type for your web service, argument_size is no longer an issue, therefore it could be a potential solution. The cost is of course in ease of use, as you lose all those benefits from a struct type. (Note that xmlHandle cannot be used as a struct member, it can only be used as a parameter or return type for a published web service.)

The second idea - array
Another possibility is to use something like Char[] instead of String. Arrays are not affected by argument_size, so therefore it's also a potential solution. The good thing is that it can easily be used together with the struct type, so you still get all the benefits of ease of use and the schema in the WSDL etc., making it a better candidate in our case.

But, the downside is that it's now an array, which will make for an awkward structure of the data in XML, as it would create additional elements for each and every character of the string. That would be an awful lot of waste, let's throw that idea in the trash right away.

The winner - Address type to the rescue
As I've mentioned in passing before, the common solution to this kind of problem in general is the Address type, which allows you to pass around a pointer to the data. The good thing is that the Address type is not affected by argument_size, so it's clearly a potential solution to this problem. It can also be used together with the struct type, so you get all the benefits of ease of use and the schema and all.

Even better, VDF web services actually have built-in additional support for the Address type, exactly for this purpose. If you use the Address type (even as a struct member) it behaves just as if it was String as far as the web service is concerned, except you get a pointer to deal with in VDF code instead of a String. The runtime automatically allocates a buffer large enough to the hold the data, and then also automatically deallocates the buffer when the function returns. This is perfect in our case, it makes it all super easy to deal with from a VDF code point of view. This whole thing actually builds on top of the same magic that makes xmlHandle work as a parameter type as well.

So the solution in this case is to change the struct type to use Address instead of String for the file content.
Code:
Struct tAttachment
    String fileName
    String comment
    ...
    Address fileContent
End_Struct
It doesn't change anything for the client side, just the VDF code for the server side. The client can now submit the file contents in base64 encoded format as a long string, and when it gets to your VDF code you can access the data in the fileContent struct member as a pointer and not worry about argument_size. Of course, you then have to write the code to do something interesting with the data, and that I will leave up to your imagination.
Tags: sonny falk
Categories
Uncategorized

Comments

  1. Jakob Kruse's Avatar
    Very useful, that Address type!
  2. Knut Sparhell's Avatar
    Could you give an example of how to access the data in an Address variable?
  3. Rafael M Heise's Avatar
    I think as you are wainting for a string, you can use CStringLength function to get the length, change the argument_size for your needs, and then move the Address variable to some String variable with explicit cast: Move (String(aAddress)) to sString.

    I don't remeber if there is a easier way to do that. Anyway, I belive it should works.

    Excellent post Sonny. Congratulations!
    Updated 26-May-2010 at 01:58 PM by Rafael M Heise (Add a little bit comment)