View RSS Feed

Development Team Blog

Under the Hood: The Studio Parser

Rate this Entry
The Visual DataFlex Studio parser is actually two separate parsers, one is parsing source code and the other is parsing the tree structure output by the first one. The first one is internally referred to as the CDS(Code Data Structure) Parser, and the second one is referred to as the Metadata Parser. But often they are referred to collectively as the Studio Parser, as the technical distinctions are often irrelevant unless you're discussing Studio internals. The CDS and Metadata parsers are also often referred to collectively because even though they serve different purposes, and indeed one is built on top of the other, most pieces of the Studio require information from both sources in order to perform its intended function.

The Studio also has a third type of parser for the editor, it's the piece of software that performs syntax highlighting in the editor and it's completely separate and independent from the general Studio Parser. To form a complete picture there's even a fourth parser, the CodeSense Parser, which analyzes the source line you're currently editing to give you all the wonderful CodeSense lists and tooltips. It's a separate line/statement parser that inspects the line of code in much greater detail than the Studio parser. The CodeSense system then uses information from all four parsers to analyze the current line and provide the CodeSense lists and tooltips. But I digress.

Parsing Source Code
Let's illustrate with an example, consider the following code:

Class cMyButton is a Button
When the Studio Parser sees this code it goes "oh, cool, a class called cMyButton, I'll remember that for later." It then analyzes that a little closer and goes "ahh, it's derived from the Button class, let's look that up and see what we know about Button". If the Studio Parser has seen the class Button before, it would have some information about that class. This type of information about classes and methods etc. that are kept for later lookup and retrieval is referred to as metadata.

Most of the metadata comes directly out of the source code, such as the above code where it concludes that the superclass of cMyButton is Button, and all the functions & procedures defined within the class belong to the class. Properties defined within the class are also picked up etc. This is where the often repeated mantra "source code rules" comes from, despite what other uses you may heard for it, this is the original definition of that term coined at a very early stage in Project Horizon. It's in contrast to the old IDE where the whole structure of classes and all properties etc. had to be redefined and replicated in those dreaded .dfc files. The Studio Parser now simply reads the source code itself, and whatever the source code says is what it models and interprets, as opposed to some external special configuration file, hence "source code rules".

Meta Tags
Despite parsing the source code in all details, there are some things the Studio needs to know that can't be deduced from the source code itself. For example, the Visual Designer has to map certain classes to types of visual representations in the designer. These are virtual properties of a class so to speak. This information is not directly present in normal source code, and not of interest to the running program. But rather than storing the information separately and reintroducing the .dfc nightmare(or worse, storing the information hardcoded in the Studio), it's provided within the source code attached to the class/method as meta tags, e.g. { DesignerClass=cDTButton }.

When the Visual Designer encounters an object based on cMyButton, it consults the metadata system and looks up the class and the corresponding DesignerClass attribute. If it finds that the DesignerClass attribute (inherited from the Button class) is cDTButton, it models the object and displays it in the Visual Designer as a button.

Common Problems Seen in the Studio
Now consider what happens if the Studio looks up cMyButton but cannot find any information about that class. In that case the DesignerClass attribute has no value, and the Visual Designer consequently won't know how to display the object, so it ignores it and moves on to the next object. This results in the phenomena of invisible objects in the Visual Designer.

Luckily, all is not lost when this happens, as the Studio has several ways of informing you of such problems. Meet the Problem Resolution window, it's your friend. Whenever something doesn't seem to display or model properly in the Studio, you should reach for View -> Problem Resolution first before anything else, chances are it will point out the cause of the problem. We'll look at several common problems in turn.

Notice the little red icon, it means it's considered a metadata error, as opposed to a metadata warning with a yellow warning triangle. A metadata error means the Studio Parser immediately knows something's wrong, and it has an incomplete picture of the specified class/object, which will most likely result in problems in the Studio Visual Designer, Properties window etc.

Most likely you'll notice that the affected objects are not visible/displayed in the visual designer. You'll also notice that while the Visual Designer does its best to model the remaining objects, it refuses to let you drop new objects from the class palette onto the view/dialog. It does this to reduce chances of conflicts arising from adding new objects without first fixing the problems so the Visual Designer can have a complete picture of the view/dialog.

The error message should be pretty self explanatory, it tells you that it refers to an undefined class. The solution is usually very simple, add the corresponding Use statement so that the Studio Parser can find the class.

Notice that this message has a warning triangle icon instead, and although it looks similar to the above message, it's slightly differently worded, saying it's an "External reference to superclass..." instead of "Undefined superclass..." The actual problem in this case is really the same as above, the corresponding Use statement is missing. However, the Studio is smart enough to realize that the corresponding Use statement exists elsewhere in the project, and the Studio already recognizes the class.

The class reference is indirectly resolved via an external reference to the superclass from a Use statement in another file within the same project. The fancy term used to refer to this in VDF Studio lingo is non-autonomous files. The documentation also has a great explanation of how this is resolved via Project Context Sensitive Parsing.

What is needed is some way to tell the Studio that a non-autonomous file is part of a larger project. The Studio would then have the information needed to understand and model the file. The Studio provides this capability in a straight forward manner. If your non autonomous file is a member of the current project (i.e., it will get used and compiled when the current project is compiled), the Studio will use all of the files available in the current project to help satisfy any missing dependencies. You might think of this as a shortcut. If the file is available to the current project, it is also available to a non-autonomous file, if that the file is a member of the current project.
You'll notice that this approach of resolving non-autonomous files depends on two other requirements. First, the non-autonomous file must be part of the project. If the specified file is not part of the project, i.e. there's no Use statement anywhere in the project(yet) to bring in the specified file, then the Studio is not able to resolve the external reference.

Second, the Studio always attempts to resolve external references in the context of the current project. The current project is a concept used throughout the Studio. Each workspace can have multiple projects, and at any given time only one project is considered the current/active project. When you click the Run toolbar button or Compile button, the Studio compiles & runs the current project. You can switch the current project in Workspace Explorer for example by selecting the specified project in the combobox. You can also switch current project from the toolbar.

This brings us to the next problem. What happens when the file is non-autonomous and it's not part of the current project? John van Houten recently discussed this in a forum thread. Different pieces of the Studio react in different ways to this situation, but most notably the Visual Designer and DDO explorer will simply refuse to model the component. This may seem a little harsh, but these designers allow high level modeling and manipulation of the source code, and if their view of the code is incorrect then chances are it will just cause confusion and potential to add conflicting code. Instead the Studio takes the safer approach and informs you that it doesn't understand how to properly model the code because of the missing classes, and how you can resolve the problem.

There are several different things to notice here. First, it tells you that the file is not part of the current project. It even tells you what the current project is, in case the wrong project had been selected by accident. Second, it tells you that the file refers to classes that cannot be resolved, making it non-autonomous. Third, it tells you that there are two ways to resolve this problem, you can add the missing Use statements to turn it into an autonomous file, or you can try to switch the current project to see if the class references can be resolved elsewhere in another project.

In all cases, the Problem Resolution window helps with pointing out the problems and understanding how to resolve the situation and help the Studio model the code better. The finer details of the whole error message displayed in the corresponding designer gives you additional clues.

In summary:
  • Objects invisible/missing in Visual Designer: View -> Problem Resolution.
  • Error: This file is not a member of the current project: View -> Problem Resolution or switch Current Project.
  • Missing properties in Properties window: View -> Problem Resolution.
  • Unable to drag and drop new controls onto Visual Designer: View -> Problem Resolution.

Conflicting Classes
There is another common situation where the Problem Resolution window isn't able to point out the problem. This happens when the Studio is able to resolve a class reference to not just one, but multiple classes. It's very common to have multiple classes with the same name in different projects(even within the same workspace), and the metadata system is designed to handle that properly, so it stores information about both classes.

The Studio then filters out classes and files that are not part of the current project when looking up information, thereby it will effectively find the appropriate class for the project. Naturally, each project can only have one instance of a class with a specific name, or it won't even compile without error. But sometimes you can hide one class from the compiler with #ifdef symbols, and thereby have two classes with the same name in the same project, even though only one class is actually compiled. In this case the Studio will need some help to pick the right side of the #ifdef so to speak. You can specify such conditional symbols in the Configure Workspace Properties dialog.

When this situation occurs, the Studio finds that there are multiple classes that satisfies the current lookup request from the Visual Designer or Properties Panel or similar. Since it cannot know which one to pick, it will just pick one to satisfy the request and hope for the best. If it's the one you expected, you won't even notice that this conflict occurred. If it's not the class you expected, you'll most likely see some strange side effects, as the information for the class is not what you expected.

The most common side effects of this problem are missing properties in the Properties window, and invisible/missing objects in the Visual Designer. For example, the Label properties of form/dbForm objects seem to disappear from the Properties window, and dbContainer3d objects seem to disappear from the Visual Designer. In most cases the conflicting classes are separate versions with the same name intended for use with WebApp vs. Windows GUI programs respectively.

If you suspect this problem you can very easily find out which class with that name that the Studio selects, by right clicking the class name in the editor and selecting Go To Definition. If it takes you to the wrong class with the same name, you know this problem is hitting you.

There are generally two ways to fix such class conflicts. The best solution is to eliminate the conflict altogether, by moving out the two different classes into separate files. And then simply Use'ing the file and class that is appropriate for the specific project, e.g. WebApp or Windows, so that the Studio only sees the correct class for the corresponding project.

Although eliminating the conflict is the best solution, it's also more intrusive, which can be problematic if it's not your own code. In that case you can typically resolve the conflict with a hint to the Studio by specifying the appropriate conditional symbol and value in the Configure Workspace Properties dialog. e.g. define Is$WebApp to False or something.
Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	pr1.jpg 
Views:	1256 
Size:	24.1 KB 
ID:	3761   Click image for larger version. 

Name:	pr2.jpg 
Views:	1291 
Size:	24.7 KB 
ID:	3762   Click image for larger version. 

Name:	vd1.jpg 
Views:	1214 
Size:	50.4 KB 
ID:	3763   Click image for larger version. 

Name:	ddoexpl.jpg 
Views:	1215 
Size:	38.9 KB 
ID:	3764  


  1. Rafael M Heise's Avatar
    I think that post clarify a lot of things!

    Congratulations Sonny
  2. Chris Spencer's Avatar
    Thanks for that in-depth coverage
  3. Ola Eldoy's Avatar
    Excellent post!

    What I would love, though, is access to the parsed data for analysis and perhaps adding extended functionality to the Studio.

    As a start:
    • Unused local variables
    • Obsolete code
  4. Clive Richmond's Avatar
    Hi Sonny,

    We've modified the VsiualDataFlex.lng so it fits in with our style of coding. The changes take place when you move away from the line being edited.

    Where does the VsiualDataFlex.lng fit into all of this?

  5. TBrauner's Avatar
    Hi Sonny,

    i use the vdfcds.dll to export all possible information about a project (classe, methotds, properties etc.) to an UML tool (Enterprise Architect, SPARX). Today i only use the metadata and it was much work to find out, how all the methods works. Is there any "internal", unofficiel description with all the undocumented features of this DLL?
    I just don't find out how i get all object and their methods.

  6. Dennis Piccioni's Avatar
    Hi Thomas,

    I replied in the thread you started about this subject.