View RSS Feed

Development Team Blog

Diagnosing Studio and compile time performance problems

Rate this Entry
When it comes to performance working in the Studio as a developer, as opposed to running the application, there are generally two performance critical areas. The first one is the Studio Parser, which is the foundation for CodeSense, Visual Modeling, Go-To-Definition and more. The second performance critical area is when compiling. Unnecessarily long and slow compile times can make for a very frustrating experience working with the Visual DataFlex Studio.

Sometimes you may not even realize it's a problem that can be fixed. It's very common that as a developer you get used to the performance problem when compiling for example, and then you don't really notice it until you run into it in a different way, such as when upgrading the VDF Studio.

Every time I've seen or heard of any performance problem in the Studio, it's almost always been one of two things. Therefore these are the two things you should look at first when investigating a performance problem.

1) Virus Scanner. I can't begin to tell you how many times I've seen a virus scanner slowing down performance dramatically, sometimes making all file operations tens of times slower than it should be. More often than not you're convinced it's not the virus scanner, and that it's been configured properly, but after lots of back and forth it turns out that it really was the virus scanner after all. And it's often such a dramatic difference, not just a few seconds here and there, but it can be the difference between a minute down to seconds just for opening the workspace for example.

Usually it's a virus scanner that insists on scanning all file types on demand, including VDF source code files which are really just plain text files. Some virus scanners have separate configurations for scanning local drives and network resources, which can easily be overlooked.

2) Keeping the workspace source files on a network share. This is something that can be hard to appreciate, often because it may be a practice maintained for years. And for many reasons it can be a little difficult and take some effort to move the source files locally, so a quick test gets postponed at the cost of the continued peformance problem. All without really knowing how much of a difference it could be.

This one in particular came up a lot when upgrading to the new Studio and VDF 12+. Often you're so used to the compile times that you don't even consider it a problem (you don't know what you're missing). But with the new Studio and CodeSense/Metadata system, which relies heavily on parsing source code, you're all of a sudden exposed to that same network resource latency problem in a new way.

There's an excellent white paper discussing this subject here, unfortunately there were no real figures available at the time, so the issue of performance got watered down a bit. I'll present some numbers (and pretty bar charts) here so that you can really see the difference in performance between keeping the source code locally vs. a network resource.

I've used the standard Order Entry example and the VDF Studio project, compiling the exact same source code locally vs. over a network resource mapped to Z:, with VDF installed locally.

As you can see, compiling Order.src locally was around 6 seconds, vs. 21 seconds over the network. With a bigger project like Studio.src, it was around 29 seconds locally, vs. 1 minute and 33 seconds over the network. The numbers speak for themselves, and the larger the project the more you're paying for the network latency as it starts to add up.

As for the Studio Parser performance, I ran a similar test with both Order.src and Studio.src.

You see the same trend here, the time it takes for the Studio to parse the entire project for Order.src locally is less than 1 second, vs. 3.4 seconds over the network. That's obviously still reasonably fast over the network, but the relative difference is substantial. And as the project grows, this becomes even more noticeable. With Studio.src it takes about 1.7 seconds locally, vs. 11.8 seconds over the network. With a larger project this could easily be the difference between up to a minute over the network and a few seconds locally.

Obviously, parsing the entire project from scratch is not something you do all that much. Once you close the workspace, the metadata database is saved, and the next time it will load the .mtd file instead of parsing the whole project. But even that can be much slower over the network, as it will still verify the file timestamps when loading the workspace.

Often you might think it won't apply to your situation because you have such a fast network (gigabit network with super-duper ultra servers), but the bottleneck is usually not the network bandwidth, so it makes little difference in reality when it comes to the Studio and compiling. Hints of this could also be seen in my tests, where the network bandwidth utilization was typically around 5% when compiling over the network, and only spiking up to 60% very briefly on two occasions (once for a large file and then at the end when writing out the .exe).

I can only speculate here of course, but my educated guess is that it's the overhead of the many expensive network roundtrips required by SMB (Windows network file sharing), compared to accessing the local file cache. You can actually see a reasonable difference when enabling op-locks as well (which enables local caching), but that's of course not recommended as it will lead to unreliable results with the DataFlex database, as well as other weird problems with file timestamps and compilation.

Diagnosing Performance Problems
So how do you know if you've got a performance problem? It's obviously difficult to say, since performance varies with hardware and different environments. Compile times can be very tricky to compare. But the Studio has some excellent performance statistics available in Help -> Studio Performance Statistics. This will give you all sorts of information, but the most telling numbers are the total workspace load time at the top, and the various metadata statistics which I will go into in more detail. In general, if your workspace opens in a few seconds, you probably have nothing to worry about. If it takes over 10 seconds every time you open the workspace, you might want to check it out just to be sure. Similarly, the rule of thumb is that any and all CodeSense lists should generally display instantly (less than one second). If it takes longer than that almost all the time you bring up the CodeSense list, something's not right. And of course, if the Studio is performing very slowly, you may not realize it, but you're probably paying in slow compile times as well.

When loading the Studio workspace with an up to date .mtd file, this is what it typically looks like:

Note the total workspace load time is less than one second. Next check the database load size. This number tells you how many files were present in the database when the workspace was opened. The more files, the bigger the .mtd file, the longer it would take to load. So what you're looking for is if it's an unusal ratio of fewer files with a long load time.

The next thing is "validation time". This is the time it takes to verify the file timestamps of each file to see if the information in the database is up to date. If this takes a long time, it's a strong indication of a problem, as it's not even parsing the files here, it's merely checking the file timestamps. Most virus scanners can trigger a file scan when the timestamp is checked (thus causing this to take a very long time), in which case you would know that your virus scanner is unnecessarily scanning your source files, which can be a real drag on performance. Again, you weigh the total file validation time against the number of files (load size).

Next we'll look at the parsing performance, which indicates general performance when working in the Studio as opposed to just when loading the workspace. To illustrate these numbers I've removed the .mtd file so it would be forced to re-parse all files in the project.

Look at Total File Search time. That's the time spent searching for files along the path. Again, that time does not include parsing, merely the time it takes to locate and open the files. This number is usually significantly higher when source code is located on a network share, due to network roundtrips and chatter when searching for files.

Last but not least you can look at Total Files Parsed, as well as average files processed per second. All these numbers in the second section are live, and specify a snapshot of the parser statistics for all files parsed since the workspace was opened.

For comparison, here are some statistics for a large workspace with over 3000 source files, located on a local drive.

These numbers are all on a fast desktop computer, your mileage may vary of course.

So, in summary. Most performance problems with the Studio/CodeSense and compilation are due to either virus scanners or source code located on a network share. These two things should always be at the top of your list of things to check whenever you suspect a performance problem in the VDF Studio. And of course, make sure you're on the latest version of VDF. There have been several Studio performance related enhancements in both VDF 14 and 15.
Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	comp-perf.jpg 
Views:	1913 
Size:	26.8 KB 
ID:	3799   Click image for larger version. 

Name:	parse-perf.jpg 
Views:	1862 
Size:	29.0 KB 
ID:	3800   Click image for larger version. 

Name:	studio-load.jpg 
Views:	1862 
Size:	29.8 KB 
ID:	3801   Click image for larger version. 

Name:	studio-parse.jpg 
Views:	1935 
Size:	51.5 KB 
ID:	3802   Click image for larger version. 

Name:	bigws2.jpg 
Views:	1889 
Size:	50.7 KB 
ID:	3803   Click image for larger version. 

Name:	bigws1.jpg 
Views:	1921 
Size:	23.6 KB 
ID:	3804  

Updated 17-Mar-2010 at 05:25 PM by Sonny Falk