View RSS Feed

Development Team Blog

On the same (code) page

Rate this Entry
International characters have always been very complicated to deal with in computer systems, and configuring your computer for your locale is often more difficult than it should be. Have you ever had the problem where extended ascii characters such as ends up being displayed in a dbForm or similar in your VDF application as instead? Or ends up as . Another common symptom is losing characters when typing and saving the record. Often it gets converted to some other character when saved. Usually this is due to a simple Windows configuration that is quite confusing and far too easy to miss.

Technically speaking, when this problem occurs in a Visual DataFlex application, it's an OEM code page configuration problem, where the data is stored in OEM codepage 850 for example, but accidentally read back and interpreted using another OEM codepage, such as 437. Or alternatively, while the particular extended ascii character is present in code page 850, it's not available in code page 437, and therefore disappears when you save the data using the wrong code page. There are also many other reasons for various types of international character issues, but if you haven't done anything special, and the characters usually display correctly, this is probably the most likely reason.

Ansi, OEM code pages are confusing enough as is, configuring Windows correctly for your locale is even more confusing. Because VDF relies on OEM/Ansi translations, it's important that Windows is configured correctly both for Current User and System locale. I won't go into detail about OEM/Ansi for now (that's a whole topic in itself for another time), this article will just focus on what happens when it's not configured correctly, a utility to detect misconfigured systems, and how to fix conflicting configurations.

You may not notice right away if Windows has conflicting settings, because your program will appear to run just fine. But you'll eventually notice that certain extended ascii characters that are supposed to work with your locale settings are not always displayed correctly.

VDF uses the Windows functions CharToOEM() and OEMToChar() to perform the translation, and these functions rely on the System Locale settings, rather than the Current User Locale as one might expect. So therefore the System Locale settings must be properly configured. Conflicting settings between Current User and System configurations are probably not intentional anyway, but it can easily happen by accident since these configurations in Windows are quite confusing. It's also a very common mistake when installing a US/English version of Windows, and then changing the Current User locale but forgetting to change the System locale, ending up with different and conflcting Ansi/OEM code pages.

Detecting Conflicting Code Page Settings
To simplify testing and verification of the configuration so that VDF operates correctly, I've put together a simple little test utility that detects conflicting code page configurations. The full source code is available as an attachment to this article. I've also provided a VDF 15.1 compiled executable version for convenience. The program requires no workspace and can be compiled and run anywhere.

This utility displays some information about the Current User and System locale configurations. With a standard US English configuration it looks like this:


Note that the sections for Current User and System info are identical. The Ansi and OEM code pages in the two respective sections are what's most important in regards to VDF OEM/Ansi translations.

If we change this to Swedish without properly changing the System locale, it might look like this:


Note that the OEM code page in the two sections are not the same. In fact, you'll notice that while the Current User section is configured for Swedish, the System section is still entirely configured for US English. It's very easy to miss this configuration since the date format and everything appears in Swedish. The only practical difference to most VDF programs is that it's still using the wrong Ansi and/or OEM code pages, which can produce the wrong result.

If conflicting settings are detected as in above, you'll be presented with this message:




Resolving the Problem and Getting Back on the Same Code Page
To resolve the conflict you have to change the System Locale so it matches your Current User locale. In Windows XP the System Locale setting can be found in Regional and Language Options, in the Advanced tab, select your specific language for non-unicode programs. And then also check the box in Default user account settings. Checking this box is what applies your changes to the System account also.



In Vista it's similar but with different names and actually better descriptions, on the Administrative tab, click on Change system locale to pick the same language, and also click Copy to reserved accounts.



In Windows versions prior to Vista and XP these settings may sometimes be called something else, like system default settings or something. It's all the same idea, just different names and descriptions for the same underlying settings. When the changes have been made, you usually have to reboot to apply them. You may even have to reboot between the different changes depending on the Windows version.

Once you've corrected the conflicting settings you can re-run the utility to verify that it's working correctly.

Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	us-good.jpg 
Views:	843 
Size:	43.0 KB 
ID:	3779   Click image for larger version. 

Name:	swe-bad.jpg 
Views:	891 
Size:	42.3 KB 
ID:	3780   Click image for larger version. 

Name:	conflict.jpg 
Views:	839 
Size:	37.9 KB 
ID:	3781   Click image for larger version. 

Name:	settings-xp.jpg 
Views:	855 
Size:	93.2 KB 
ID:	3782   Click image for larger version. 

Name:	settings-vista.jpg 
Views:	886 
Size:	104.1 KB 
ID:	3783   Click image for larger version. 

Name:	swe-good.jpg 
Views:	836 
Size:	49.5 KB 
ID:	3784  
Attached Thumbnails Attached Images     
Attached Thumbnails Attached Files

Updated 24-Mar-2011 at 02:03 PM by Marcia Booth

Categories
Uncategorized

Comments

  1. Anders Ohrt's Avatar
    In order to get proper language support, you need three more things: df_collate.cfg, dfini.cfg and a collate for your DB. The df_collate.cfg is used for VDF string operations and for the embedded DB; dfini.cfg is used for (among many other things) the Lowercase and Uppercase functions, and the collate in the DB for indices to sort.

    I've created an application what generates perfect versions of these files (the collate is for PSQL), so if anyone is interested they can contact me in the forum.
  2. Samuel Pizarro's Avatar
    Hi Sony.

    This is really a good topic. Some decades ago, I was told that VDF uses OEM code-page to keep compatibility with DF character.

    How long will DAW keep it this way? ANSI code-page exist a long time. There are a lot of DAW Customers that don't uses DF Character anymore, or never have used it before, (like me). I just have Windows and Web-App. Why I have to keep using OEM code-pages.

    Don't you think that should be great we have an option on VDF to choose If we want our database to save ANSI or OEM. So when choosing ANSI, VDF runtime and classes don't need to translate it anymore. Everything is simpler.

    In fact, we are so late, that newer systems and DataBases like DB2, are using UNICODE. UNICODE is the future.

    Have you a tip, or can you talk a little more about it!
    HAS DAW plans to suport only ANSI or UNICODE in a very soon future?

    Regards