PDA

View Full Version : Win32 metadata project



wila
25-Jan-2021, 02:24 PM
In case you missed this:

https://blogs.windows.com/windowsdeveloper/2021/01/21/making-win32-apis-more-accessible-to-more-languages/

Now how do we get DataFlex on that list ;)

FWIW.. looks like the metadata is in:
https://github.com/microsoft/win32metadata/blob/master/scripts/BaselineWinmd/10.0.19041.5/Windows.Win32.winmd
(navigate up to get the version of another windows version)

and the metadata file is a binary...
--
Wil

Anders Ohrt
26-Jan-2021, 03:42 AM
Wil,

That's interesting! Annoying that they release the metadata as a WinMD file, but dumping it with Ildasm gives a nice text file (only 50MB!) which could be parsed in DataFlex. =)

wila
26-Jan-2021, 06:23 AM
Nice!

I had looked a little bit at the project before -without loading it into VS- and poked around the repo for a bit, but didn't see the winmd file at first, was wondering where they hid the goodies.
The description page was interesting, but busy with other things, so no time for a deep dive yet.

As it could help us all a lot if we get full Win32 API wrapper coverage I figured to post here, which is when I finally bumped into the winmd files.
Your discovery takes it another step further. I'll have to run that later on.

Wonder how hard it is to start generating API wrappers.

PS: I doubt we'll get DataFlex included in the project, unless somebody wants to work with MS and sign their CLA's and spent a lot of time on making them happy, but it could still prove very very useful for the DataFlex world itself because of the metadata and the ability to generate your Win32 wrappers from that.

--
Wil

Samuel Pizarro
26-Jan-2021, 06:56 AM
Sorry about my ignorance.... and for sure.. dummy question on the subject...

What does this actually means, in practical terms for a regular developer ?

We would not be required to import external function (External_Function declaration) anymore from the regular OS Win32 APIs ?

Anders Ohrt
26-Jan-2021, 07:01 AM
Samuel,

It means we could auto-generate the External_Function definitions, and just include a package file with them instead of manually writing them for each function you want to use.

wila
26-Jan-2021, 07:06 AM
Samuel,

Not a dummy question at all.

That's exactly what it would mean.
Instead of having to wonder if DAW or somebody else wrote a wrapper for any the standard Win32 functionality offered by Microsoft, you know it is already available.
Using the metadata it should be possible to generate the external_function declarations for ALL of the Microsoft API's available, not just a selection of the API's.

As the project already claims it is functional for C++, C# and rust (with other languages coming), I expect the metadata to also be sufficient to generate wrappers for DataFlex as well.

This is HUGE as it would take away the manual writing of wrappers and be able to completely automate the generation of said wrappers.

--
Wil

Samuel Pizarro
26-Jan-2021, 07:12 AM
Oh... that would be really nice...


now... Another silly question on top of that...

How would we avoid naming conflicts, with existing External_Function declarations out three, either inside DAW pkgs, or the ones created by our selves..

I guess this one will be hard to avoid 100% of the conflicts right, and some adjustments would be required

Again, Thanks for the explanations..

Samuel Pizarro
26-Jan-2021, 07:15 AM
Hi Anders

Nice.. really nice..

I spent a lot of times just doing that for the last couple of weeks, declaring a few functions from a db2 api I wanted to test here..
and I am not even at 10% of the work... If I would import all of the functions available there.

This looks nice. I would love to help.. but I am afraid my knowledge on this area won't be enough.

wila
26-Jan-2021, 07:51 AM
Samuel,

Once you have the logic and the metadata, there's a number of ways you can implement this.

For example generate all wrappers and only include in your project what you need.

Or.. generate on demand, eg. by pointing to the API call you want to use, import it into the current project.

Having potential collisions is not my main worry, yes they can be a pain.

Once you are able to generate these kind of wrappers, you can also do things like add a fixed prefix/postfix to all declarations.
It is even possible to scan DAWs code and only generate what isn't already available.

You first need to have the logic to generate wrappers, after that maybe start worrying about these potential issues.

A more interesting question for me is "who is going to built this".

If DAW is going to look into this and set aside developer resources for this then the community shouldn't spent much time on it.
If DAW is busy elsewhere, then it might be interesting for the community to see if we can join forces.
--
Wil

Samuel Pizarro
26-Jan-2021, 09:39 AM
Nice

starzen
26-Jan-2021, 09:42 AM
you wouldnt want to generate all of them and include. the df compiler would spend a lot of time compiling unused code and it isnt really good at that.

you could however have a utility that would allow you to generate what you need when you need it

for df of course you would have functions that may not be possible due to technical limitations (with differences between DF versions) and you may need some extra code to convert data types (again with differences between DF versions)

as far as the meta data you get everything you need. method declarations with parameters and types as well as type declarations etc.

here is an example of SetDefaultPrinter



[DllImport("WINSPOOL", ExactSpelling = true)]
public unsafe static extern BOOL SetDefaultPrinterA([Optional][In][Const][NativeTypeInfo(UnmanagedType.LPStr, IsNullTerminated = true)] sbyte* pszPrinter);


or the GDI function Pie



[DllImport("GDI32", ExactSpelling = true)]
public static extern BOOL Pie([In] HDC hdc, [In] int left, [In] int top, [In] int right, [In] int bottom, [In] int xr1, [In] int yr1, [In] int xr2, [In] int yr2);


and then typedefs like



[NativeTypedef]
public struct HDC
{
public IntPtr Value;
}


so you would need is a program to read the metadata file, have a search function then to generate need to look at the declaration and generate the ext function then possibly an intermediate DF function and all required type declarations

Frank Cheng
26-Jan-2021, 03:06 PM
That looks interesting. I did a quick google search on how to read .winmd file in C++, I didn't see anything promising just yet.

Frank Cheng

Anders Ohrt
27-Jan-2021, 01:23 AM
I did a quick google search on how to read .winmd file in C++, I didn't see anything promising just yet.

There is a reader in the cpp language project for this, but it's not easy to comprehend. This is really only usable in C#, I think. But as I discovered, just run ildasm on the winmd file and you can dump everything to text.

Anders Ohrt
27-Jan-2021, 01:25 AM
Samuel,



I spent a lot of times just doing that for the last couple of weeks, declaring a few functions from a db2 api I wanted to test here..


Note that this is just for the Windows API, you would need to get a WinMD file for the db2 API and I doubt that exists.

wila
27-Jan-2021, 08:26 AM
That looks interesting. I did a quick google search on how to read .winmd file in C++, I didn't see anything promising just yet.

Frank Cheng

This post appears to be helpful:
https://stackoverflow.com/questions/54375771/how-to-read-a-winmd-winrt-metadata-file

--
Wil

Frank Cheng
27-Jan-2021, 10:11 AM
Hi Wil,

The example did help alot. However, IMO, in order to make use of it, you kinda have to load the information to a database in order to search for stuff. If you look at the interface for IMetaDataImport, most member functions start with "Enum". IOW, I can't really ask the IMetaDataImport interface to give me the definition of the API "Shell_NotifyIcon" directly. I am thinking about loading the entire thing to SQLite first, then I can make some dynamic queries.

Frank Cheng

Samuel Pizarro
27-Jan-2021, 12:43 PM
Note that this is just for the Windows API,

Yeah.. I got that.. I just gave an example of similar that I was working on couple of weeks ago..

Thanks

Anders Ohrt
10-Feb-2021, 03:28 AM
Wil,

I could not leave this alone, so I wrote a parser. I now have ~600 files. =)

I'll put it on VDF-guidance once I've cleaned it up a bit.

wila
10-Feb-2021, 11:51 AM
Anders,


Wil,

I could not leave this alone, so I wrote a parser. I now have ~600 files. =)

I'll put it on VDF-guidance once I've cleaned it up a bit.

Was hoping that someone would play with it before me. :)

Happy to hear that you want to share it, it is a really interesting project and it has the potential of helping a LOT of people.

Let me know if/when you need help with getting it on vdf-guidance, you can reach me at wil-a@vdf-guidance.com
--
Wil

Frank Cheng
15-Mar-2021, 08:58 AM
Wil,

Just curious if you have received anything from Anders yet. I am interested in seeing those ~600 files.

Frank Cheng

Anders Ohrt
15-Mar-2021, 11:41 AM
Frank,

Unfortunately this did not go as I had hoped. Writing a solution handling 90% of the metadata was quick, but the last 10% has taken more than 90% of the effort. Things like structs with unions with structs with unions got a bit too tricky.
Iíve given up on parsing this in DF and am working on creating a DLL using the Windows Metadata API instead. Since Iím working on this when I get some time from regular work it is not going fast.

Frank Cheng
15-Mar-2021, 03:36 PM
Hi Anders,

No worries, I was only curious. Thank you for doing this for the entire dataflex community.

Frank Cheng

Samuel Pizarro
15-Mar-2021, 09:07 PM
+1.