PDA

View Full Version : Cleanmarkers update to clean up your DataFlex 19.1 source



wila
8-Jun-2018, 12:08 PM
Hi,

As introduced in the thread https://support.dataaccess.com/Forums/showthread.php?62798-Splitting-an-if-else-case-line-with-a-line-separator-(-)-will-make-it-easier-to-debug

Here's the full workspace for the new Clean Markers workspace that helps you cleaning up your source.
It has been extensively tested, but I still advise caution. (aka Make a backup!)

12033

Let me list some of the features here:

- Split up your single line if/else via either a semicolumn ( ; ) or a begin/end block and you can set your tab size.
- Remove keywords public/private used by properties
- Remove keyword local in variable declarations
- Remove comments after end_function/end_procedure/end_object/end_class

This is all optional, so if you only want to clean up one part and not another, just uncheck a checkbox.

The file date/time if the file was not changed is now preserved (note that your file does get rewritten either way, but if it is identical we now preserve the last write datetime from before it was processed)


You can select a single file to clean up or a folder to clean up all files in that folder at once. If you select a folder then ALL of the subfolders will be cleaned up as well.

We now have statistics that shows you how much changes have been made and on what.

Note that Cleanmarkers has run over its own source to remove the warnings and I also updated the rest of the packages to be warning free.

The workspace runs in DataFlex 19.0 and DataFlex 19.1, but can of course be run against older versions of your workspace.

Update 2018/06/09 : Trimming the line before breaking it and added an option to choose between ' ;' to split a line or ';' (by default we add a space before the semicolumn)
Also redesigned the screen a bit so it wasn't dropping off my screen at the bottom.
Update 2018/06/11 : Added Studio Integration. There's now a menu option to add the CleanMarkers application into the DataFlex 19.1 Studio menu. The studio menu option will automatically select the current source file into the CleanMarkers application so you can immediately run the process on one file.
Update 2018/06/11c : Addressed issue as reported by Hans (see below)
Also addressed a UI incompatibility with first running against a single file and then immediately afterwards running against a folder.
Added a few more obsolete commands to test for as well.
Update 2018/06/12: For the people who like to clean file by file and compare the file changes I have now added support for a tighter integration with your file comparison tool of choice.
Update 2018/06/14: Add support for files with LF instead of CRLF as end of line character.
Support more file types by default: .bp, .mn, .h
Do not cleanup in folders with name: .svn, .git, .hg or AppHtml
Fix an issue with detecting files that have a file extension shorter than 3 characters
Added library vWin32fh

Update 2018/06/14a: Found a solution for files with CRLF line endings that have lines with LF in them and not muck it up. Instead it will be rewritten with CRLF instead.
Update 2018/06/21: Todd added Drag and Drop support for single files.

Update 2018/07/13: Dave discovered that files that are left unchanged still generate .cmbak files, we now delete that backup file if it was not changed.

Update 2018/07/16: Added DataFlex 18.1 and 18.2 support. Also updated vWin32fh library to not use #IFNDEF for adding the global function zerostring workaround, but use the compiler revision (!@ <191) instead, so as to make sure we never use the workaround function in DF19.1+ (thanks Nils for pointing that out)


Credits:
This is a workspace based on earlier work by Allan Greis Eriksen found here:
http://vdf-guidance.com/ContribPage.asp?Page=SOFTGENCLEANMARK&ContribRecId=115
which by itself was based on work by Marcia Booth

I have had help and code from: Marco, Peter, Sture and Todd.
Borrowed some code from Nils's excellent Manifest SideBySide studio for the integration with the DataFlex 19.1 Studio.

Enjoy!
--
Wil

Stephen W. Meeley
8-Jun-2018, 01:16 PM
Wil,

I tried running this on a large project and it crashes on this line...

Set_File_Mod_Time sFileName To dFileDate iFileHour iFileMinute iFileSecond

...so it seems like when it runs into files that it didn't change and tries to touch the time back to the original it fails.

12028

wila
8-Jun-2018, 02:41 PM
Hi Stephen,

Curious.

Down here I tested with the sources from the Hammer project and some specific test files.
Here it did change the date/time back without issues.

A few questions if you don't mind in trying to chase this one down.
Could it be that the file it tried to restore the datetime from was open somewhere else?
Perhaps it is read only?
Are the files on a network (maybe closing the files takes more time and it needs a tiny delay?)
Does the file have a strange datetime stamp?
Is there something else different about that file?
Oh I forgot to ask.. can you reproduce this, or is it a one off?

--
Wil

Todd Forsberg
8-Jun-2018, 02:52 PM
I can reproduce it.

My first guess, off the cuff, is that SET_FILE_MOD_TIME does not like long file names. When it aborted, it was on a file name larger than the old 8.3 spec.

wila
8-Jun-2018, 03:08 PM
That's correct. I can reproduce it now too.
So it looks like I have to do this with the windows API then :s

--
Wil

Stephen W. Meeley
8-Jun-2018, 03:11 PM
Wil,

Could it be that the file it tried to restore the datetime from was open somewhere else?
No

Perhaps it is read only?
No

Are the files on a network (maybe closing the files takes more time and it needs a tiny delay?)
No

Does the file have a strange datetime stamp?
No

Is there something else different about that file?
Not that I can see

Oh I forgot to ask.. can you reproduce this, or is it a one off?
Reproducable

Fernando Faria
8-Jun-2018, 03:15 PM
wila,

When I try to compile in DF19.0 I get many erros about zerostring function:

----Compiling CleanMarkers.src----
- Compiling Program: Y:\Libraries\CleanMarkers\AppSrc\CleanMarkers.src
- Memory Available: 2147483646
- Using pre-compiled package DFALLENT.PKG
- Including file: dfallent.pkd (C:\Program Files (x86)\DataFlex 19.0\Pkg\dfallent.pkd)
- Including file: cCJStandardCommandBarSystem.pkg (C:\Program Files (x86)\DataFlex 19.0\Pkg\cCJStandardCommandBarSystem.pkg)
- Including file: cCJStandardMenuItemClasses.pkg (C:\Program Files (x86)\DataFlex 19.0\Pkg\cCJStandardMenuItemClasses.pkg)
- Including file: cCJAboutMenuItem.pkg (C:\Program Files (x86)\DataFlex 19.0\Pkg\cCJAboutMenuItem.pkg)
- Including file: cCJMDIWindowsMenuItem.pkg (C:\Program Files (x86)\DataFlex 19.0\Pkg\cCJMDIWindowsMenuItem.pkg)
- Including file: oEditContextMenu.pkg (C:\Program Files (x86)\DataFlex 19.0\Pkg\oEditContextMenu.pkg)
- Including file: oDEOEditContextMenu.pkg (C:\Program Files (x86)\DataFlex 19.0\Pkg\oDEOEditContextMenu.pkg)
- Including file: StdAbout.pkg (C:\Program Files (x86)\DataFlex 19.0\Pkg\StdAbout.pkg)
- Including file: Dfabout.pkg (C:\Program Files (x86)\DataFlex 19.0\Pkg\Dfabout.pkg)
- Including file: DFBTRDRV.PKG (C:\Program Files (x86)\DataFlex 19.0\Pkg\DFBTRDRV.PKG)
- Including file: MSSqldrv.pkg (C:\Program Files (x86)\DataFlex 19.0\Pkg\MSSqldrv.pkg)
- Including file: CleanMarkers.vw (Y:\Libraries\CleanMarkers\AppSrc\CleanMarkers.vw)
- Including file: dirform.pkg (Y:\Libraries\CleanMarkers\AppSrc\dirform.pkg)
- Including file: akefs.pkg (Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg)
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 326) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 477) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 906) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 911) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 1007) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 1010) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 1033) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 1034) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 1035) Símbolo não definido em argumento ZEROSTRING
- Summary
- Memory Available: 2147483646
- Total Warnings : 0
- Total Errors : 9
- Total Symbols : 46796
- Total Resources: 0
- Total Commands : 82693
- Total Windows : 1
- Total Pages : 1
- Static Data : 521629
- Message area : 560783
- Total Blocks : 29630
- 9 ERRORS HAVE BEEN FOUND.
----Compile Finished (With Errors!)----
----Compiler Error Summary----
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 326) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 477) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 906) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 911) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 1007) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 1010) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 1033) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 1034) Símbolo não definido em argumento ZEROSTRING
- Error 4328: Y:\Libraries\CleanMarkers\AppSrc\akefs.pkg (ln 1035) Símbolo não definido em argumento ZEROSTRING

wila
8-Jun-2018, 03:21 PM
Hi,

Interesting.

I had specifically tested that for another project and it compiled and worked in DF19.0 for me, so I figured that was where it got introduced.
Guess it is new in DF19.1 then. This was also why I removed support for older versions of DF... If it only compiles in DF19.1 that's a bit too narrow.

You can use this for now to work around it:


// This is the one function that keeps DataFlex versions from before DF19.0 from using
// modern syntax, so we declare it instead.
Function ZeroString Global Integer iSize Returns String
String sRetVal
Move (Repeat(character(0), iSize)) to sRetVal
Function_Return sRetVal
End_Function


--
Wil

Fernando Faria
8-Jun-2018, 03:25 PM
wila,

It worked now. Thank you!

wila
8-Jun-2018, 04:48 PM
Both issues reported by Stephen as well as Fernando are fixed.

See the updated attachment in the first post.

--
Wil

Marco
8-Jun-2018, 07:02 PM
Hi Wil

Great project, but there is one bug I found.
One of the public domain libraries I use, has LF line endings.
The clean markers, appends all code into one line.

I've attached a source file, that also demonstrates this issue (I made it LF).

Kind regards
Marco

wila
9-Jun-2018, 02:05 PM
Hi Marco,

Thanks.

Umm.. personally I think that this falls outside of the scope of the project and I'm not sure if I would call that a bug.
The target is DataFlex 19.1+ source which is only available for Windows.
If I would patch that, then the next question is going to be to preserve the LF line endings...

If your library uses Unix line endings I'd say use a file editor and change the line endings to CRLF which is normally used on Windows.
It should be easy to change with a text editor such as: "EditPad Pro" or "Notepad++".

You might even want to contact the developer of the library and ask them to change the library to windows line endings, because chances are that the LF line endings are used by accident.

--
Wil

wila
9-Jun-2018, 04:28 PM
Marco,

Iin an earlier reply in the previous thread you asked:

1. I think the code left of the inserted ';' should be right trimmed.

This has been added in today's update (see the zip in the first post)

In addition to that I also added a choice between use a single semicolumn to break immediately behind the if statement


if (a>b);
Move 1 to c

and (which is the default) the option to break after inserting a space and then a semicolumn behind the if statement


if (a>b) ;
Move 1 to c


Then there's a couple of small UI tweaks.

Hope this helps,
--
Wil

Stephen W. Meeley
9-Jun-2018, 04:51 PM
Wil,

I don't know if it matters to you (or others), but just so it didn't go unsaid - we adopted an indent of 4 spaces on the line break (not 2). Do with that information what you wish :cool:

wila
9-Jun-2018, 06:04 PM
Hello Stephen,


Wil,

I don't know if it matters to you (or others), but just so it didn't go unsaid - we adopted an indent of 4 spaces on the line break (not 2). Do with that information what you wish :cool:

That's already configurable, via the tab stop width spinform at the top right, it even defaults to 4 (where I personally prefer 2).

So I'd say that has been taken care of.
--
Wil

Hans van de Laar
11-Jun-2018, 03:57 AM
Hi Wil,

First thanks for this great tool for cleaning up the old style DF code!

I tried one of my projects and run into the following issue:


If (iErrorNumber <> ERROR_NO_MORE_FILES and iErrorNumber <> 0) Send DoShowError iErrorNumber ("Search path: "+sFilePathMask)

gets converted into non compilable source code:



If (iErrorNumber <> Begin
ERROR_NO_MORE_FILES and iErrorNumber <> 0) Send DoShowError iErrorNumber ("Search path: "+sFilePathMask)
End


Is this something that can easily be fixed?

I've used the 'Begin/End' option in this case

Stephen W. Meeley
11-Jun-2018, 05:31 AM
Hi Wil,

I tried the latest version on a big test project and it worked great! Went from 14,313 warnings down to 8,427 in a matter of minutes. Nicely done!

wila
11-Jun-2018, 07:02 AM
Stephen,

Cool!

Also note that I just updated the project once more (see zip at first post)

The update adds
"Studio Integration".

This was a request from Marco and on our request Sture donated code for that.
Unfortunately it turned out to not be the best fit due to some specific function name clashes between our code and Sture's and having an external dependency on Sture's public library, while I preferred to keep things frugal and easy to install. Sorry Sture for the trouble.
BTW, do look at Sture's StureApsPublicLib (ftp://ftp.stureaps.dk/software/dataflex/stureapspubliclib/) It is Great!

In the end I borrowed some code from Nils's Manifest SideBySide (http://projects.vdf-guidance.com/projects/manifestsidebyside) project to register the Cleanmarkers app into the DataFlex 19.1 studio menu.

After doing that, I realized that it would probably be even more convenient if it could preselect the file you had open in the Studio and I ended up adding that as well.

Enjoy!
--
Wil

wila
11-Jun-2018, 08:47 AM
Hi Hans,

Sorry initially missed seeing your reply or I would have patched this in the first release today.

The reason you were seeing this is because when I added support for all current and legacy commands out there I saw that not all commands required a space at the end. (eg. the abort command can be followed up immediately by an end of line character)
So I figured to only test for a preceding space character.
Boy was I wrong :)

As you can see in your example it would break on " ERROR_NO_MORE_FILES" because the code detected an " Error" command.
But for an Error command to be valid it has to be followed up by parameters so it HAS the extra space I should have tested for.

As a result I have checked ALL the commands and added a space character when a parameter is required, hereby eliminating most of these type of problems.

This has been addressed in the latest current zip (the version 11c) and passed my testing when I tested your code in my test file.
--
Wil

Hans van de Laar
11-Jun-2018, 08:58 AM
Hi Wil,

Thanks!!!

Marco
12-Jun-2018, 02:35 AM
Hi Wil

I had to think a bit more about this. I do think it is a valid request though for the following reasons;
- Dataflex will compile LF code without problems, after cleanup it will fail with compiler errors
- Correct or not, a DataFlex source code could even have mixed LF and CRLF line endings, and it might not become immediately obvious how clean markers affected the code
- Version control systems like git can amend the way of line endings are stored and retrieved from the repository (for mixed environment teams)

Please note that I am NOT asking to maintain LF only line endings (which would be nice but ok with that), all I am suggesting to not making the 'valid' code fail.

At the very least perhaps just a warning 'Sourcecode file ... contains LF line endings', and skip the file from cleanup.

Sounds better?

wila
12-Jun-2018, 03:10 AM
Marco,

Never said it wasn't a valid request.
You are correct that the compiler has no problem with unix/linux file endings.

If you would have edited that same file in Windows notepad for the past 20 years, it would also have broken.

It is such a rare corner case that I don't feel like spending my free time on, need more than just one odd bug report on this one to convince me to spent a few hours on.
This is where for commercial software one says "we have logged your bug report".

So I agree with you, but will not be spending my time on it right now, there are better areas to spent that time on.

--
Wil

wila
12-Jun-2018, 06:16 AM
Hi,

Today's update has tighter integration with you file comparison tool of choice.

This is for the workflow where people want to clean up file by file and then compare via a file comparison tool what changes have been made.
First you setup your file comparison tool of choice:
12045

Then when run a file clean up you now have an extra button to compare:
12046

After you run the cleanup that compare button will call your selected file comparison tool while comparing the .cmbak (original) version of your file against the changed file.

--
Wil

Todd Forsberg
12-Jun-2018, 12:52 PM
Nice! I like that. :cool:

wila
14-Jun-2018, 08:52 AM
Another update:

Add support for files with LF instead of CRLF as end of line character. It will detect for the first line of the file if it is a file with LF or CRLF line endings and treats it accordingly. If otoh you have a text file with CRLF line endings and mixed that with LF line endings then you will still see the issue that Marco reported earlier. This is hard to fix while continuing to use readln without introducing other issues. A rewrite for using readblock is not planned for the moment.

Other changes are:

Support more file types by default: .bp, .mn, .h

Do not cleanup in folders with name: .svn, .git, .hg or AppHtml

Fix an issue with detecting files that have a file extension shorter than 3 characters as sometimes it would skip those files.

Added dependency on library vWin32fh (included in the zip and preconfigured).

--
Wil

wila
14-Jun-2018, 09:34 AM
Well I know I just said this:


If otoh you have a text file with CRLF line endings and mixed that with LF line endings then you will still see the issue that Marco reported earlier. This is hard to fix while continuing to use readln without introducing other issues.

But after stepping away and thinking about it some more there did turn out to be a relative easy solution for it by reading ahead a line.
By doing that it is now even able to correct a file with mixed line endings and will just rewrite it with CRLF, which imo is correct.
It also does not stick a CRLF on the end if there wasn't one before, which was one of the difficulties.

So version 2018/06/14a should have the "it works with any line endings in your source" problem resolved.

--
Wil

wila
14-Jun-2018, 09:35 AM
So fixed as well and yes it did eat too much time, as expected.

--
Wil

Marco
14-Jun-2018, 05:05 PM
Thanks Wil!
Much appreciated!

wila
22-Jun-2018, 04:26 PM
Hi,

Todd added Drag and Drop support for single files, the zip file at the first post has been updated with that patch.

Also note that Todd has helped me with the Subversion setup.
Last update the vWin32fh files had been moved to the library instead of embedding them into the project.
While that worked great for the zip as I can supply everything at once. Updating with Subversion did break your workspace as you suddenly missed out on the library.

I did not know that Subversion also had a feature for this called externals.
Eg. you can link an external project into your subversion repository and it will pull that external library in _at the same time_.

Really great feature and with that changed config the subversion repository also stays always up to date with the latest versions and works now again just as fine as the zip distribution does. Sorry for breaking that, but also a bit glad as I learned a better way to handling things.

Thanks Todd!
--
Wil

Garret Mott
22-Jun-2018, 05:58 PM
Another example of DF developers working together for the community! Thanks guys - it's a great thing.

Dennis Piccioni
23-Jun-2018, 12:01 AM
:)

DaveR
11-Jul-2018, 05:06 PM
Maybe I'm being picky, or it's a WIP bug, but running cleanmarkers on a folder seems to create a .cmbak whether there are changes or not. Makes seeing what changed a bit of a drag. Would it make sense to only generate the cmbak if changes are required? (or at least make that an option)

edit: or Am I supposed to look at file dates afterwards as those that have changed will have a later timestamp? seems counter-intuitive...

wila
13-Jul-2018, 08:26 AM
Howdy Dave,

You are absolutely correct. It makes no sense to leave those .cmbak files for a file that has not changed.
That has been addressed in the latest zip (see first post)

--
Wil

DaveR
14-Jul-2018, 09:01 AM
Just a thought, but because this tool is a work in progress and extra options keep getting added, I’ve run it multiple times on the same folder(s) and each .cmbak overwrites the previous.
Might it be more useful to create the backups in a timestamp-named subfolder so that multiple passes are kept separate? That would also allow the old sources to keep their original names, too.

Marco
14-Jul-2018, 06:37 PM
Hi Wil

I was thinking of instead of creating a CMBAK, issuing a git commit after each type of update.

So looping to do one fix at the time, then a
git add .
and a
git commit -m “<fix friendly name>”

Cheers
Marco

wila
15-Jul-2018, 04:40 AM
Dave,

Is it a work in progress? Not as far as I know, I stopped working on it as it does the job.
Only addressing bug reports at the moment as I've got plenty of other things on my plate.

Personally I don't use the .cmbak files as backup files, which is also why you bumped into this as it wasn't part of my testing much.
Instead I check my source into subversion.

What you're asking for means have to revise some of the core functionality.

The way it works is that on the start of processing the file it copies the original file into the .cmbak, overwriting any pre-existing .cmbak file.
Then it starts reading that .cmbak file, line by line and apply the requested changes to the output file, overwriting the original file.

Once done it adjust the timestamp if there had been no changes and deletes the .cmbak file if either requested or if there had been no changes.
--
Wil

wila
15-Jul-2018, 04:48 AM
Marco,

Yeah I am not a fan of the .cmbak file, but it has been there since the beginning. Actually it was a .bak file, I renamed it into a .cmbak so as not to meddle with a developer's original .bak files, it they existed.

The problem with your request is that not everybody uses git, some use subversion (like me), some use mercurial, some use Vault etc...
In other words if I would add such a feature then it would have to abstract the source code control tool.
Certainly doable, but it is a feature request, not a bug report and I am not extending functionality at the moment.

--
Wil

DaveR
15-Jul-2018, 10:35 AM
Wil, thanks for the response.

Yeah, I’d call it a WIP, even if there've only been three iterations in 12 years :)

this might be the kind of thing I’ll have a go at myself. Let’s see....

wila
16-Jul-2018, 07:56 AM
Update 2018/07/16: Added DataFlex 18.1 and 18.2 support. Also updated vWin32fh library to not use #IFNDEF for adding the global function zerostring workaround, but use the compiler revision (!@ <191) instead, so as to make sure we never use the workaround function in DF19.1+ (thanks Nils for pointing that out)

cheers!
--
Wil

Nils G. Svedmyr
16-Jul-2018, 08:45 AM
Good to hear, thanks Wil. I think this, albeit tiny change, will make the vWin32fh package more robust in all sorts of environments.