PDA

View Full Version : aws amazon cloud service for dataflex webapp



YASA NATADITIA
31-Jul-2015, 09:41 AM
Hi

I'm yasa

I wondered if dataflex web based application could be run within aws amazon.

do dataflex web app could able be run within aws amazon?

should dataflex web app database must be configure ready in ms-sql/my-sql/oracle/etc before run within aws amazon?

do dataflex web app using datflex native database could able be run in aws amazon cloud service?

Regards
Yasa

chuckatkinson
31-Jul-2015, 10:11 AM
Hi Yasa,

Yes DataFlex web apps run very well on Amazon AWS. You can use both embedded and SQL backends on AWS.

Data Access has a Amazon Cloud Starter kit Amazon Machine Image you can use to launch a fully configured instance with DataFlex 18.0 (currently) and DataFlex Reports. This is ready to run AWS configuration. We are currently in the process updating it with DataFlex 18.1 version. Please contact DAW to get more details on availability and pricing.

Bob Cergol
31-Jul-2015, 04:49 PM
Hi Yasa,

Here (http://www.dataaccess.com/LiveDynamicAI) is an live example of a Dataflex WebApp running from an AWS server. The data is in an MS-SQL Server 2012 also running on the same server.

Bob

YASA NATADITIA
31-Jul-2015, 07:46 PM
Hi Chuck

Thank a lot for the informations

Regards
Yasa

YASA NATADITIA
31-Jul-2015, 07:46 PM
Hi Bob

Thanks a lot for the informations

Regards
Yasa

YASA NATADITIA
31-Jul-2015, 07:48 PM
Hi Chuck

How do I find starter kit in Amozon Aws. Is there any keyword there?

Regards
Yasa

chuckatkinson
3-Aug-2015, 04:51 PM
Yasa,

It is not currently published in the open Amazon Marketplace. You will need to contact your local Data Access resource and/or Fern at the main Data Access office.

YASA NATADITIA
4-Aug-2015, 05:13 AM
Hi Chuck

Yes I have search it and not found.
Thats ok. I get acquainted with mr Fern. I will contact him.
Thanks a lot

Regards
Yasa

mooner
4-Nov-2015, 03:28 PM
Why did you not use a SQL server RDS instance to create this example, would this not be a better use of what AWS provide in terms of redundancy and scalability as a real world example?

Has anyone done any testing in this area at all?

We have just migrated our webapp to AWS (sydney) using the dataflex backend on a local machine, but this is again not great using a local dataset instead of being able to replicate and have redundancy available in case of an availability zone goes down etc. We decided to move first and restructure later after initial testing went well.

One issue we currently have is we are running in to some constraints in regards to the IOPS we can consume through some of our dataflex processes, we have over provisioned general purpose SSD volumes but I believe the large number and small size of the database read/write requests is limiting the maximum throughput we can achieve in a few key parts of our system.

I am new the the forums and am yet to have a good look around, are there any tuneable parameters within the dataflex backend? Or are there optimum block size volumes to use to gain the most out of the system? We have never had concerns previously when running a physical machine and have just run with the defaults but now I am curious what else can be tweaked so I can do some further testing to try to achieve better performance on AWS in our current configuration.

FYI we are running VDF 18.0 on Server 2012 R2 on T2 and C4 instance types hosted in Sydney Australia. I am very new to AWS but learning fast...

Bob Cergol
4-Nov-2015, 03:42 PM
I'll defer tp Chuck regarding pros and cons of the increasing number of hosting options Amazon is providing. I'm not sure what defines "better" and I know the cost variables are numerous.

The instance used is a server set up several years ago as a generic public demo server. Its an m3 medium instance and running MS-SQL Server 2012.
I don't recall what the choices were back then and I wasn't the one who set it up. I only asked to have IIS and SQL server running.

Bob

chuckatkinson
4-Nov-2015, 04:31 PM
I have a lot of experience using RDS on AWS.

You should know that RDS is slightly expensive on AWS. But it is really fast if configured properly. The advantages are automatic backups, Multi-Zone (replication) and Amazon applies all patches as part of the service.

What kind of SQL Server are you running locally ? Is it perhaps SQL Express ? If so then as you know Express has limitations on number of processors and also memory that can be used. RDS can be any version of SQL all the way to Enterprise. And much like your EC2 instances you can choose different instance types along with processors and memory. I have a client running on an m3.large and it is more than sufficient for a 40 user Windows desktop and and a separate web application with 100's of users. Also this client is a heavy user of Crystal Reports.

Don't forget you can set up CloudWatch to monitor metrics on the instances.

Samuel Pizarro
4-Nov-2015, 04:34 PM
One issue we currently have is we are running in to some constraints in regards to the IOPS we can consume through some of our dataflex processes

are you measuring the IOPS from your regular app workload? Or an specific job runing isolated? Or What testing the real IOPS your hardware can give using specific benchmark tools?

if measuring your df webapp, do you have centralized system tables to hold auto-increment values for your tables?
we had issues with that in a bigger webapp (more than 60 tables) and several concurrent webusers as the system table locking becomes a bottleneck! As it has to be locked too often and is involved in almost all transactions.

maybe your case as well?

we moved the tables to a SQL DBMS and converted the traditional Df auto-increments to the backend db auto-increment columns. So you don't have a single lock spot anymore involved on every transaction!
Regards

mooner
5-Nov-2015, 03:37 PM
Hi Chuck,

Thanks for the reply, we are using Dataflex Native DB, not SQL at this stage.

I believe RDS with SQL server would be the best way forward for our WebAPP but I would say I have a lot of learning and testing to do before that day comes.

I have been using cloudwatch to monitor our instances and finding our throughput in IOPS is not matching our capacity. I would say Samuals response in regards to locking might be on the right track, I have recently started looking at the output from procmon and am seeing larger numbers of locking and unlocking of many tables during our application running.

I will follow up some more details on his response.

Thanks,

Mike

mooner
5-Nov-2015, 04:08 PM
Hi Samuel,

Thanks for replying. I have not be able to put a great test case together, we do have a single task which we know really affects the system performance though.

We are still running the native dataflex backend and have not yet looked at upgrading to SQL although it is something I wish to achieve in the future.

For some background we have a webapp running (up to 20 concurrent processes in our process pool), but along side this on the server are 4 dataflex applications which are accessing the same database doing system tasks and responsible for printing and emailing and importing and other various tasks that happen in our application.

The intensive task I mention is running in one of our application exe's. It is updating a single large table many many times shifting numeric data from field to field.

While this is running the other exe's all start to slow down, and their usual simple tasks take much longer, it appears they may be waiting for locks on tables and then each one consumes very high amounts of CPU whilst waiting causing the entire system to slow down.

Not specific to the above intensive task I mention, I have been watching procmon on a couple of our instances and am seeing huge numbers of tables locking and unlocking for what appears to be a single table update. I am unsure if the mechanism for locking and unlocking is being handled correctly in our application. does this scenario sound normal in a dataflex context?

In these applications,
1/ all tables appear to be opened at the start of executing and are not closed until application is closed (I believe this is bad practice)
2/ a record is found via index requiring updating.
3/ a Reread for this table is called.
4/ fields are updated
5/ Saverecord is called
6/ Unlock is called
(simple example shown below)

Find eq oDayStat by index.1
[Found] Reread oDayStat
Move (oDayStat.Quotes_Sent + 1) to oDayStat.Quotes_Sent
Saverecord oDayStat
Unlock

PS I am not a dataflex developer, I am simply trying to "fix" the application.

Would the above example cause all open tables to lock and then unlock as I am seeing through procmon? Should all tables be closed unless being updated or read from? What is the impact of opening and closing tables constantly?

Any advice given is greatly appreciated, I am sorry this is probably non specific to amazon cloud solution and more a simple dataflex question.

FrankValcarcel
5-Nov-2015, 04:14 PM
Heavily transactional system will really benefit from going to SQL which gives you record locking. We did some tests a number of years ago on the biggest RS6000 available and were limited to 1700 orders per minutes (our test was order entry) with the native database. With DB2 (and SQL will be similar) we went up to over 10,000 per minutes.

Now why we were testing - our customer who was a competitor of Ingram Micro ( who were doing 5 Billion in sales) was concerned about out growing our system. So we went to IBM and tested. When we cam back with the results they asked "What happens when we get bigger?". Shortly thereafter I talk to someone in the Ingram IT group who advised that Ingram was doing 17 (seventeen) order per minute (average, 30 peak) and had a long term plan to support 30 average 60 peak. Oh Well.

Dennis Piccioni
5-Nov-2015, 04:43 PM
Well one issue I see is that the reread is only done conditionally, while the rest of the code is not. Reread does a lock as part of it, so if the record is not found, no reread or lock happens, while the rest of the code does. I recommend putting everything from reread to unlock in a Begin..End block:

Find eq oDayStat by index.1
If (Found) Begin // code change from [Found] to (Found) is irrelevant in this case, but it's more modern code
Reread oDayStat
Move (oDayStat.Quotes_Sent + 1) to oDayStat.Quotes_Sent
Saverecord oDayStat
Unlock
End

mooner
5-Nov-2015, 04:56 PM
So is the outcome of a index match not being made in my example that a different record might be updated in its place? Or would the record pointer be at the end of the file and a new record be created? Or something else entirely??

Thanks for your help,

Mike

Samuel Pizarro
5-Nov-2015, 05:16 PM
How many tables in the file-list do you have ?

Performing transaction outside the Data-Dictionaries like this you have, rereading and unlockiing the table directly, can HAVE a huge impact in LOCKING the entire system if this process takes long to execute.



Find eq oDayStat by index.1
[Found] Reread oDayStat
Move (oDayStat.Quotes_Sent + 1) to oDayStat.Quotes_Sent
Saverecord oDayStat
Unlock


When you execute a REREAD <table> , in fact DATAFLEX LOCKS all tables opened in NORMAL-mode. not only the table you are trying to update!
And all of them will got UNLOCKED, when you issue the Unlock cmd.

So, if this process you have is LONG (minutes), and in this particular program you have more or all tables OPENED, all of them are being LOCKED/UNLOCKED, LOCKED/UNLOCKED, LOCKED/UNLOCKED, over and over, and over, for the entire loop until your batch -job completes.

That could explain why you see a slown-down in performance in the other .exes you have !

Doing the same thing usind DDOS, inside a business-process does not have the same impact, becuase the DDO has an smart-logic to CLOSE or change the OPEN-MODE of all other tables that does not partitipcate in your transaction logic to READ-ONLY. so if you are updating a single table, only few tables that belongs to that particupar transaction will be locked, and not ALL of the others like before!

Another approach, is if you don't want to change your code too much (convert it to start using DDOs, and Business-Process) , you may want to try to change the open mode for all other tables to READ-ONLY, before starting your procedure, and setting them back to normal at the end. This way, only the target table will got locked during your job.

I suggest you to read the Locking mechaninsg in the help. It explains this better than I !

Good Luck

Regards
Here

Dennis Piccioni
5-Nov-2015, 05:23 PM
It depends on a few variables, but most likely that code would try to create a new record, which may or may not succeed, based on whether all required index segments are filled in.

mooner
5-Nov-2015, 08:42 PM
We have about 450 files in the filelist so its a fairly big system, not all tables are always open for every process though but most are.

Can I find a version of the help online? I am curious how you set the open-mode of a table while it is currently open. Also we do have DD definitions for our data files, no table is opened using a simple open command from what I can tell. Would implementing the smart-logic be relatively straight forward?

Thanks again for your help.

FrankValcarcel
5-Nov-2015, 09:04 PM
You set the file mode with

FILE_MODE filename Mode (Default, RO, SU) see file_mode doc

You cannot change it in the middle of a transaction. Before we went to SQL we would set all the files read only and then set the ones needed to default. I advise against you doing this since SQL Express is free/AWS SQL is cheap and the amount of code you have to touch is high with too many chances for error. We do most of our work outside the DDs and just leave all files set to Default as the server only locks the RECORDS used.

Garret Mott
5-Nov-2015, 11:47 PM
While not a permanent fix, Tranlock.pkg can minimize the effect of file locking. http://support.dataaccess.com/Forums/showthread.php?936-TranLock-Pkg-Version-2&highlight=tranlock.pkg

As has been said, the best way is to 1) use DD code & 2) go SQL.

mooner
8-Nov-2015, 05:12 PM
Thanks again everyone for your very helpful replies :)

We do currently use DD rather then straight file open in the application, is it then as simple as we need to add a line of code to each DD construct to enable smart mode and the rest is handled for us in the background?

mooner
8-Nov-2015, 05:24 PM
Hi again,

Just thinking some more about my last post, for this to work would every database transaction have to be a method of the DD rather then what is currently being done where database transactions are created in the main code outside of the DD? Is this how the smart locking is able to function correctly, and why changing to this method was considered time consuming from a system which currently doesn't implement it.

Samuel Pizarro
8-Nov-2015, 06:53 PM
The code could be written anywhere. In the body of a view, inside a button, or business process, or a method inside a Data-Dictionary.

The place does not matter.

What we are talking here is a HUGE difference in the approach of coding. so YES, you will have a lot of work to convert your code.

Basically, you need to stop using the Command interface.
from:


find GT <table> by Inxex.x
Lock / Reread <table>
SaveRecord <table>
DeleteRecord <table>
Move <value> to <table>.<Field>
Unlock



to the Data-Dictionary Methods


Send Request_find of <oDD> ...
Set Field_Changed_Value of <oDD> ...
Send Request_Validate of <oDD>
Send Request_save of <oDD> ...
Send Request_delete of <oDD> ...


Respectively!

When you execute a Request_save of <oDD> , instead of the SaveRecord command directly, the Request_save logic (and bunch of other internal DD methods) will change the open mode for all tables not invovled in your transaction to READ-ONLY, and when the RERAD/LOCK cmd get executed (also internally inside of DD class methods), those tables will not be locked anymore!

Regards

Garret Mott
8-Nov-2015, 07:32 PM
Just to be picky...



Send Request_find of <oDD> ...
Set Field_Changed_Value of <oDD> ...
Get Request_Validate of <oDD> to bErr
If (not(bErr))Send Request_save of <oDD> ...

Samuel Pizarro
9-Nov-2015, 10:39 AM
Oops!

thanks Garret!