PDA

View Full Version : REST processes stopping - nothing in event log



Sjoerd
8-May-2020, 07:35 AM
For since a few months we are using a REST webservices. However during the day the processes are stopped (and automatically started) but why are they stopped?
We have build in a failsave that creates a record at the beginning of the function(s) and removes the record in the last line. The table is empty which lets us beleave the entire function is being executed.

When using the webservice with soap there was logging in the Event Log. With REST we haven't seen this so far. Should this be enabled somewhere?

Any thoughts?

starzen
8-May-2020, 08:00 AM
can you be more specific what the issue is? Are they not working?

when you call a REST service it calls the function executes it and returns.

Sjoerd
8-May-2020, 08:10 AM
Every night we restart our webapps. So all the processes should run all day for (eventually) 23:59:59 hour. But during the day a processes get terminated and restarted by the web application server.

We don't get complaints from customers that data is missing. It looks like all functions are executed and returning. But we can't explain the restarted processes.

Have a look at the added image. 1 hour and 38 minutes ago I manually restarted the webapp. However 10 and 52 minutes ago two processes are restarted as well.
We can't figure out why this is happening. The event log is empty (is it being filled when using REST?)

starzen
8-May-2020, 08:23 AM
2 things

first process pooling will add and remove processes as needed depending on setup

second there is a setting to automatically reload processes every so many hours

when they are sitting idle they dont do anything so they can be recycled. Probably to prevent memory loss from things being left behind etc.

Sjoerd
8-May-2020, 09:10 AM
2 things

first process pooling will add and remove processes as needed depending on setup

second there is a setting to automatically reload processes every so many hours

when they are sitting idle they dont do anything so they can be recycled. Probably to prevent memory loss from things being left behind etc.

Very true.
The automatically reload is set to reload daily at 0:55 --> not during the day
The process pooling is set to 8 up to 16. Doesn't this mean there are always 8 "on" and start more processes when needed? Or does it mean that when 8 processes are processing data and call 9 comes in there is started a (new) process 9 and when one of the first 8 is done this one is stopped (because there are 8 processes active)?

We are migrating from SOAP to REST. At the SOAP web service we never had these proces restarts.

Stephen W. Meeley
8-May-2020, 02:03 PM
When you start the webapp, the min processes are started. As traffic increases, so does the chance that all processes in the min pool will be busy when a call is made - that will trigger an additional process to start (up to the max setting). The additional process(es) will remain in the pool until the next sweep (which I believe happens every couple of minutes) and, if all processes are still active they remain, and if not, some will be terminated until the process pool is back to the min. The processes that get terminated are whichever ones happen to be idle during the sweep - so you can't count on the up time for any given process to be indicative of a problem (or lack of one). There are rare cases where the min and max pool are set the same and then all processes should have similar up times.

Sjoerd
11-May-2020, 04:38 AM
Thanks you Stephen. Setting the minimum and maximum processes to the same value seemed to solve this. Thank you!

starzen
11-May-2020, 04:45 AM
not really a good idea. there is nothing wrong with whats happening.

if you set the max down to the min you will now experience times where there are no processes available and users have to wait until a process becomes available.
if you set the min up to the max you are probably creating unnecessary processes.

the process pooling with the min is meant to provide a set of default processes that can handle the normal workload and the max is mean to allow for additional processes to be spawned in case there are times of higher usage.

the processes between min and max are going to be closed when not needed releasing resources used on the computer.

Stephen W. Meeley
11-May-2020, 05:33 AM
Just to be clear, I wasn't recommending that you set the min and max the same as a solution - just explaining what happens. If there are no errors and no bad effect from processes restarting, my guess is that there are merely times when the load is triggering a new process (expected) and then one or more of the original processes are being terminated (also expected) when the peak is past. You can look at the metrics of current, peak and total processes to confirm that is the case.

For instance, let's say your min is 8 and your max 12 - look at your peak processes to see if you ever exceeded your min and, if so, by how much. Exceeding the min is not a bad thing, it's the way process pooling is supposed to work. So lets say we see a peak of 10 - we now know that there were times when the load exceeded the capacity and we needed two additional processes to handle it.

Now look at total. If we see that total also equals 10 - we know that the peak load only exceeded the capacity once during the observation period, and process pooling did it's job and handled the peak. OTOH, if total equals 20 or 30, then we know that the min pool was exceeded a number of times and we may want to watch more closely to see if the min should be adjusted a bit higher. The higher the total processes, the more often additional processes needed to be started.

In all cases, as long as there are no errors in the webapp or windows event log, the coming and going of additional processes is not a bad thing.

As Michael mentioned, by setting the mix and max the same, you are forcing the application to pend requests that can't be serviced rather than starting up additional processes to handle them. Again, in and of itself not a terrible thing (it's the way the system is designed to work) - unless it happens a lot. Use process pooling under observation to determine if the pool settings are sufficient to handle even peak loads if you want to go the route of setting min and max the same.