20 Replies Latest reply: May 15, 2014 10:06 AM by joey RSS

    Monitored Processes

    Tim Racht

      I am interested to hear from other MDP users how many monitored processes they have set up on their MDP server?  We have approximately 45 monitored processes and would like to know if this could put a burden on our server.  We would appreciate feedback as to what version you are on, how many monitored processes you run and indicate if you had to revise your registry settings to accomodate the MDP to run effeciently.

        • Monitored Processes
          joey

          We are on 9, but looking to upgrade to 10 as soon as it arives. We have well over 100 monitored processes (166 total processes on the machine). Server is 3Ghz processor with 1GB of ram. We dont' have any performance issues with the machine. We don't do anything too crazy, like process files over 100 MB in size.

           

          Of course, we have Data Pump on a dedicated server.

            • Monitored Processes
              Chickenman _

              We have around 125 monitored processes running under Windows Server 2003. Dedicated server is 3.16 GHz and 2 GB RAM.

               

              About 10 files are submitted via an automatic nightly schedule and users intermittently submit perhaps an average of 10/day on demand, with a given process taking a few seconds to a max of a few minutes, so we consider this very light duty for the server.

               

              We have received version 10 and will install shortly coincident with a planned move of the server. We hope (but are not optimistic) that the upgrade to V10 cures a chronic and very annoying issue: the majority of the processes run fine but sporadically, users' on demand submissions fail to trigger and must be run manually. Quite annoying.

               

              MDP throws hundreds of error messages for no valid reason that we can discern. Tech support provided a couple of weak "try this" recommendations but we haven't heard from them since September despite our prompt feedback. Ya think they are stumped?

               

              It is a very powerful product, but support is non existant so YMMV. :mad:

               

              CM

                • Monitored Processes
                  joey

                  Chickenman out of curiousity, could you post a few of the error messages?

                    • Monitored Processes
                      Chickenman _

                      Chickenman out of curiousity, could you post a few of the error messages?[/quote]

                       

                      Sure.

                       

                      Event ID: 210[/FONT]

                      /FONT

                      DwchServer. InvalidNoOfFileSpecsForMonitoredProcessException: Too many file specsifications for monitored process .[/FONT]

                        at DwchServer.a.c[gA_0, Server DBA_1][/FONT]

                       

                      It threw about 100 (!!) of these messages at 0524 yesterday, another 200 at 0633, another 100 an hour or so later and another 100 an hour after that. No files were sent to the monitored folder early yesterday and there seems to be no pattern, rhyme nor reason.

                       

                      CM

                    • Monitored Processes
                      Gareth Horton

                      Hi,

                       

                      I see you have got some of the information already, but here are two posts you should check out:

                       

                      This one concerns some of the tweaks that can be made to MDP regarding monitoring

                       

                      http://www.monarchforums.com/showpost.php?p=9826&postcount=6[/url]

                       

                      This one is the source of the information from Dave G, regarding some Windows limitations.

                       

                      http://www.monarchforums.com/showpost.php?p=10188&postcount=8[/url]

                       

                      I would impress on you the level of overhead of monitoring on the system, which is not purely related to the the amount of actual file activity that may take.

                       

                      Monitoring for 125 processes is going to put considerable strain on the system, even if it is only monitoring for one folder / filespec per process.

                       

                      In addition, if the load is too great, the communication between MDP and the back-end database could be affected, if SQL Server cannot consume information and respond back to MDP in a timely manner. This can obviously affect the monitoring itself.

                       

                      Another option may be to use scheduling instead, in a kind of polling workaround.  This is not optimal if you have heavy traffic of identically named files coming into a folder, but otherwise should be OK.

                       

                      Set up a Schedule for say, every 10 minutes and then set the retry settings in the Process Properties to avoid constant job failures.

                       

                      This polling method will be a much lower load on the system.  Perhaps you could do a mix of monitoring and scheduling, depending on the importance of the process in question.

                       

                      Gareth

                       

                       

                      We have around 125 monitored processes running under Windows Server 2003. Dedicated server is 3.16 GHz and 2 GB RAM.

                       

                      About 10 files are submitted via an automatic nightly schedule and users intermittently submit perhaps an average of 10/day on demand, with a given process taking a few seconds to a max of a few minutes, so we consider this very light duty for the server.

                       

                      We have received version 10 and will install shortly coincident with a planned move of the server. We hope (but are not optimistic) that the upgrade to V10 cures a chronic and very annoying issue: the majority of the processes run fine but sporadically, users' on demand submissions fail to trigger and must be run manually. Quite annoying.

                       

                      MDP throws hundreds of error messages for no valid reason that we can discern. Tech support provided a couple of weak "try this" recommendations but we haven't heard from them since September despite our prompt feedback. Ya think they are stumped?

                       

                      It is a very powerful product, but support is non existant so YMMV. :mad:

                       

                      CM[/QUOTE]

                        • Monitored Processes
                          Chickenman _

                          Thank you for the response - we'll look at your suggestions.

                           

                          Scheduling on one of the chronically failing processes doesn't seem possible since the user submits 2 or 3 versions of the report serially within a very short time.

                           

                          This had worked perfectly but is now sporadic - just this morning the user notified he didn't receive output so we ran manually. But on Monday this same user submitted reports and the process triggered.

                           

                          We think our only hope is rehosting on a new server and upgrading to V10. We'll see.

                           

                          CM

                            • Monitored Processes
                              Tim Racht

                              We are still having monitored processes that do not process for no apparent reason.  We have noticed this message in Event Viewer System and would like to know if this would have something to do with our processes not kicking off.

                               

                              The description for Event ID ( 1 ) in Source ( LGTO_Sync ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: , Sync Stop done.

                                • Monitored Processes
                                  Gareth Horton

                                  Tim,

                                   

                                  Yes, it might well be that there are IO issues caused by the VMWare driver that mean the Win32API is not reporting all events correctly.

                                   

                                  "Filesystem Sync Driver (LGTO_SYNC - can halt I/O disrupting services/users) "

                                   

                                  See point 7 on this page:

                                   

                                  http://theether.net/kb/100072[/url]

                                   

                                  Here is some information on disabling it

                                   

                                  http://theether.net/kb/100017[/url]

                                   

                                  Gareth

                                   

                                   

                                   

                                  We are still having monitored processes that do not process for no apparent reason.  We have noticed this message in Event Viewer System and would like to know if this would have something to do with our processes not kicking off.

                                   

                                  The description for Event ID ( 1 ) in Source ( LGTO_Sync ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: , Sync Stop done.[/QUOTE]

                              • Monitored Processes
                                ServerGuy _

                                BIG time Thank you Gareth!! & this thread for resolving Event Viewer Error 210 (Too many file specifications for a monitored process) after upgrade from V.9 to 10.5.  Fought this battle for about a week and a half before resorting to the forum's for desperate help. :confused:

                                 

                                I used Gareth's link http://www.monarchforums.com/showpos...26&postcount=6[/url] to resolve the problem. Had to create these DWORD values because they DO NOT EXIST by just installing Datapump![/B]

                                 

                                "There are two DWORD registry settings in HKEY_LOCAL_MACHINE\SOFTWARE\Datawatch\DWCH Server that tell the Monitor how many threads to create, NumberOfMonitorThreads[/U] and how many file specifications to monitor within each thread, NumberOfFileSpecsPerMonitorThread[/U]."[/B]

                                 

                                At first I made the value 10 as he said was the default.  Recycled the registry & Datapump service and the 210 errors still popped up.  I then made it 200 after not sure how many monitored processes the users had.  I believe we have nearly 210(in quanity) monitored process.  I thought this might be a direct correlation but through much trial and error I found the threshold at which the 210 Errors disappear.  27 seems to be the number for an error free Event Viewer after stopping starting Data Pump for us.  I currently set it to 50 for some room to grow.

                                 

                                I have NumberofFilesSpecsPerMonitorThread set to 10 for now.

                                 

                                Why the installation doesn't install these registry keys is beyond me. :eek:    

                                 

                                I even went to the work of moving this installation to a new server.  It is currently running on VMWare Windows 2003.  4Gb RAM.

                                 

                                Hi,

                                 

                                I see you have got some of the information already, but here are two posts you should check out:

                                 

                                This one concerns some of the tweaks that can be made to MDP regarding monitoring

                                 

                                http://www.monarchforums.com/showpost.php?p=9826&postcount=6[/url]

                                 

                                This one is the source of the information from Dave G, regarding some Windows limitations.

                                 

                                http://www.monarchforums.com/showpost.php?p=10188&postcount=8[/url]

                                 

                                I would impress on you the level of overhead of monitoring on the system, which is not purely related to the the amount of actual file activity that may take.

                                 

                                Monitoring for 125 processes is going to put considerable strain on the system, even if it is only monitoring for one folder / filespec per process.

                                 

                                In addition, if the load is too great, the communication between MDP and the back-end database could be affected, if SQL Server cannot consume information and respond back to MDP in a timely manner. This can obviously affect the monitoring itself.

                                 

                                Another option may be to use scheduling instead, in a kind of polling workaround.  This is not optimal if you have heavy traffic of identically named files coming into a folder, but otherwise should be OK.

                                 

                                Set up a Schedule for say, every 10 minutes and then set the retry settings in the Process Properties to avoid constant job failures.

                                 

                                This polling method will be a much lower load on the system.  Perhaps you could do a mix of monitoring and scheduling, depending on the importance of the process in question.

                                 

                                Gareth[/QUOTE]

                          • Monitored Processes
                            joey

                            Interesting that there is a typo in the error message

                             

                            It looks like this is something Gareth will need to have cleaned up. It may already be fixed in version 10.

                             

                            I've found that that Tech Support offers little, if any help when things go wrong.  However, it's ususally worth the time to report the error, and if possible the files needed to reproduce a problem.  I've had several bugs that they have passed onto Gareth after confirming a problem.

                              • Monitored Processes
                                Chickenman _

                                Interesting that there is a typo in the error message

                                 

                                /quote

                                 

                                Oh? We thought that was the Queen's English spelling...:D

                                 

                                CM

                                • Monitored Processes
                                  Tim Racht

                                  Yes we do experience the failure to trigger any processes as they do not run.  We are running on a virtual machine with 2g ram.  In our case we have approx 48 processes and less than that are monitored.  We were doing some testing with our IT department and a funny thing occurred.  We had seven process fail to trigger, yet the reports the processes were monitoring were there.  As soon as we disabled three of the monitored processes, then this triggered all of the processes to run successfully.  If you have any ideas or suggestions as to what is going on would be greatly appreciated.

                                   

                                  Tim

                                    • Monitored Processes
                                      Chickenman _

                                      Yes we do experience the failure to trigger any processes as they do not run. We are running on a virtual machine with 2g ram. In our case we have approx 48 processes and less than that are monitored. We were doing some testing with our IT department and a funny thing occurred. We had seven process fail to trigger, yet the reports the processes were monitoring were there. As soon as we disabled three of the monitored processes, then this triggered all of the processes to run successfully. If you have any ideas or suggestions as to what is going on would be greatly appreciated.

                                       

                                      Tim[/quote]

                                       

                                      This mimics our failure in that the monitored report is there, but it happens to us on a solitary report sporadically, never when multiple processes might get "confused"

                                       

                                      Further, disabling/re-enabling the process sometimes[/U] triggers, but usually not. We have gone through lots of hoops for nearly two years and the solution defies being found, so no, I can't offer any believable suggestions here. Except contact support and let them ignore you... :eek:

                                       

                                      CM

                                        • Monitored Processes
                                          Dave-G _

                                          Tim, what you are describing happened to us. We had the same error message that appeared mulitple times. Currently, we have over 160 monitored processes with 500 total processes. Our system has this configuration:

                                           

                                          OS is Windows 2003 Server Standard Edition SP3

                                          4GB RAM

                                          Dual Intel Xeon - quad core 2.8Ghz processor

                                           

                                          Monarch Data Pump Version 9.0.0.14

                                           

                                          The server Data pump resides on also contains the files it uses.

                                           

                                          We are using Microsoft SQL Server 2005 Express SP3

                                          The SQL Database, DWCHHServer, is local on the machine.

                                           

                                          What we did was to add two registry keys that we did not have. We got this from Datawatch's tech. support.

                                          1. MaxCmds is 500 (decimal) or 1f4 (hexadecimal)

                                          2. MaxMpxCt is 500 (decimal or 1f4 (hexadecimal)

                                           

                                          1.     HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanworkstation\parameters

                                           

                                          In the right pane, double-click the MaxCmds value.

                                           

                                          In the Value data box, verify the value

                                           

                                           

                                          2.     HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters

                                          In the right-pane, double-click the MaxMpxCt value.

                                          In the Value data box, verify the value

                                           

                                          Since we added those changes, we have been error 210 free and monitored processes have started when they should.  I am not a tech. person, but just a user of Datapump and know what you are going through. Hope I have been helpful.

                                            • Monitored Processes
                                              Chickenman _

                                              We are in the process of re-hosting to a new server and have upgraded the target to V10 which we hope eliminates any problems.

                                               

                                              But if they return, we will surely try this

                                               

                                              That was potentially a very helpful post.

                                               

                                              CM