7 Replies Latest reply: May 15, 2014 10:10 AM by Jim Weeks RSS

    So Far Data Pump is more trouble than its worth.

    Jim Weeks

      In the last several weeks, I have been transferring processes from .bat files run by Windows Scheduler to Monarch Data Pump (10.5).  I have experinced the following issues:

      1) The monitoring feature appears not to work in the manner I would have anticipated.  When I set up a new one, I basically have to either stop and restart the services OR reboot the server.  Then, if there is more than one .xprj file in the job the second one often errors out.  But I can run them fine if I just submit the process manually.  I have stopped all monitoring and am running those processes as scheduled ones.  Which renders the next issue moot.

      2) Why was it designed to not monitor for data base files?

      3) Several times, I have come in in the morning (like today) and there are 21 jobs in the queue and one process is "running" but is actually hung up. The log shows no errors.  I have to clear that job, reboot the server and then baby sit the processes as the jobs run.  And field the phone calls as to why reports are late.

      Over the next sebveral weeks, I'll be moving back to scheduled .bat files.

      Jim Weeks

        • So Far Data Pump is more trouble than its worth.
          joey

          There are a few times that Data Pump doesn't seem the most intuitive, and I've had to design a few solutions to get more functionality that I need, but overall I think it works better for us than scheduling batch scripts. That said, we've done the fighting with several issues and what we have works pretty good for the time being.  I haven't found tech support to be helpful with Data Pump, but they are Fantastic with Monarch. 

           

          Data Pump does have the advantage of error reporting and notification emails, which are a little more tricky than scheduled batch scripts.  There's plenty of other functionality that you can do yourself, but not in the same amount of time (at least I can't).

           

          Before you abandon Data Pump, you might consider using slowly rolling out processes to it, to keep the number of issues manageable. I've been where you are where it seems that nothing is working on it - and it is very stressfull, expecially when your manager is at your neck asking why the reports aren't available.  If you have one specific process that is giving you problems, maybe we can try to troubleshoot on the forum?

            • So Far Data Pump is more trouble than its worth.
              Olly Bond

              Hello Jim,

               

              I'd like to echo what Joey's posted - that yes, it's frustrating when things don't work, but that help is at hand. I've certainly always received helpful support from the UK office for any DataPump issues. I don't know how things work where you are, but we serve about five different departments with daily or weekly reports via about 50 processes. I discourage the model-builders from using monitoring wherever possible (see other threads here about monitoring stopping) - it seems safer just to schedule a job to run as often as required, and if the file isn't there, just skip the job.

               

              Best wishes,

               

              Olly

            • So Far Data Pump is more trouble than its worth.
              joey

              We do use monitoring at our company, and have almost 100 processes monitored. In total we have arround 125 jobs.  Monitoring had a few issues we had to discover and V9 made it much more usable. There was also an issue of networked files being monitored but with the forums help (Gareth) we're running pretty stable here.

                • So Far Data Pump is more trouble than its worth.
                  Jim Weeks

                  First, I must apologize.  Although it felt good, me venting is not a productive or appropiate use of the forum.  And I appreciate the responses back to my personal "data dump".

                   

                  I would agree that there are some excellent features with DP but I have to agree with Joey that the support for data pump is not as good as with Monrach.  They were learning along with me on the issues that I have used them for.

                   

                  I have switched all of my monitored processes to scheduled ones which seems to have settled down DP a bit.  However, even before I set up any monitored processes there were still issues.  The biggest one is that the data pump seems to "stall" which then causes other processes to queue up behind it.  It does not seem to have a specific job that stalls BUT it seems to happen mostly when a pre-process script is running.  Some of the pre-processes I run are rather lengthy as they are transferring raw data files through ftp.  It would appear that while a pre-process is running, another process will NOT start.  Is that correct or is there a setting somewhere that would allow another job to start?

                   

                  Also, when the server reboots, DP loads but does not automatically start.  I can't seem to find a place where I could set that differently.

                   

                  Thanks again for your patience and your help.

                   

                  Jim Weeks

                • So Far Data Pump is more trouble than its worth.
                  joey

                  Jim, it looks like you'll be getting the help you need from Datawatch, but I'd like to take a stab at answering your questions as well. Providing answers on the forums is also helpful for future people that need help.

                   

                  Data Pump is a windows service.  You need to open up services, and set Monarch Data Pump to have a startup type of Automatic, if it doesn't already. If you want to have the user interface start up when you log on, put a shortcut to it in your startup folder.

                   

                  Processes run until they are completed. This includes pre and post process scripts.  If you ever have a bug in your code and the script hits a loop you'll need to have a timeout limit set or else kill it yourself. You can set the maximum number of concurrent jobs in the Server Settings of Data Pump.  I think the default was two (though that was 5 years ago when we went to Data Pump).  We currently have it set to 10. 

                   

                  There is also the option for each process to run all jobs together or run them as separate jobs.  If you have one process with four projects, this will combine the four projects into a line line entry in your jobs list and one job log.  The job would take up only one of your concurrent job slots instead of four.  I check this for all processes, but it is not the default.

                   

                  Hopefully these can alleviate your bottleneck somewhat.

                   

                  Also, monitoring for files can help make scheduling easier.  If the job that produces the input file changes schedule, then Data Pump adjusts automatically.  At our company most of the reports are created on the mainframe at different times of the night. Starting the jobs exactly when a given report is available helps spread out the processing and avoids bottlenecks. That's just my thoughts on why monitoring may be worth your time.

                   

                   

                  Hope this helps.  Best of luck with Datawatch.

                    • So Far Data Pump is more trouble than its worth.
                      Jim Weeks

                      Hello, all.

                       

                      I had a very nice conversation with Gareth who was VERY helpful.  That, together with not using monitoring, has caused DP to run quite smoothly.  I will walk before I run when I get back into monitoring.  I do have another issue that has cropped up this weekend but I'll start a new thread for that.

                       

                      Thank you all for your patience!

                       

                      Jim Weeks