10 Replies Latest reply: May 15, 2014 10:00 AM by Grant _ RSS

    DPF'S AND DATAPUMP

    Ontiveros _

      I AM USING DATA PUMP AND MONARCH (BOTH V6). THE MODEL SET UP WITH A REPORT I HAVE WORKS FINE BUT WHEN I USE SET UP A DPF USING THE SAME REPORT AND MODEL, THE FILTER DOES NOT WORK EVEN THOUGH I SELECTED IT IN THE WORKFLOW-PROCESS OPTIONS. WHY IS THIS HAPPENING?

        • DPF'S AND DATAPUMP
          Ontiveros _

          I AM TRYING TO FILTER RECORDS FROM A FILE MAINTENANCE REPORT AND PUT THE RESULTS IN ANOTHER FOLDER FOR USERS TO RETREIVE WHEN THEY COME INTO THE OFFICE IN THE MORNING.  DATA PUMP SHOULD BE ABLE TO DO THIS AT A SCHEDULED TIME. THE WEIRD THING IS THAT I DON'T GET AN ERROR MESSAGE.  IN FACT, I CHECKED THE DPF.LOG FILE AND IT SAID THAT THE FILTER RAN.

          IF I DON'T USE A FILTER, I GET APPROX. 50000 RECORDS. WITH THE FILTER, I GET 1000 OR SO.  I CAN SEE NO LOGICAL REASON WHY THE FILTER ISN'T RUNNING USING THE DPF IN THE DATA PUMP ADMINISTRATOR.

          • DPF'S AND DATAPUMP
            Ontiveros _

            I have contacted Rose Cavaretta in the Tech Support area and they are analyzing it.  I was just trying to see if anyone else has had the problem or knows of why it would do that.

            • DPF'S AND DATAPUMP
              Ontiveros _

              Will Do.

              • DPF'S AND DATAPUMP
                Jim Weeks

                When you say it the filter is not running, do you mean that you get an output file but it is empty or do you not get a file at all?  Perhaps I have been experiencing a similar problem....I have some large filters running from .prj calls

                in a .bat file.  By large, I mean it takes a while for them to run.  If I run them manually, that is open the project from Monarch and the use project export, it works fine.  But from a .bat file, its almost like the project export kicks in before the open and filter has completed and it outputs a file with no records.  Does that sound similar?

                • DPF'S AND DATAPUMP
                  Ontiveros _

                  THE ISSUE I AM HAVING IS THAT DATAPUMP BASICALLY IGNORES THE FILTER SET UP IN THE DP ADMINISTRATOR.  I HAVE IT SET UP TO OPEN THE REPORT WITH A PARTICULAR MODEL. THE THE REPLY I RECEIVED FROM DATAWATCH WHEN i INQUIRED ABOUT THE PROBLEM IS :

                  There is a bug in version 6 of Data Pump that limits the expression in the filters and calculated fields to 1024 characters.  In Monarch this limit is at 4095 so this is why your filter works in Monarch but not in Data Pump.

                  This issue does not exist in version 7, so you may want to consider an

                  upgrade to v7.

                   

                  OUR COMPANY HAS ONLY HAD DATAPUMP AND MONARCH ON A SEPARATE SERVER FOR 6 MONTHS.  MY QUESTION IS, "WHY WOULD THEY SELL OUR COMPANY PRODUCTS THAT THERE ARE KNOWN BUGS IN RATHER RECOMMEND THE UPGRADED VERSION.

                  • DPF'S AND DATAPUMP
                    Darren _

                    Hey everyone,

                     

                    I have experienced this problem as well when using Monarch 6 and MDP 6 when automating larger reports. I have seen this same scenario happen to me when I have the following events set in motion:

                     

                    1). I have a filter in place - same as Ontiveros is talking about. Whereas the .dpf runs but misses the filter. When run manually everything is fine. When run automatically through MDP, the filter is missed.

                       

                    2). I have some sort of formatting in place in the Monarch model (summary). Again, when all the automation is run from Momnarch to MDP, the formatting is missed.

                     

                    This is only the case when really large files are executed through the process.

                     

                    The resolution to my problem was that I had set my MDP Process Scheduler to look for the report every 1 minute. If I changed the Execution Frequency time frame from 1 minute to 5 minutes or longer, these problems went away.

                     

                    I always kind of wondered about this scenario. Ontiveros, please let me know what your Execution frequency is for your .dpf's...

                    • DPF'S AND DATAPUMP
                      Ontiveros _

                      MY EXECUTION FREQUENCY IS DAILY AT 3:00 A.M.

                      • DPF'S AND DATAPUMP
                        Darren _

                        OK, fair enough. So if your Execution Frequency is set to run only once a day, then my scenario is not valid. Now that I understand your case better I think I've run into that scenario too. My workaround was to filter it elsewhere, like in Excel, but still this does not resolve the problem within MDP 6. Interesting to know that it's actually a documented bug in ver. 6.

                        • DPF'S AND DATAPUMP
                          Ontiveros _

                          A solution I found was to break up the filter into groups.  For example, say you have a filter that is over 1024 characters long. Use an IF statement in a calculated field. Here is an example of what I have made one of my calculated fields:

                          If(.NotIn.("SERVICE CODE:","QUANTITY:","PER ITEM AMOUNT:","TOTAL AMOUNT:","TRANSACTION DATE:","DESCRIPTION 1:","INTEREST CYCLE:*","ACCOUNT STATUS CODE"),"1","0").  This will give the value of 1 in the calculated field if values in the field are not in the "not in" statement.  In the actual filter of the model, you just need to specify whether you want the calculated field to equal a 1 or 0. It worked for me anyways.

                          • DPF'S AND DATAPUMP
                            Grant _

                            I had this same problem.  My issue was simply resolved, open your template in Monarch and ensure that under your summary options you select filter <DEFAULT> and save the model again.  If no filter is selected here it overrides the filter specified in the MDP processing options.