4 Replies Latest reply: May 15, 2014 10:12 AM by Olly Bond RSS

    data pump bug. are you aware?

    roger_27 _

      There has been this bug in datapump since v9 , I never really wondered if you were aware of it or not, so here goes.

       

      we have a customer who provides us 1 statement per text file.

       

      they give us about 1200-3400 statements per run.

       

      After around 600 statements, the information will import, but it will repeat with the exact identical information until it reaches the end.

       

      example?

       

      there are a thousand statements. when I run the data pump process to import all 1000 files, somewhere in the middle, around the 600th account, John Smith, every subsequent account, untill we reach 1000 will be "john smith" with different detail information, but the same header info.

       

       

      I hope this makes sense.

       

      I can very easily send you a model and statement files if this will help solve the problem.

        • data pump bug. are you aware?
          Olly Bond

          Hello Roger,

           

          It sounds to me that there's a problem with the trap characters in the Append template in the model rather than in DataPump.

           

          Please feel free with to email me samples and I'll try to help.

           

          Best wishes,

           

          Olly

            • data pump bug. are you aware?
              roger_27 _

              I described the problem wrong, what happens is let’s say you have 1000 accounts, it will import 1-700 okay (using a select query in the database), then it show accounts 950-1000 as imported as well.

               

              But it completely SKIPS (does not import at all) all the accounts in between. so we haveto import 600 at a time.

               

              I have sent you an email.

                • data pump bug. are you aware?
                  Olly Bond

                  Hello Roger,

                   

                  Your sample hasn't arrived so it may have been stopped by a mail router if the attachments were too big, or encrypted. Could you try emailing me without any attachments and we can take it from there?

                   

                  Best wishes,

                   

                  Olly

                   

                  UPDATE: email arrived - thanks.

                    • data pump bug. are you aware?
                      Olly Bond

                      Hello Roger,

                       

                      I'd misunderstood - I thought the statements were all in one file, hence assuming that the Append template wasn't well defined. Given that you're dealing with multiple HTML files, one per statement, I'm pretty sure that the issue is the limit of 1024 files in one Monarch session.

                       

                      Your files all seem to start with a filename format like 4000000018XXXXXXX._AAAAA_YYYYMMDD. I'd suggest that if you change the input mask from .HTML to 40000000181.HTML for one model, this will only pick out those statements from accounts starting in 181.... Define the exports for this project to create a new local database, Access format. Copy and paste your project file 10 times, keeping the same model, and define the inputs as 40000000182.HTML, 40000000183.HTML etc., and defining the exports as appending to the Access database. Then at the end, open the Access database and append to any remote SQL databases as required.

                       

                      This approach should minimise the risk of any timeouts from database connections, and give you an Access table that you can double check against to confirm that you've got all the data you expect. Assuming that the number of files doesn't increase dramatically towards 10,000 in a batch, this method should be robust. An alternative approach would be to script a command to concatenate all the HTML files into one, at the start of the process, (in DOS, "copy *.html big.html" would do the trick) and then run the model once just against big.html.

                       

                      Hope this helps,

                       

                      Olly