5 Replies Latest reply: May 15, 2014 10:10 AM by stigerk _ RSS

    Maximum Number of Database import lines

    stigerk _

      Evening,

       

      I'm guessing someone will have asked this before but......

       

      What's the maximum number of database lines that Monarch Pro v10 can cope with?

       

      I'm trying to import a delimited text file with in excess of 10million records and it stalls part way through the process.....

       

      I'm assuming it's just too much data to process......

       

      anyone able to confirm whether I'm asking the impossible?

        • Maximum Number of Database import lines
          Olly Bond

          Hello stigerk and welcome,

           

          I believe the limit is 10 million, but you might also hit the limit of 2GB of data in memory (Monarch stores the data in Access MDB format, which is limited to this) before you get to that point, depending on the amount of data in each row. You can get around this with DataPump which can work with SQL Server directly - but I think you'd best contact Datawatch for help with that.

           

          There is another trick which might help you - but you'd need to change your approach a little. Assume your data has a few tens of millions of rows and looks like:

           

          123, More data, More data...

          7658, More data, More data...[/CODE]

           

          Then open it as a report, not as a database, and make a literal trap on the first character of "1". Extract the fields you need, export to a database or CSV file. Then replicate the model, change the literal trap to "2" and repeat, appending the export, and so on through "3",..."9","0".

           

          Assuming you don't have more than 10,000,000 records which start with any one number, then this should work. You could also use "00", "01", etc for the traps to break the data into 100 sections, or use "a"..."z" and 26 models if the first character is a letter.

           

          Does this make sense?

           

          Best wishes,

           

          Olly

          • Maximum Number of Database import lines
            Data Kruncher

            Hello and welcome, stigerk.

             

            According to the online help, V10's input file size specification for CSV files is "Limited only by system resources".

             

            My first guess would be that you need more RAM in that machine for that work. Also, be sure to read input from local drives instead of network drives. Others may have additional ideas. (Edit: Cross-posting with Olly. He has recommended this "divide-and-conquer" approach before here with great success.)

             

            It's not impossible, you're just likely asking a bit too much of your present hardware.

             

            HTH,

            Kruncher

              • Maximum Number of Database import lines
                elginreigner _

                Hello and welcome, stigerk.

                 

                According to the online help, V10's input file size specification for CSV files is "Limited only by system resources".

                 

                My first guess would be that you need more RAM in that machine for that work. Also, be sure to read input from local drives instead of network drives. Others may have additional ideas. (Edit: Cross-posting with Olly. He has recommended this "divide-and-conquer" approach before here with great success.)

                 

                It's not impossible, you're just likely asking a bit too much of your present hardware.

                 

                HTH,

                Kruncher[/QUOTE]

                 

                I believe the limit is still 10million. I read the line differently, that the size of the CSV file is limited to your system resources, i.e. 1GB file versus 10GB both with 1 million records but different number of columns. But as Olly stated Datawatch can help you better here to determine the issue.

                 

                Also as Olly stated the issue is the MS Access database on which Monarch data is stored, 2GB is still the limit. I would actually think it would be a smarter solution for Monarch to run on SQL express back end as their limit is being increased to 10GB and is much much faster than Access.

                  • Maximum Number of Database import lines
                    Grant Perkins

                    The limit is 9,999,999 records but, as pointed out previously, there are other constraints what might com into play earlier than that related to number of fields (including calculated fields) and so on. See the "Specifications" section of the Help for full details.

                     

                    The other concern one might have with such large records is that performance, in almost any application that involves indexing of some sort, usually becomes comparatively slow - to the extent that splitting things up, if possible, could be advantageous.

                     

                    Grant