3 Replies Latest reply: May 15, 2014 9:56 AM by Grant Perkins RSS

    Data base connections limits

    Bruce _

      Is there an upper limit for connecting to external databases? I looked on the specs page & it listed export row limits, but I am looking for import limitations.

       

      Why I am asking is that I am currently importing about 400k rows, and wonder if I will need to re-think my methods down the road.

       

      Ver 7.02 pro.  smile.gif[/img] 

       

      Thank you

        • Data base connections limits
          Grant Perkins

          Bruce,

           

          One documented limit is a maximum of 9 external lookups. I don't think that is what you are asking about but I felt it was worth mentioning as a general reference in the subject matter.

           

          As far as the number of records is concerned the situation is less clear. In part it depends on what sort of external database you are using and it's overall size rather than the number of records.

           

          In theory the Monarch internal table has a maximum size of 2Gb. However I have seen some imports stop when the import table file being created hits 1Gb.

           

          Obviously the number of records that 1Gb represents is somewhat dependant on the size of the records although other factors may also come into play.

           

          But it does also depend on what sort of file you are using as a source database.

           

          I have some more detailed information somewhere and I will try to find in in the next day or so - unless someone else has easier access to the information and can post sooner!

           

           

          Grant

           

          Originally posted by Bruce:

          Is there an upper limit for connecting to external databases? I looked on the specs page & it listed export row limits, but I am looking for import limitations.

           

          Why I am asking is that I am currently importing about 400k rows, and wonder if I will need to re-think my methods down the road.

           

          Ver 7.02 pro.   smile.gif[/img]  

           

          Thank you /b[/quote]

          • Data base connections limits
            Bruce _

            Thanks Grant

             

            I recall that Monarch (ver 5 or 6) used to have a 1 million page limit, so I thought I might have a 1 million row limit on imports from a database.

             

            I'm using MS Access for importing into Monarch. Right now there are about 400k rows, and the file size is about 320k. I know I'm OK for now, but was wondering if I would hit an upper limit importing this a year from now, as the file is growing by about 3-6k rows per week.

             

            Regards

             

            [size="1"][ January 12, 2005, 06:03 PM: Message edited by: Bruce ][/size]

            • Data base connections limits
              Grant Perkins

              Hi Bruce,

               

              Quick update.

               

              Whichever way run the database load - Access or a delimited file for example - Monarch creates a file for the working table collection, a .rdf file which seeks to keep some data out of the main database file ans a very small lock file. All in the temporary work folder defined in the registry settings.

               

              The table file seems to be 'full' at 1,048,576kb and further processing stops. However I know that my data are not very efficient and my .rdf file is very small, therefore not providing much extended capacity. If your records are small you may well get several million into the analysis - I can import all my nearly 6 million records well enough providing I don't select all the fields available and I make sure that the field sizes are optimised to the size of data characters they contain. The default import setting rules are not very economical in that respect, so a model redefining the input sizes seems to help a great deal.

               

              I hope this is of interest for you.

               

               

              Grant

               

               

              Originally posted by Bruce:

              Thanks Grant

               

              I recall that Monarch (ver 5 or 6) used to have a 1 million page limit, so I thought I might have a 1 million row limit on imports from a database.

               

              I'm using MS Access for importing into Monarch. Right now there are about 400k rows, and the file size is about 320k. I know I'm OK for now, but was wondering if I would hit an upper limit importing this a year from now, as the file is growing by about 3-6k rows per week.

               

              Regards /b[/quote]