4 Replies Latest reply: May 15, 2014 10:13 AM by monarchman7 _ RSS

    too many footers causing monarch to become "buggy"?

    monarchman7 _

      Hi all,

       

      Recently I made a model which had 7 or 8 footers attached to the detail template. This worked fine on all but the first and last detail within each individual PDF (about 200 PDFs total).

       

      Example of one:

       

      Take the first detail from a PDF, say 4 out of 7 footers apply to that detail. Sometimes 1 of the remaining 3 gaps would be filled from information (usually that of the last detail from the respective PDF) from another detail. All footers are set to be "Cleared By" the detail so that's not the issue. It's some buggy thing involving having lots of footers so I'm curious if anyone else has come across it.

       

      Any help would be great, thank you guys.

        • too many footers causing monarch to become "buggy"?
          Olly Bond

          Hello,

           

          Multiple files can be a pain to debug, and PDFs are notoriously a headache to verify, but you might be able to avoid this problem by reducing the number of templates.

           

          Assuming your detail template is picking up every record you want, what about changing it to a multi-line sample and grabbing a text block of data that includes any relevant footers that occur before the next detail.

           

          You can then create calculated fields in the table that chop up the text block using if(), instr(), lsplit(), textline() etc, to give you the fields you were getting from your footer templates.

           

          If your records split over page breaks then you may need to define a Page Header template for this approach to work.

           

          Hope this helps,

           

          Olly

            • too many footers causing monarch to become "buggy"?
              monarchman7 _

              Hello Olly,

               

              Do you think the multiple files is more likely to be the problem than the fact that they are PDFs? One thing I have done in the past is combine PDFs to cut down on the table load time. Hours of table load time would be cut down to nothing by combining 600+ PDFs into 4 or less. Works fine as long as the formats don't conflict in some way and it may solve my problem.

               

              This is one of the few projects that grabbing the text in a "block" won't work. The records split over any page break I could create. I actually need to set every single footer as a floating trap to get it to work. It's pretty amazing that it works so cleanly, other than the bugs of course.

               

              The good news is these "bugs" seem to have a pattern like everything else, so I can manually edit them if needed. It's not all that bad to edit 600 lines in a 20,000 sheet, but this monarcher is lazy :).

               

              Thanks for the help as always.

                • too many footers causing monarch to become "buggy"?
                  Olly Bond

                  Hello,

                   

                  I've not tried to wrestle with hundreds of PDF files so can't comment to table load times, but I can see that combining the files might reduce the number of times Monarch has to load the import engine into memory.

                   

                  One idea might be to get rid of the PDF issue by firstly opening the files with a dummy model, which simply traps every line and grabs a single line text block, and then export this table as a fixed width text file.

                   

                  Three tricks to help before the export - ltrim the field to lose any leading spaces so you don't need the floating trap next time round, and add a calculated field based on File(), Page(), and Line() to help you track data back to the source. You might find that a summary with hidden key fields on File() might give you simple templates to grab as Append data in the second round.

                   

                  You can then model your data with multiple templates, or the text block approach if you can filter out the page header lines from the PDF.

                   

                  Hope this helps,

                   

                  Olly

                    • too many footers causing monarch to become "buggy"?
                      monarchman7 _

                      Will definitely try to manipulate the PDFs in that way. I have found that some of the calculated fields, especially trim, have been buggy as well (not ltrim specifically, but the general trim which trims both sides and in-between). However, this is usually when I grab blocks of sentences together, which is different from just using ltrim on each individual line of the PDF.

                       

                      I had another situation where I used strip to remove a "-" from a 12 digit number, and it ended up removing some numbers along with the dash. My guess is both that issue and the one I presented in the OP are related.