10 Replies Latest reply: May 15, 2014 10:00 AM by ZJSoule _ RSS

    documentation tools

    tomkrier _

      Just wondering if you have any documentation tools for datapump in progress or beta?

        • documentation tools
          Gareth Horton

          Tom

           

          I'm not sure exactly what you mean.  Are you referring to the stylesheet for the job logs?

           

          Gareth

           

          Originally posted by tomkrier:

          Just wondering if you have any documentation tools for datapump in progress or beta? /b[/quote]

          • documentation tools
            tomkrier _

            No.  They are bugging me to document each "project" (AKA process, usually).  So for that they want to see.

            -Models that are used

            -projects that are used

            -files that are need

            -when it is sent

            -where it is sent to (email, file transfer, ect.)

             

            As we both know all this info is already there and it seems pointless to me to retype it to make someone happy.(which I will probably end up doing)  Plus just retyping this data provides a very inaccurate form of documentation.  Basically I was wondering if you have and documentations systems (web app. maybe) where anyone can view the jobs, its corresponding models and projects, and all that good stuff they want to view.

            • documentation tools
              Gareth Horton

              Tom

               

              As Todd mentioned, everything is there in the job log, other than the model, but that is documenting after the fact, rather than the Processes/projects them selves.

               

              Maybe a stylesheet for a Data Pump edited project file might be useful?

               

               

              Gareth

               

              Originally posted by tomkrier:

              No.  They are bugging me to document each "project" (AKA process, usually).  So for that they want to see.

              -Models that are used

              -projects that are used

              -files that are need

              -when it is sent

              -where it is sent to (email, file transfer, ect.)

               

              As we both know all this info is already there and it seems pointless to me to retype it to make someone happy.(which I will probably end up doing)  Plus just retyping this data provides a very inaccurate form of documentation.  Basically I was wondering if you have and documentations systems (web app. maybe) where anyone can view the jobs, its corresponding models and projects, and all that good stuff they want to view. /b[/quote]

              • documentation tools
                Joel F

                This is a feature that our company would like to see as well.  It would be nice have it exported to a file that could be importing into an Access database so that we could create new tables to add additional documention for each of the processes on Datapump.  Things such as what Job creates the input file for datapump, description of why we are using monitoring, etc.

                • documentation tools
                  Ontiveros _

                  Maybe an update to the Datapump Administrator could have a section under Create, View and Edit Processes that gives show consolidated details regarding all processes (i.e. Input file, output, schedule,log history, etc.).

                   

                  There could even be a utility created to export a report or text file to a chosen folder or keep an updated log file with process details.

                   

                  This type of request has been brought up occasionally for at least two years.  I know this because I have brought it up on a couple of occasions and unnortunately it is something that has not been listened to.

                   

                  I am really hoping that this is something Datawatch would seriously consider because it would be very useful in documenting the 100+ processes I use.

                  • documentation tools
                    Gareth Horton

                    Hi,

                     

                    This would be a fairly large amount of work to bring all the elements under one roof. Plus, the amount of information would likely not conform to any type of print format.

                     

                    Currently there are 3 main objects in Data Pump,

                     

                    Process - Documented only within the User Interface, the data is stored in a SQL Server table as a mix of standard SQL values and an XML data blob.

                     

                    Project - Self-documenting XML file.  This can be viewed via any application, including text editors.  Internet Explorer will render the file with a collapsible tree structure.  File can be viewed with XPRJ Editor (General Tab|Project Information), supplied with Data Pump, which will show some important information, using an XML Stylesheet inherited from Monarch.

                     

                    Model - Self-documenting XML file.  This can be viewed by any application, inclusing text editors.  Internet Explorer will render the file with a collapsible tree structure,  File can be viewed with XPRJ Editor (Model Tab|Model Information), which will show some important information, using an XML Stylesheet inherited from Monarch

                     

                    I am assuming that the item which is the real problem is the Process, since the other elements are self documenting, to an extent, and can be viewed easily.

                     

                    The schedule is also very transparent, as it uses the Windows Task Scheduler, so you simply need to go to Scheduled Tasks to see them.  You can also view them from a remote machine, with sufficient rights.

                     

                    The log history is also maintained in the Jobs view in Data Pump.  If you have not purged/deleted the job, then you can just click on View Log to obtain a self-documenting XML log file.  Even if you move the logs, you can still open them with a text editor or Internet Explorer.

                     

                    There is an enormous amount of information contained within the elements in Data Pump, so if you could put forward your list of the most important items that you would like to see in a single view/page/report etc, I would be grateful.

                     

                    Gareth

                     

                    Originally posted by Ontiveros:

                    Maybe an update to the Datapump Administrator could have a section under Create, View and Edit Processes that gives show consolidated details regarding all processes (i.e. Input file, output, schedule,log history, etc.).

                     

                    There could even be a utility created to export a report or text file to a chosen folder or keep an updated log file with process details.

                     

                    This type of request has been brought up occasionally for at least two years.  I know this because I have brought it up on a couple of occasions and unnortunately it is something that has not been listened to.

                     

                    I am really hoping that this is something Datawatch would seriously consider because it would be very useful in documenting the 100+ processes I use. /b[/quote]

                      • documentation tools
                        Olly Bond

                        Hello everyone,

                         

                        We're running about 100 processes, involving 500 projects, and documenting all of these by hand would be a tedious task. We've spent some time working on a way to automate the documentation of DataPump processes from the joblogs. It seemed to me that the best tool to process the information was Monarch itself.

                         

                        Essentially, we use Monarch to read the job log for the project name, the input files, and the export files. We then fetch the project files, prep them using Monarch Utility, then use Monarch to read the model file names from these. Fetching the model files and prepping these, we can then use Monarch to read the filepaths of the external lookups.

                         

                        All this data is exported to an Access table, then a macro in Visio read this table and drops shapes onto a Visio document. The macro then runs through the shapes, looking for exports that are associated with the same filename as an input or lookup file, and draws a connecting arrow from one to the other.

                         

                        This is still in beta - but has just about got to the stage where it's ready to be tested in anger. An example of the output can be seen at: http://www.greenbar.info/examples/example.pdf[/url]

                         

                        I'd be delighted to hear your feedback - please drop me a line or PM me.

                         

                        Best wishes,

                         

                        Olly

                          • documentation tools
                            thompssc _

                            That's very cool Olly, like an ERD for a Monarch process/project. I can definitely see how that would be handy with some of the more complex solutions.

                              • documentation tools
                                Olly Bond

                                Hello thompssc,

                                 

                                Thanks! At the start of the project I was trying to follow Jesse James Garrett's "Visual Vocabulary for Information Architecture" (http://www.jjg.net/ia/visvocab/[/url]). The alpha version used different flowchart shapes from the Visio basic flowchart stencil to distinguish reports, lookups, exports but these were unwieldy as soon as you had more than one export, so we switched to simple rectangles.

                                 

                                This way we can fit 16 projects on an A3 page in a 4 x 4 layout - we could manage 5 x 4 if we dropped the Notes section. One of the jobs on my snag-list is handling multiple page outputs. If you have a process with more than 16 projects, I'd be delighted if you could send me the job log, project and model files so I could try adapt the scripts to handle them.

                                 

                                Best wishes,

                                 

                                Olly