Merge remote-tracking branch 'upstream/develop' into develop

Core/src/org/sleuthkit/autopsy/ingest/IngestTasksScheduler.java

@@ -251,6 +251,7 @@ final class IngestTasksScheduler {
         this.removeTasksForJob(this.directoryTasks, jobId);
         this.removeTasksForJob(this.pendingFileTasks, jobId);
         this.removeTasksForJob(this.pendingDataSourceTasks, jobId);
+        this.shuffleFileTaskQueues();
     }
 
     /**

Core/src/org/sleuthkit/autopsy/modules/fileextmismatch/mismatch_config.xml

@@ -228,6 +228,7 @@
   <ext>tile</ext>
  </signature>
  <signature mimetype="image/png">
+  <ext>avatar</ext>
   <ext>gif</ext>
   <ext>jpeg</ext>
   <ext>jpg</ext>

Core/src/org/sleuthkit/autopsy/modules/sevenzip/SevenZipIngestModule.java

@@ -466,7 +466,8 @@ public final class SevenZipIngestModule implements FileIngestModule {
                                 "SevenZipIngestModule.unpack.notEnoughDiskSpace.details");
                         //MessageNotifyUtil.Notify.error(msg, details);
                         services.postMessage(IngestMessage.createErrorMessage(ArchiveFileExtractorModuleFactory.getModuleName(), msg, details));
-                        logger.log(Level.INFO, "Skipping archive item due not sufficient disk space for this item: {0}, {1}", new Object[]{archiveFile.getName(), fileName}); //NON-NLS
+                        logger.log(Level.INFO, "Skipping archive item due to insufficient disk space: {0}, {1}", new Object[]{archiveFile.getUniquePath(), fileName}); //NON-NLS
+                        logger.log(Level.INFO, "Available disk space: {0}", new Object[]{freeDiskSpace}); //NON-NLS
                         continue; //skip this file
                     } else {
                         //update est. disk space during this archive, so we don't need to poll for every file extracted
@@ -555,7 +556,7 @@ public final class SevenZipIngestModule implements FileIngestModule {
                 //TODO decide if anything to cleanup, for now bailing
             }
 
-        } catch (SevenZipException ex) {
+        } catch (SevenZipException | TskCoreException ex) {
             logger.log(Level.SEVERE, "Error unpacking file: " + archiveFile, ex); //NON-NLS
             //inbox message
             String fullName;

docs/doxygen/footer.html

@@ -0,0 +1,7 @@
+<hr/>
+<p><i>Copyright &#169; 2012-2014 Basis Technology <br/> 
+This work is licensed under a
+<a rel="license" href="http://creativecommons.org/licenses/by-sa/3.0/us/">Creative Commons Attribution-Share Alike 3.0 United States License</a>.
+</i></p>
+</body>
+</html>

docs/doxygen/main.dox

@@ -24,7 +24,6 @@ These pages are more detailed if you want to modify Autopsy code instead of writ
 - \subpage workflow_page
 - \subpage regression_test_page 
 - \subpage native_page
-- \subpage mod_debugTsk_page
 
 <!-- Note that Case still has lots of content in package.dox -->
 */

docs/doxygen/modDevPython.dox

@@ -9,6 +9,7 @@ This page describes the basic concepts and setup that are needed for all types o
 Autopsy uses Jython (http://www.jython.org) to enable Python scripting. Jython looks like Python and gets converted into Java byte code and run on the JVM.  Its biggest limitations are:
 - Limited to Python 2.5
 - Can't use Python libraries that have native code
+- You can't make any UIs.  This means that you can't make content viewer modules or have configuration settings for your ingest modules. 
 
 Using it is very easy though in Autopsy and it allows you to access all of the Java services and classes that you need. 
 

docs/doxygen/modIngest.dox

@@ -4,7 +4,7 @@
 \section ingest_modules_getting_started Getting Started
 
 This page describes how to develop ingest modules using either Java or Python (Jython).  It assumes you have 
-already set up your development environment as described in \ref mod_dev_page or \ref mod_dev_python_page.
+already set up your development environment as described in \ref mod_dev_page or \ref mod_dev_py_page.
 
 
 \section ingest_module_types Ingest Module Types
@@ -15,33 +15,32 @@ of logical files). There are two types of ingest modules in Autopsy:
 - Data-source-level ingest modules
 - File-level ingest modules
 
-The difference between these two types of modules is what gets passed in to their process() methods during a data source ingest. 
-The process() method of a data-source-level module is called once per ingest and receives a reference to the data source.
-The process() method of a file-level ingest module is called for each file in the data source and receives a reference to the current file. Here are some guidelines for choosing the type of your ingest module:
+The difference between these two types of modules is what gets passed in to them:
+- Data source-level ingest modules get passed in a reference to a data source and it is up to the module to search for the files that it wants to analyze. 
+- File-level ingest modules are passed in a reference to each file, one at a time, and it analyzes the file that gets passed in.  
 
-- Your module should be a data-source-level ingest module if it only needs to 
-retrieve and analyze a small subset of the files present in a data source and it can find those files based on data in the database (such as file names).  
-For example, a Windows registry analysis module that only processes 
-registry hive files should be implemented as a data-source-level ingest module because there are only a few registry hives and you can find them by name. 
-- Your module should be a file-level ingest module if it analyzes most or all of 
-the files from a data source, one file at a time. If you cannot rely on finding a file based on its name, it will need to be a file-level ingest module. For example, a hash look up 
-module might process every file system file by looking up its hash in one or 
-more known files and known bad files hash sets (hash databases). 
+Here are some guidelines for choosing the type of your ingest module:
 
-As you will learn a little later in this guide, it is possible to package a 
-data-source-level ingest module and a file-level ingest module together. You
+- Your module should be a data-source-level ingest module only if the following are true:
+ - It needs to retrieve and analyze only a small number of files.
+ - It can find the files by name or other metadata in the database.
+ - It does not depend on results from other file-level ingest modules.
+ - It does not need access to the contents of ZIP and other archive files. 
+For example, our Windows registry analysis is done as a data source-level ingest module because it can query for the registry hives and process the small number of files that are found. 
+- If your needs are not met by a data source-level ingest module, then it should be a file-level ingest module. 
+
+As you will learn a little later in this guide, it is possible to make an ingest module that has both file-level and data-source level capabilities. You
 would do this when you need to work at both levels to get all of your analysis
 done. The modules in such a pair will be enabled or disabled together and will 
 have common per ingest job and global settings. 
 
+
 The text below will refer to example code in the org.sleuthkit.autopsy.examples package.
 The sample modules don't do anything 
 particularly useful, but they can serve as templates for developing your own 
 ingest modules.  
 
 
-
-
 \section ingest_modules_lifecycle Ingest Module Life Cycle
 
 Before we dive into the details of creating a module, it is important to understand the life cycle of the module. Note that this life cycle is much different for Autopsy 3.1 modules compared to Autopsy 3.0 modules. This section only talks about 3.1 modules. 
@@ -57,7 +56,7 @@ Here is an example sequence of events.  Details will be provided below.
 -# Autopsy presents the list of available ingest modules to the user and uses the utility methods from FooIngestModuleFactory class to get the module's name, description, and configuration panels. 
 -# User enables your module (and others).
 -# Autopsy uses FooIngestModuleFactory to create two instances of FooIngestModule (Autopsy is using two threads to process the files).
--# Autopsy starts up the module and calls its process method. 
+-# Autopsy calls FooIngestModule.startUp() on each thread and then calls FooIngestModule.process() to pass in each file. 
   
 
 
@@ -66,18 +65,18 @@ Here is an example sequence of events.  Details will be provided below.
 \subsection ingest_modules_implementing_basic_factory Basic Ingest Module Factory
 
 The first step to write an ingest module is to make its factory.  There are three general types of things that a factory does:
--# Provides  basic information such as the module's name, version, and description. (required)
+-# Provides basic information such as the module's name, version, and description. (required)
 -# Creates ingest modules. (required) 
 -# Provides panels so that the user can configure the module. (optional)
 
-This section covers the required parts of a basic factory so that we can make the ingest module.  A later section (\ref ingest_modules_making_options)
-covers how you can use the factory to provide options to the user. 
+This section covers the required parts of a basic factory so that we can make the ingest module.  A later section (\ref ingest_modules_making_options) covers how you can use the factory to provide options to the user. 
 
-To make writing a simple factory easier, Autopsy provides an adapter class that implements the "optional" methods in the interface. 
-Our basic factory will use the adapter. 
+To make writing a simple factory easier, Autopsy provides an adapter class that implements the "optional" methods in the interface.  Our basic factory will use the adapter. 
 
--# Define a new class that extends (Java) or inherits (Jython) org.sleuthkit.autopsy.ingest.IngestModuleFactoryAdapter. If you are using Java, NetBeans will likely complain that you have not implemented the necessary methods and you can use its "hints" to automatically generate stubs for them. 
--# Use the documentation for the org.sleuthkit.autopsy.ingest.IngestModuleFactory interface for details on what each method needs to do. You can also refer to org.sleuthkit.autopsy.examples.SampleIngestModuleFactory or the code in org.sleuthkit.autopsy.examples.ingestmodule.py as an example.  
+-# Make a factory class by either:
+ - Copy and pasting the sample code from org.sleuthkit.autopsy.examples.SampleIngestModuleFactory (Java) or org.sleuthkit.autopsy.examples.ingestmodule.py.  
+ - Manually define a new class that extends (Java) or inherits (Jython) org.sleuthkit.autopsy.ingest.IngestModuleFactoryAdapter. If you are using Java, NetBeans will likely complain that you have not implemented the necessary methods and you can use its "hints" to automatically generate stubs for them.  
+-# Update and create the needed methods using the org.sleuthkit.autopsy.ingest.IngestModuleFactory interface documentation.  You can also use the sample code mentioned above for examples of what the methods should do if you did not already copy and paste them. 
 -# If you are using Java, import org.openide.util.lookup.ServiceProvider and add a dependency on the NetBeans Lookup 
 API module to the NetBeans module that contains your ingest module. Then add a NetBeans ServiceProvider annotation so that the factory is found at run time: 
 \code
@@ -88,41 +87,38 @@ At this point, when you add a data source to an Autopsy case, you should see the
 or extended or inherited org.sleuthkit.autopsy.ingest.IngestModuleFactoryAdapter. If using Java, make sure that you added the service provider annotation. 
 
 
-\subsection ingest_modules_implementing_ingestmodule Understanding the IngestModule Interface
 
-Data source and file ingest modules have similar APIs. The main difference is what data gets passed 
-to the process() method.  Let's first cover the common concepts. 
+\subsection ingest_modules_implementing_ingestmodule Common Concepts of Ingest Modules
 
-Both modules implement the org.sleuthkit.autopsy.ingest.IngestModule interface, which defines a module initialization method:
-- org.sleuthkit.autopsy.ingest.IngestModule.startUp() 
+Before we cover the specific interfaces of the two different types of modules, let's talk about some common things. 
+- They both have a startUp() method and the expectation is that if there is any error in setting up the module, then they will throw an exception from this method so that ingest can be immediately stopped before analysis begins and the user can fix the problem. 
+- Autopsy will call a module from only a single thread.  It may make several threads, but it will make an instance of the module for each thread. You do not need to worry about thread safety unless you are using static variables or other kinds of variables that would be shared among multiple instances of the module. 
 
-Use the previous links to get the details of this method. The ingest modules have to implement a process()
-method that will get passed in either a data source or a file. A file ingest module will also have to implement a shutDown() method.  
 
-- startUp() will be called before any data is analyzed to initialize and allocate resources. Any setup procedures that could fail should be done in startUp() so that it can throw an exception and cause the ingest job to stop and notify the user. 
-- shutDown() will be called on file ingest modules after all of the files have been analyzed.
-- startUp(), process(), and shutDown() will be called from a single thread.  So, basic modules do not need to worry about thread safety if they allocate resources for each instance of the module. However, if the module wants to share resources between instances, then it is responsible for  synchronizing access to the shared resources.  See org.sleuthkit.autopsy.examples.SampleFileIngestModule as an example that shares resources. 
 
-The org.sleuthkit.autopsy.ingest.DataSourceIngestModule and org.sleuthkit.autopsy.ingest.FileIngestModule
-interfaces both extend org.sleuthkit.autopsy.ingest.IngestModule. 
-
-\subsection ingest_modules_implementing_datasourceingestmodule Creating a Data Source Ingest Module
+\subsection ingest_modules_implementing_datasourceingestmodule Creating a Data Source-level Ingest Module
 
 To create a data source ingest module:
--# Define a new class that implements (Java) or inherits (Jython) org.sleuthkit.autopsy.ingest.DataSourceIngestModule.  If you are using Java, the NetBeans IDE 
+-# Create the ingest module class by either:
+ - Define a new class that implements (Java) or inherits (Jython) org.sleuthkit.autopsy.ingest.DataSourceIngestModule.  If you are using Java, the NetBeans IDE 
 will complain that you have not implemented one or more of the required methods.
 You can use its "hints" to automatically generate stubs for the missing methods.  
--# Use this page and the 
-documentation for the org.sleuthkit.autopsy.ingest.IngestModule and 
-org.sleuthkit.autopsy.ingest.DataSourceIngestModule interfaces for guidance on 
-what each method needs to do.  Or you can copy the code from  
-org.sleuthkit.autopsy.examples.SampleDataSourceIngestModule or org.sleuthkit.autopsy.examples.ingestmodule.py and use it as a 
-template for your module.  
+ - Copy and paste the sample modules from org.sleuthkit.autopsy.examples.SampleDataSourceIngestModule or org.sleuthkit.autopsy.examples.ingestmodule.py. 
+-# Configure your factory class to create instances of the new ingest module class. To do this, you will need to change the isDataSourceIngestModuleFactory() method to return true and have the createDataSourceIngestModule() method return a new instance of your ingest module. Both of these methods have default "no-op" implementations in the IngestModuleFactoryAdapter that we used.  Your factory should have code similar to this Java code:
 
-All data source ingest modules must implement the single method defined by the 
-org.sleuthkit.autopsy.ingest.DataSourceIngestModule interface:
+\code
+    @Override
+    public boolean isDataSourceIngestModuleFactory() {
+        return true;
+    }
 
-- org.sleuthkit.autopsy.ingest.DataSourceIngestModule.process()
+    @Override
+    public DataSourceIngestModule createDataSourceIngestModule(IngestModuleIngestJobSettings ingestOptions) {
+        return new FooDataSourceIngestModule();  // replace this class name with the name of your class
+    }
+\endcode
+-# Use this page, the sample, and the documentation for the 
+org.sleuthkit.autopsy.ingest.DataSourceIngestModule interfaces to implement the startUp() and process() methods. 
 
 The process() method is where all of the work of a data source ingest module is
 done. It will be called exactly once. The 
@@ -138,48 +134,17 @@ The best way to do that is using one of the findFiles() methods of the
 org.sleuthkit.autopsy.casemodule.services.FileManager class.  See 
 \ref mod_dev_other_services for more details. 
 
-The final step to getting the basic ingest module working is to configure your factory class to create instances of it. To do this, you will need to change the isDataSourceIngestModuleFactory() method to return true and have the createDataSourceIngestModule() method return a new instance of your ingest module. Both of these methods have default "no-op" implementations in the IngestModuleFactoryAdapter that we used.  Your factory should have code similar to this Java code:
-
-\code
-    @Override
-    public boolean isDataSourceIngestModuleFactory() {
-        return true;
-    }
-
-    @Override
-    public DataSourceIngestModule createDataSourceIngestModule(IngestModuleIngestJobSettings ingestOptions) {
-        return new FooDataSourceIngestModule();  // replace this class name with the name of your class
-    }
-\endcode
-
 
 \subsection ingest_modules_implementing_fileingestmodule Creating a File Ingest Module
 
 To create a file ingest module:
--# Define a new class that implements (Java) or inherits (Jython)
-org.sleuthkit.autopsy.ingest.FileIngestModule. If you are using Java, the NetBeans IDE 
-will complain that you have not implemented one or more of the required methods. 
+To create a data source ingest module:
+-# Create the ingest module class by either:
+ - Define a new class that implements (Java) or inherits (Jython) org.sleuthkit.autopsy.ingest.FileIngestModule.  If you are using Java, the NetBeans IDE 
+will complain that you have not implemented one or more of the required methods.
 You can use its "hints" to automatically generate stubs for the missing methods.  
--# Use this page and the 
-documentation for the org.sleuthkit.autopsy.ingest.IngestModule and 
-org.sleuthkit.autopsy.ingest.FileIngestModule interfaces for guidance on what 
-each method needs to do.  Or you can copy the code from  
-org.sleuthkit.autopsy.examples.SampleFileIngestModule or org.sleuthkit.autopsy.examples.ingestmodule.py and use it as a 
-template for your module.  
-
-All file ingest modules must implement the two methods defined by the 
-org.sleuthkit.autopsy.ingest.FileIngestModule interface:
-
-- org.sleuthkit.autopsy.ingest.FileIngestModule.process()
-- org.sleuthkit.autopsy.ingest.FileIngestModule.shutDown()
-
-The process() method is where all of the work of a file ingest module is
-done. It will be called repeatedly between startUp() and shutDown(), once for 
-each file Autopsy feeds into the pipeline of which the module instance is a part. The 
-process() method receives a reference to a org.sleuthkit.datamodel.AbstractFile 
-object. 
-
-The final step to getting the basic ingest module working is to configure your factory class to create instances of it. To do this, you will need to change the isFileIngestModuleFactory() method to return true and have the createFileIngestModule() method return a new instance of your ingest module. Both of these methods have default "no-op" implementations in the IngestModuleFactoryAdapter that we used.  Your factory should have code similar to this Java code:
+ - Copy and paste the sample modules from org.sleuthkit.autopsy.examples.SampleDataSourceIngestModule or org.sleuthkit.autopsy.examples.ingestmodule.py. 
+-# Configure your factory class to create instances of the new ingest module class. To do this, you will need to change the isFileIngestModuleFactory() method to return true and have the createFileIngestModule() method return a new instance of your ingest module. Both of these methods have default "no-op" implementations in the IngestModuleFactoryAdapter that we used.  Your factory should have code similar to this Java code:
 
 \code
     @Override
@@ -193,31 +158,35 @@ The final step to getting the basic ingest module working is to configure your f
     }
 \endcode
 
-
-\section ingest_modules_services Platform Services
-
-The previous section will allow you to get a module up and running that will be passed in either a file or a data source to analyze. 
-This section covers how you get access to more data and how you can display data to the user. 
+-# Use this page, the sample, and the documentation for the 
+org.sleuthkit.autopsy.ingest.FileIngestModule interface to implement the startUp(), and process(), and shutDown() methods. 
 
 
-\subsection ingest_modules_services_ingest Ingest Services
+The process() method is where all of the work of a file ingest module is
+done. It will be called repeatedly between startUp() and shutDown(), once for 
+each file Autopsy feeds into the pipeline of which the module instance is a part. The 
+process() method receives a reference to a org.sleuthkit.datamodel.AbstractFile 
+object. 
 
-The singleton instance of the org.sleuthkit.autopsy.ingest.IngestServices class 
-provides services tailored to the needs of ingest modules, and a module developer 
-should use these utilities to log errors, send messages, get the current case, 
-fire events, persist simple global settings, etc.  Refer to the documentation 
-of the IngestServices class for method details. 
 
-\subsection ingest_modules_making_results Giving the User Feedback
 
-Ingest modules run in the background.  There are three ways to send messages and 
-save results so that the user can see them:
+\subsection ingest_modules_implementing_next Next Steps
 
-- Use the blackboard for long-term storage of analysis results. These results
-will be displayed in the results tree.
-- Use the ingest messages inbox to notify users of high-value analysis results 
-that were also posted to the blackboard.
-- Use the logging and/or message box utilities for error messages.
+This section gave you the outline of making the module.  Now we'll cover what you can do in the module. The following sections 
+often make use of the org.sleuthkit.autopsy.ingest.IngestServices class, which provides many convenient services to make module writing easier. 
+Also make sure you refer to \ref mod_dev_other_services if you are looking for a feature.
+
+
+
+
+\section ingest_modules_making_results Doing Something With Your Results
+The previous section outlined how to make the basic module and how to get access to the data.  The next 
+step is to then do some fancy analysis and present the results to the user. 
+
+The first question that you must answer is what type of data do you want the user to see.  There are two options:
+
+-# Data that can be accessed from the tree on the left-hand side of the UI and can be displayed in a table.  To do this, you will make blackboard artifacts.
+-# Data that is in a big text file or some other report that the user can review. There are two ways of doing this: make a report or make a TSK_TOOL_OUTPUT artifact. 
 
 
 \subsection ingest_modules_making_results_bb Posting Results to the Blackboard
@@ -239,6 +208,21 @@ artifacts in a loop, it is better to invoke org.sleuthkit.autopsy.ingest.IngestS
 only once after the bulk write, so as not to flood the system with events.
 
 
+\subsection ingest_modules_making_results_report Making a Report 
+
+If your module makes a text or HTML file (or some other report format) that has some complex structure (either because you prefer to write output that way or your module is simply a wrapper around another tool), then you can simply call the output a report and then it will be shown in the UI in the reports area.  You can do this by calling the org.sleuthkit.autopsy.casemodule.Case.addReport() method. 
+
+As mentioned above, another way of adding a report into Autopsy is using the org.sleuthkit.datamodel.BlackboardArtifact.ARTIFACT_TYPE.TSK_TOOL_OUTPUT artifact.  However, this has been known to cause problems with large reports in Autopsy, so the reporting mechanism is easier for the user.
+
+
+
+
+
+\section ingest_modules_users Getting User Attention
+
+The ingest modules are running in the background and the user will not notice everything you put in the tree. 
+
+
 \subsection ingest_modules_making_results_inbox Posting Results to the Message Inbox
 
 Modules should post messages to the inbox when interesting data is found. 
@@ -258,10 +242,12 @@ and posted to the inbox using the org.sleuthkit.autopsy.ingest.IngestServices.po
 method. 
 
 
+
 \subsection ingest_modules_making_results_error Reporting Errors
 
 When an error occurs, you should write an error message to the Autopsy logs, using a 
 logger obtained from org.sleuthkit.autopsy.ingest.IngestServices.getLogger().
+
 You could also send an error message to the ingest inbox. The 
 downside of this is that the ingest inbox was not really designed for this 
 purpose and it is easy for the user to miss these messages.  Therefore, it is 
@@ -273,7 +259,10 @@ org.sleuthkit.autopsy.coreutils.MessageNotifyUtil.Notify.show().
 
 
 
-\section ingest_modules_making_options User Options
+
+
+
+\section ingest_modules_making_options User Options and Configuration
 
 Autopsy allows a module to provide two levels of configuration:
 - When an ingest job is being configured, the user can choose settings that are unique to that ingest job / pipeline.  For example, to enable a certain hash set.
@@ -288,23 +277,28 @@ that the samples do not do anything particularly useful.
 
 \subsection ingest_modules_making_options_ingest Ingest Job Options
 
+Autopsy allows you to provide a graphical panel that will be displayed when the user decides to enable the ingest module.  This panel is supposed to be for settings that the user may turn on or off for different data sources. 
+
 To provide options for each ingest job:
-- hasIngestJobSettingsPanel() must return true.
-- getIngestJobSettingsPanel() must return a IngestModuleIngestJobSettingsPanel that displays the needed configuration options and returns a IngestModuleIngestJobSettings object based on the settings.
-- You are free to implement IngestModuleIngestJobSettings and store whatever you want in it (as long as it is serializable).
-- The IngestModuleIngestJobSettings object that was created during configuration will be passed back to the factory with each call to createDataSourceIngestModule() or createFileIngestModule().  The factory should cast it to its internal class that implements IngestModuleIngestJobSettings and pass that object into the constructor of its ingest module so that it can use the settings when it runs. 
+- Update org.sleuthkit.autopsy.ingest.IngestModuleFactory.hasIngestJobSettingsPanel() in your factory class to return true.
+- Update org.sleuthkit.autopsy.ingest.IngestModuleFactory.getIngestJobSettingsPanel() in your factory class to return a IngestModuleIngestJobSettingsPanel that displays the needed configuration options and returns a IngestModuleIngestJobSettings object based on the settings.
+- Create a class based on org.sleuthkit.autopsy.ingest.IngestModuleIngestJobSettings to store the settings.
 
+Your panel should create the IngestModuleIngestJobSettings class to store the settings and that will be passed back into your factory with each call to createDataSourceIngestModule() or createFileIngestModule().  The factory should cast it to its internal class that implements IngestModuleIngestJobSettings and pass that object into the constructor of its ingest module so that it can use the settings when it runs. 
 
-You can also implement the getDefaultIngestJobSettings() method to return the default settings that Autopsy should use when the module has not been run before. 
+You can also implement the getDefaultIngestJobSettings() method to return an instance of your IngestModuleIngestJobSettings class with default settings.  Autopsy will call this when the module has not been run before. 
 
 NOTE: We recommend storing simple data in the IngestModuleIngestJobSettings-based class.  In the case of our hash lookup module, we store the string names of the hash databases to do lookups in.  We then get the hash database handles in the call to startUp() using the global module settings. 
 
 
+
 \subsection ingest_modules_making_options_global Global Options
 
+Global options are those that are not specific to a data source or ingest pipeline.  They are big-picture settings.
+
 To provide global options: 
-- hasGlobalSettingsPanel() must return true.
-- getGlobalSettingsPanel() must return a org.sleuthkit.autopsy.ingest.IngestModuleGlobalSetttingsPanel with widgets to support the global settings. 
+- Update org.sleuthkit.autopsy.ingest.IngestModuleFactory.hasGlobalSettingsPanel() in your factory class to return true.
+- Update org.sleuthkit.autopsy.ingest.IngestModuleFactory.getGlobalSettingsPanel() to return a org.sleuthkit.autopsy.ingest.IngestModuleGlobalSetttingsPanel with widgets to support the global settings. 
 - You are responsible for persisting global settings and may use the module 
 settings methods provided by org.sleuthkit.autopsy.ingest.IngestServices for 
 saving simple properties, or the facilities of classes such as 
@@ -318,7 +312,11 @@ ingest job settings panel. When a module instance runs, it gets the relevant
 databases from the hash databases manager. 
 - You are responsible for having the ingest job options panel update itself if the global settings change (i.e. if a new item is added that must be listed on the ingest panel). 
 
-\section ingest_modules_api_migration Migrating Older Java Ingest Modules to the Current API
+
+
+
+
+\section ingest_modules_api_migration Migrating 3.0 Java Ingest Modules to the 3.1 API
 
 This section is a guide for module developers who wrote modules for the 3.0 API.  These API changes occurred so that 
 we could make parallel pipelines of the file-level ingest modules.   This section assumes you've read the above description of the new API.

docs/doxygen/platformConcepts.dox

@@ -1,39 +1,41 @@
-/*! \page platform_page Platform Concepts
+/*! \page platform_page Module Development Overview
+
+
+This page covers the basics of Autopsy, including the types of modules and the basic services that are provided.  Other pages will cover development environment setup and how to build specific types of modules.  This page applies to both Java and Python modules.
+
+
+\section platform_frameworks Types of Modules
+Autopsy was designed to be an extensible platform for other developers to leverage. There are several places in the platform where plug-in modules can be applied. 
+- <b>Ingest Modules:</b> These modules are run when a new data source (such as a disk image) is added to a case (and can be re-run afterwards too).  These modules come in two forms:
+ - File Ingest Modules are called for every file in the data source.  Use this type of module if you want to examine the contents of all or most of the files.  Examples include hash calculation, hash lookup, file type identification, and entropy calculation.  These modules are passed in a reference to a file to analyze. 
+ - Data Source Ingest Modules are called once for every image or set of logical files.  These modules can use the database to query for one or more files and perform analysis on them.  Examples include web artifact analysis and searches that can rely only file names and extensions.  Note that these modules will not have access to the contents of ZIP files. 
+See \ref mod_ingest_page for details on building these modules. 
+- <b>Report Modules:</b> These modules create different types of outputs that contain the analysis results. See \ref mod_report_page for details on creating these modules. 
+- <b>Content Viewers:</b> These graphical modules show information about a specific file. These are the modules in the lower right of the interface.  The platform comes with viewers to view the file in hexadecimal, extract the strings from the file, and view images and movies.  See \ref mod_content_page for details on creating these modules. 
+- <b>Result Viewers:</b> These modules show information about a set of files. These modules are in the upper right of the interface. The platform comes with viewers to view the set of files in a table and thumbnails.  See \ref mod_result_page for details on creating these modules.
+
+
+\section platform_languages Languages
+
+All modules can be written in Java.  The \ref mod_dev_page page covers setup of a Java environment.
+
+Some of the modules can be written in Python, namely ingest an report modules.  See \ref mod_dev_py_page for setup details. Python modules have access to all of the same services as Java modules do, except they cannot currently make UI elements. 
+
+
 
 \section platform_basics Basic Concepts
 
-These are the basic concepts that you should be aware of before writing a module:
+These are the basic concepts that you should be aware of before writing any type of module.  
 
-- <b>Phases:</b> The platform has been design to support different phases in the investigation process:
- - Case Creation: Use wizards to create a new case.
- - Data Source Adding: Where disk images and logical files are added to a case and file systems in disk images are analyzed to populate the database.  The end result of this phase is that the central database has a basic record of each file so that it can be analyzed.  This happens in the Add Image Wizard.
- - Ingest Module Analysis: A variety of analysis modules then run on the  files referenced in the database to perform specific tasks.
- - Browsing and searching: User manually browses and searches the data using the user interface.  They can browse through the results from the ingest modules that may still be running in the background. 
- - Report: A final report is generated at the end of the case.
-- <b>Central Database</b>: All data except for the disk image is stored in a SQLite database.  This includes information about what files exist in a disk image and the output from modules. Access to this database can be found from the org.sleuthkit.datamodel.SleuthkitCase class, but you'll probably never need to directly interact with it. The services and data model classes will interact with it. 
-- <b>Case</b>: A case class (org.sleuthkit.autopsy.casemodule.Case) is the top-level object for the data being analyzed.  From here, you can access all of the files and query it. 
-- <b>Blackboard:</b> The platform uses the blackboard to enable modules to communicate with each other and to display data in the GUI. See the \ref platform_blackboard section for more details. 
-- <b>Services</b>: There are services provided by the platform. See the \ref mod_dev_other_services section for more details.
-- <b>Utilities</b>: There are core utilities that the platform provides to modules. See the \ref mod_dev_other_utilities section for more details. 
-- <b>Single tree:</b> Results from the various modules can generally be found in a single tree. This makes it easy for users to find their results. 
-
-
-\section platform_frameworks Frameworks in the Platform
-Autopsy was designed to be an extensible platform for other developers to leverage. There are several places in the platform where plug-in modules can be applied. 
-- <b>Ingest Modules:</b> These modules are run when a new data source is added to a case (and can be re-run afterwards too).  These modules come in two forms:
- - File Ingest Modules are called for every file in the image.  Use this type of module if you want to examine the contents of all or most of the files.  Examples include hash calculation, hash lookup, file type identification, and entropy calculation. 
- - Data Source Ingest Modules are called once for every image or set of logical files.  These modules can use the database to query for one or more files and perform analysis on them.  Examples include web artifact analysis and searches that can rely only file names and extensions. 
-See \ref mod_ingest_page for details on building these modules. 
-- <b>Report Modules:</b> These modules create different types of outputs that contain the analysis results. See \ref mod_report_page for details on creating these modules. 
-- <b>Content Viewers:</b> These modules show information about a specific file. These are the modules in the lower right of the interface.  The platform comes with viewers to view the file in hexadecimal, extract the strings from the file, and view images and movies.  See \ref mod_content_page for details on creating these modules. 
-- <b>Result Viewers:</b> These modules show information about a set of files. These modules are in the upper right of the interface. The platform comes with viewers to view the set of files in a table and thumbnails.  See \ref mod_result_page for details on creating these modules. 
+- <b>Data Source</b>: Data source is the term used in Autopsy to refer to disk images and logical files that are added to a case.  Data sources are represented in Autopsy using several types of classes from the org.sleuthkit.datamodel package. 
+- <b>Case</b>: A case is a container for one or more data sources in Autopsy.  A case is represented by a  org.sleuthkit.autopsy.casemodule.Case class. Only one case can be open at a time.  You can get the current case using org.sleuthkit.autopsy.casemodule.Case.getCurrentCase().
+- <b>Central Database</b>: A central SQLite database exists and stores all file metadata and analysis results. Access to this database can be found from the org.sleuthkit.datamodel.SleuthkitCase class, but you'll probably never need to directly interact with it. All modules can query it for information, though many do not need to.  For example, file-level ingest modules will be passed in a reference to a specific file to analyze and may never need to directly go to the database for more information. 
+- <b>Blackboard:</b> The blackboard is how modules communicate back and forth.  Modules post their results to the blackboard in the form of artifacts and the UI will display them.  See the \ref platform_blackboard section for more details. 
+- <b>Services and Utilities</b>: There are a lot of convenience services and utilities that are provided to developers.  Refer to the \ref mod_dev_other_services section for more details.
 
 
 
-\section platform_services Services
-The platform provides a variety of services and utilities that you need to be familiar with.  This section outlines the basic ones and additional ones are described at the end of the document in \ref mod_dev_adv.
-
-\subsection platform_blackboard The Blackboard
+\section platform_blackboard The Blackboard
 
 The blackboard allows modules to communicate with each other and the UI. It has three main uses in Autopsy:
 - Ingest modules can communicate with each other. For example, one module can calculate a MD5 hash of a file and post it to the blackboard.  Then another module can retrieve the hash value from the blackboard and not need to calculate it again.
@@ -46,19 +48,21 @@ The blackboard is not unique to Autopsy.  It is part of The Sleuth Kit datamodel
 - Details of the Java classes can be found in \ref jni_blackboard section of the The Sleuth Kit JNI documents (http://sleuthkit.org/sleuthkit/docs/jni-docs/). 
 
 
+
 \subsection mod_dev_other_services Framework Services and Utilities
 
-The followig are basic services that are available to any module:
+The following are basic services that are available to any module.  They are provided here to be used as a reference.  When you are developing your module and feel like something should be provided by the framework, then refer to this list to find out where it could be.  If you don't find it, let us know and we'll talk about adding it for other writers to benefit. 
 
-- FileManager: the org.sleuthkit.autopsy.casemodule.services.FileManager service provides an API to access any file in the case. You can access FileManager by calling org.sleuthkit.autopsy.casemodule.services.Services.getFileManager().  Data Source-level Ingest modules and Report modules typically use this service because the other modules are passed in a reference to a specific file to do something with. 
+
+- FileManager: the org.sleuthkit.autopsy.casemodule.services.FileManager service provides an API to access any file in the case. You can access FileManager by calling org.sleuthkit.autopsy.casemodule.services.Services.getFileManager().  Data Source-level Ingest modules and Report modules typically use this service. 
 - Logging: the org.sleuthkit.autopsy.coreutils.Logger class can be used to log error and informational messages to the central Autopsy log file.
-- Pop-up Windows: 
-If you have a background task that needs the provide the user with feedback, you can use the org.sleuthkit.autopsy.coreutils.MessageNotifyUtil.Notify.show() method to make a message in the lower right hand area.
+- Pop-up Windows: If you have a background task that needs the provide the user with feedback, you can use the org.sleuthkit.autopsy.coreutils.MessageNotifyUtil.Notify.show() method to make a message in the lower right hand area.
 - Module Settings: If you want to persist settings between invocations of Autopsy, you can use org.sleuthkit.autopsy.coreutils.ModuleSettings.
 - Content Utilities: The org.sleuthkit.autopsy.datamodel.ContentUtils class has utility methods to write files from Autopsy to local disk. Specifically the org.sleuthkit.autopsy.datamodel.ContentUtils.writeToFile() method. 
 - Platform Utilities: The org.sleuthkit.autopsy.coreutils.PlatformUtil class allows you to save resources into the user folder and determine paths for the user folders.  Specifically:
  - PlatformUtil.extractResourceToUserConfigDir()
  - PlatformUtil.isWindowsOS()
 - File Utilities: The org.sleuthkit.autopsy.coreutils.FileUtil class assists with creating and deleting folders, etc.
-- IngestModules also have a class that provides additional services.  See \ref ingest_modules_services_ingest. 
+- IngestModules also have a class that provides additional services unique to their needs.  See \ref ingest_modules_services_ingest. 
+
 */