initial draft of excel in localization scripts

release_scripts/localization_scripts/README.md

@@ -1,20 +1,20 @@
 ## Description
 
 This folder provides tools to handle updates of bundle files for language localization.  There are three main scripts:
-- `allbundlesscript.py` - generates a csv file containing the relative path of the bundle file, the key, and the value for each property.
-- `diffscript.py` - determines the property values that have changed between two commits and generates a csv file containing the relative path, the key, the previous value, the new value, and the change type (addition, deletion, change).
-- `updatepropsscript.py` - Given a csv file containing the relative path of the bundle, the key, and the new value, will update the property values for a given language within the project.
+- `allbundlesscript.py` - generates a file containing the relative path of the bundle file, the key, and the value for each property.
+- `diffscript.py` - determines the property values that have changed between two commits and generates a file containing the relative path, the key, the previous value, the new value, and the change type (addition, deletion, change).
+- `updatepropsscript.py` - Given a csv or xlsx file containing the relative path of the bundle, the key, and the new value, will update the property values for a given language within the project.
 
 All of these scripts provide more details on usage by calling the script with `-h`. 
 
 ## Basic Localization Update Workflow
 
-1. Call `python3 diffscript.py <output path> -l <language>` to generate a csv file containing differences in properties file values from the language's previous commit to the `HEAD` commit.  The language identifier should be the abbreviated identifier used for the bundle (i.e. 'ja' for Japanese).  The output path should be specified as a relative path with the dot slash notation (i.e. `./outputpath.csv`) or an absolute path.
+1. Call `python3 diffscript.py <output path> -l <language>` to generate a file containing differences in properties file values from the language's previous commit to the `HEAD` commit.  The language identifier should be the abbreviated identifier used for the bundle (i.e. 'ja' for Japanese).  The output path should be specified as a relative path with the dot slash notation (i.e. `./outputpath.xlsx`) or an absolute path.
 2. Update csv file with translations
-3. Call `python3 updatepropsscript.py <input path> -l <language>` to update properties files based on the newly generated csv file.  The csv file should be formatted such that the columns are bundle relative path, property files key, translated value and commit id for the latest commit id for which these changes represent.  The commit id only needs to be in the header row.  The output path should be specified as a relative path with the dot slash notation (i.e. `./outputpath.csv`) or an absolute path.
+3. Call `python3 updatepropsscript.py <input path> -l <language>` to update properties files based on the newly generated file.  The file should be formatted such that the columns are bundle relative path, property files key, original value (or empty column), translated value and commit id for the latest commit id for which these changes represent.  The commit id only needs to be in the header row.  The output path should be specified as a relative path with the dot slash notation (i.e. `./outputpath.xlsx`) or an absolute path.
 
 ## Localization Generation for the First Time
-First-time updates should follow a similar procedure except that instead of calling `diffscript.py`, call `python3 allbundlesscript <output path>` to generate a csv file with relative paths of bundle files, property file keys, property file values.  The output path should be specified as a relative path with the dot slash notation (i.e. `./inputpath.csv`) or an absolute path.
+First-time updates should follow a similar procedure except that instead of calling `diffscript.py`, call `python3 allbundlesscript <output path>` to generate a file with relative paths of bundle files, property file keys, property file values.  The output path should be specified as a relative path with the dot slash notation (i.e. `./inputpath.xlsx`) or an absolute path.
 
 ##Unit Tests
 Unit tests can be run from this directory using `python3 -m unittest`.

release_scripts/localization_scripts/allbundlesscript.py

@@ -4,45 +4,15 @@ git >= 1.7.0 and python >= 3.4.  This script relies on fetching 'HEAD' from curr
 repo is on correct branch (i.e. develop).
 """
 import sys
+
 from envutil import get_proj_dir
 from excelutil import write_results_to_xlsx
 from gitutil import get_property_file_entries, get_commit_id, get_git_root
 from csvutil import write_results_to_csv
-from typing import Union
-import re
 import argparse
 
-from outputresult import OutputResult
 from outputtype import OutputType
-
-
-def get_items_to_be_written(repo_path: str, show_commit: bool,
-                            value_regex: Union[str, None] = None) -> OutputResult:
-    """Determines the contents of '.properties-MERGED' files and writes to a csv file.
-
-    Args:
-        repo_path (str): The local path to the git repo.
-        show_commit (bool): Whether or not to include the commit id in the header
-        value_regex (Union[str, None]): If non-none, only key value pairs where the value is a regex match with this
-        value will be included.
-    """
-
-    row_header = ['Relative path', 'Key', 'Value']
-    if show_commit:
-        row_header.append(get_commit_id(repo_path, 'HEAD'))
-
-    rows = []
-    omitted = []
-
-    for entry in get_property_file_entries(repo_path):
-        new_entry = [entry.rel_path, entry.key, entry.value]
-        if value_regex is None or re.match(value_regex, entry.value):
-            rows.append(new_entry)
-        else:
-            omitted.append(new_entry)
-
-    omitted_to_write = [row_header] + omitted if len(omitted) > 0 else None
-    return OutputResult([row_header] + rows, omitted_to_write)
+from propentry import convert_to_output
 
 
 def main():
@@ -57,16 +27,20 @@ def main():
                         help='The path to the repo.  If not specified, path of script is used.')
     parser.add_argument('-o', '--output-type', dest='output_type', type=OutputType, choices=list(OutputType),
                         required=False, help="The output type.  Currently supports 'csv' or 'xlsx'.", default='xlsx')
-    parser.add_argument('-nc', '--no_commit', dest='no_commit', action='store_true', default=False,
+    parser.add_argument('-nc', '--no-commit', dest='no_commit', action='store_true', default=False,
                         required=False, help="Suppresses adding commits to the generated header.")
+    parser.add_argument('-nt', '--no-translated-col', dest='no_translated_col', action='store_true', default=False,
+                        required=False, help="Don't include a column for translation.")
 
     args = parser.parse_args()
     repo_path = args.repo_path if args.repo_path is not None else get_git_root(get_proj_dir())
     output_path = args.output_path
     show_commit = not args.no_commit
     output_type = args.output_type
+    translated_col = not args.translated_col
+    commit_id = get_commit_id(repo_path, 'HEAD') if show_commit else None
 
-    processing_result = get_items_to_be_written(repo_path, show_commit)
+    processing_result = convert_to_output(get_property_file_entries(repo_path), commit_id, translated_col)
 
     # based on https://stackoverflow.com/questions/60208/replacements-for-switch-statement-in-python
     {

release_scripts/localization_scripts/diffscript.py

@@ -2,58 +2,17 @@
 and generates a csv file containing the items changed.  This script requires the python libraries:
 gitpython and jproperties.  As a consequence, it also requires git >= 1.7.0 and python >= 3.4.
 """
-import re
 import sys
 from envutil import get_proj_dir
 from excelutil import write_results_to_xlsx
-from gitutil import get_property_files_diff, get_commit_id, get_git_root
-from itemchange import ItemChange, ChangeType
+from gitutil import get_property_files_diff, get_git_root
+from itemchange import convert_to_output
 from csvutil import write_results_to_csv
 import argparse
-from typing import Union
 from langpropsutil import get_commit_for_language, LANG_FILENAME
-from outputresult import OutputResult
 from outputtype import OutputType
 
 
-def get_diff_to_write(repo_path: str, commit_1_id: str, commit_2_id: str, show_commits: bool, separate_deleted: bool,
-                      value_regex: Union[str, None] = None) -> OutputResult:
-    """Determines the changes made in '.properties-MERGED' files from one commit to another commit and returns results.
-
-    Args:
-        repo_path (str): The local path to the git repo.
-        commit_1_id (str): The initial commit for the diff.
-        commit_2_id (str): The latest commit for the diff.
-        show_commits (bool): Show commits in the header row.
-        separate_deleted (bool): put deletion items in a separate field in return type ('deleted').  Otherwise,
-        include in regular results.
-        value_regex (Union[str, None]): If non-none, only key value pairs where the value is a regex match with this
-        value will be included.
-    """
-
-    row_header = ItemChange.get_headers()
-    if show_commits:
-        row_header += [get_commit_id(repo_path, commit_1_id), get_commit_id(repo_path, commit_2_id)]
-
-    rows = []
-    omitted = []
-    deleted = []
-
-    for entry in get_property_files_diff(repo_path, commit_1_id, commit_2_id):
-        entry_row = entry.get_row()
-        if separate_deleted and entry.type == ChangeType.DELETION:
-            deleted.append(entry_row)
-        if value_regex is not None and re.match(value_regex, entry.cur_val):
-            omitted.append(entry_row)
-        else:
-            rows.append(entry_row)
-
-    omitted_result = [row_header] + omitted if len(omitted) > 0 else None
-    deleted_result = [row_header] + deleted if len(deleted) > 0 else None
-
-    return OutputResult([row_header] + rows, omitted_result, deleted_result)
-
-
 def main():
     # noinspection PyTypeChecker
     parser = argparse.ArgumentParser(description="Determines the updated, added, and deleted properties from the "
@@ -79,11 +38,15 @@ def main():
                         help='Specify the language in order to determine the first commit to use (i.e. \'ja\' for '
                              'Japanese.  This flag overrides the first-commit flag.')
 
+    parser.add_argument('-nt', '--no-translated-col', dest='no_translated_col', action='store_true', default=False,
+                        required=False, help="Don't include a column for translation.")
+
     args = parser.parse_args()
     repo_path = args.repo_path if args.repo_path is not None else get_git_root(get_proj_dir())
     output_path = args.output_path
     commit_1_id = args.commit_1_id
     output_type = args.output_type
+    show_translated_col = not args.translated_col
 
     lang = args.language
     if lang is not None:
@@ -98,7 +61,12 @@ def main():
     commit_2_id = args.commit_2_id
     show_commits = not args.no_commits
 
-    processing_result = get_diff_to_write(repo_path, commit_1_id, commit_2_id, show_commits)
+    changes = get_property_files_diff(repo_path, commit_1_id, commit_2_id)
+    processing_result = convert_to_output(changes,
+                                          commit_1_id=commit_1_id if show_commits else None,
+                                          commit_2_id=commit_2_id if show_commits else None,
+                                          show_translated_col=show_translated_col,
+                                          separate_deleted=True)
 
     # based on https://stackoverflow.com/questions/60208/replacements-for-switch-statement-in-python
     {

release_scripts/localization_scripts/itemchange.py

@@ -1,7 +1,12 @@
 from typing import Iterator, List, Union
+
+from outputresult import OutputResult
 from propsutil import get_entry_dict
 from enum import Enum
 
+from tabularutil import WITH_TRANSLATED_COLS, RELATIVE_PATH_COL, KEY_COL, create_output_result
+import re
+
 
 class ChangeType(Enum):
     """Describes the nature of a change in the properties file."""
@@ -49,18 +54,71 @@ class ItemChange:
         """
         return ['Relative Path', 'Key', 'Change Type', 'Previous Value', 'Current Value']
 
-    def get_row(self) -> List[str]:
+    def get_row(self, show_translated_col: bool) -> List[str]:
         """Returns the list of values to be entered as a row in csv serialization.
+        Args:
+            show_translated_col (bool): Whether or not the translated columns are showing; otherwise use default.
 
         Returns:
             List[str]:  The list of values to be entered as a row in csv serialization.
         """
-        return [
-            self.rel_path,
-            self.key,
-            self.type,
-            self.prev_val,
-            self.cur_val]
+
+        if show_translated_col:
+            return [
+                self.rel_path,
+                self.key,
+                self.cur_val
+            ]
+        else:
+            return [
+                self.rel_path,
+                self.key,
+                self.type,
+                self.prev_val,
+                self.cur_val]
+
+
+ITEMCHANGE_DEFAULT_COLS = [RELATIVE_PATH_COL, KEY_COL, 'Change Type', 'Previous Value', 'Current Value']
+
+
+def convert_to_output(items: Iterator[ItemChange], commit1_id: Union[str, None] = None,
+                      commit2_id: Union[str, None] = None, show_translated_col: bool = True,
+                      value_regex: Union[str, None] = None, separate_deleted: bool = True) -> OutputResult:
+    """
+    Converts PropEntry objects to an output result to be written to a tabular datasource.
+    Args:
+        items: The PropEntry items.
+        commit1_id: The first commit id to be shown in the header or None.
+        commit2_id: The second commit id to be shown in the header or None.
+        show_translated_col: Whether or not to show an empty translated column.
+        value_regex: Regex to determine if a value should be omitted.
+        separate_deleted: Deleted items should not be included in regular results.
+
+    Returns: An OutputResult to be written.
+
+    """
+    header = WITH_TRANSLATED_COLS if show_translated_col else ITEMCHANGE_DEFAULT_COLS
+
+    if commit1_id:
+        header = header + [commit1_id]
+
+    if commit2_id:
+        header = header + [commit2_id]
+
+    results = []
+    omitted = []
+    deleted = []
+
+    for item in items:
+        item_row = item.get_row(show_translated_col)
+        if separate_deleted and item.type == ChangeType.DELETION:
+            deleted.append(item_row)
+        if value_regex is not None and re.match(value_regex, item.cur_val):
+            omitted.append(item_row)
+        else:
+            results.append(item_row)
+
+    return create_output_result(header, results, omitted=omitted)
 
 
 def get_item_change(rel_path: str, key: str, prev_val: str, cur_val: str) -> Union[ItemChange, None]:

release_scripts/localization_scripts/propentry.py

@@ -1,3 +1,9 @@
+from typing import List, Union, Iterator
+from outputresult import OutputResult
+from tabularutil import WITH_TRANSLATED_COLS, DEFAULT_COLS, create_output_result
+import re
+
+
 class PropEntry:
     rel_path: str
     key: str
@@ -17,3 +23,45 @@ class PropEntry:
         self.key = key
         self.value = value
         self.should_delete = should_delete
+
+    def get_row(self) -> List[str]:
+        """Returns the list of values to be entered as a row in serialization.
+
+        Returns:
+            List[str]:  The list of values to be entered as a row in serialization.
+        """
+        return [
+            self.rel_path,
+            self.key,
+            self.value]
+
+
+def convert_to_output(items: Iterator[PropEntry], commit_id: Union[str, None] = None,
+                      show_translated_col: bool = True, value_regex: Union[str, None] = None) -> OutputResult:
+    """
+    Converts PropEntry objects to an output result to be written to a tabular datasource.
+    Args:
+        items: The PropEntry items.
+        commit_id: The commit id to be shown in the header or None.
+        show_translated_col: Whether or not to show an empty translated column.
+        value_regex: Regex to determine if a value should be omitted.
+
+    Returns: An OutputResult to be written.
+
+    """
+    header = WITH_TRANSLATED_COLS if show_translated_col else DEFAULT_COLS
+
+    if commit_id:
+        header = header + [commit_id]
+
+    results = []
+    omitted = []
+
+    for item in items:
+        new_entry = item.get_row()
+        if value_regex is None or re.match(value_regex, item.value):
+            results.append(new_entry)
+        else:
+            omitted.append(new_entry)
+
+    return create_output_result(header, results, omitted=omitted)

release_scripts/localization_scripts/tabularutil.py

@@ -0,0 +1,37 @@
+from typing import Generic, TypeVar, List, Union
+
+from outputresult import OutputResult
+
+T = TypeVar('T')
+
+
+RELATIVE_PATH_COL = 'Relative path'
+KEY_COL = 'Key'
+VALUE_COL = 'Value'
+
+ENGLISH_VALUE_COL = 'English Value'
+TRANSLATED_VALUE_COL = 'Translated Value'
+
+DEFAULT_COLS = [RELATIVE_PATH_COL, KEY_COL, VALUE_COL]
+WITH_TRANSLATED_COLS = [RELATIVE_PATH_COL, KEY_COL, ENGLISH_VALUE_COL, TRANSLATED_VALUE_COL]
+
+
+def create_output_result(row_header: List[str], results: List[List[str]],
+                         omitted: Union[List[List[str]], None] = None,
+                         deleted: Union[List[List[str]], None] = None) -> OutputResult:
+
+    """
+    Creates OutputResult from components.
+    Args:
+        row_header: The row header.
+        results: The results.
+        omitted: The omitted items if any.
+        deleted: The deleted items if any.
+
+    Returns: The generated OutputResult.
+
+    """
+    omitted_result = [row_header] + omitted if omitted else None
+    deleted_result = [row_header] + deleted if deleted else None
+
+    return OutputResult([row_header] + results, omitted_result, deleted_result)

release_scripts/localization_scripts/updatepropsscript.py

@@ -2,12 +2,13 @@
 This script requires the python libraries: jproperties.  It also requires Python 3.x.
 """
 
-from typing import List, Dict, Tuple, Callable, Iterator
+from typing import List, Dict, Tuple, Callable, Iterator, Union, TypedDict
 import sys
 import os
 
 from envutil import get_proj_dir
-from fileutil import get_new_path
+from excelutil import excel_to_records
+from fileutil import get_new_path, get_path_pieces
 from gitutil import get_git_root
 from langpropsutil import set_commit_for_language
 from propsutil import set_entry_dict, get_entry_dict_from_path, get_lang_bundle_name
@@ -170,42 +171,133 @@ def get_should_deleted(row_items: List[str], requested_idx: int) -> bool:
         return False
 
 
+class DataRows(TypedDict):
+    """
+    Defines pieces of an intermediate parsed result from a data source including the header row (if present), results
+    as a 2d list, and deleted results as a 2d list.
+    """
+    header: Union[List[str], None]
+    results: List[List[str]]
+    deleted_results: Union[List[List[str]], None]
+
+
+def get_csv_rows(input_path: str, has_header: bool) -> DataRows:
+    """
+    Gets rows of a csv file in a DataRows format.
+    Args:
+        input_path: The input path of the file.
+        has_header: Whether or not it has a header.
+
+    Returns: An intermediate result DataRows object for further parsing.
+
+    """
+    all_items, header = csv_to_records(input_path, has_header)
+    return {
+        'header': header,
+        'results': all_items
+    }
+
+
+def get_xlsx_rows(input_path: str, has_header: bool, results_sheet: str, deleted_sheet: str) -> DataRows:
+    """
+    Gets worksheets of an excel workbook in a DataRows format.
+    Args:
+        input_path: The input path of the file.
+        has_header: Whether or not is has a header.
+        results_sheet: The name of the results sheet.
+        deleted_sheet: The name of the sheet containing deleted items.
+
+    Returns: An intermediate result DataRows object for further parsing.
+
+    """
+    workbook = excel_to_records(input_path, has_header)
+    results_items = workbook[results_sheet]
+    header = None
+    if has_header and len(results_items) > 0:
+        header = results_items[0]
+        results_items = results_items[1:len(results_items)]
+
+    deleted_items = workbook[deleted_sheet] if deleted_sheet else None
+    return {
+        'header': header,
+        'results': results_items,
+        'deleted_results': deleted_items
+    }
+
+
+def get_prop_entries_from_data(datarows: DataRows, path_idx: int, key_idx: int, value_idx: int,
+                               should_delete_converter: Tuple[Callable[[List[str]], bool], None],
+                               path_converter: Callable) -> List[PropEntry]:
+    """
+    Converts a DataRows object into PropEntry objects.
+    Args:
+        datarows: The DataRows object.
+        path_idx: The index of the column containing the path.
+        key_idx: The index of the column containing the key.
+        value_idx: The index of the column containing the value.
+        should_delete_converter: Given a list of strings representing a row, returns true if the entry should be
+        deleted.
+        path_converter: Converts the path to the proper format.
+
+    Returns: A list of PropEntry items.
+
+    """
+
+    prop_entries = get_prop_entries(datarows['results'], path_idx, key_idx, value_idx, should_delete_converter,
+                                    path_converter)
+
+    if datarows['deleted_results'] and len(datarows['deleted_results']) > 0:
+        prop_entries += get_prop_entries(datarows['results'], path_idx, key_idx, value_idx, lambda row: True,
+                                         path_converter)
+
+    return prop_entries
+
+
 def main():
     # noinspection PyTypeChecker
     parser = argparse.ArgumentParser(description='Updates properties files in the autopsy git repo.',
                                      formatter_class=argparse.ArgumentDefaultsHelpFormatter)
 
-    parser.add_argument(dest='csv_file', type=str, help='The path to the csv file.  The default format for the csv '
-                                                        'file has columns of relative path, properties file key, '
-                                                        'properties file value, whether or not the key should be '
-                                                        'deleted, and commit id for how recent these updates are.  '
-                                                        'If the key should be deleted, the deletion row should be '
-                                                        '\'DELETION.\'  A header row is expected by default and the '
-                                                        'commit id, if specified, should only be in the first row.  The'
-                                                        ' input path should be specified as a relative path with the '
-                                                        'dot slash notation (i.e. `./inputpath.csv`) or an absolute '
-                                                        'path.')
+    parser.add_argument(dest='file', type=str, help='The path to the file (ending in either .xlsx or .csv). '
+                                                    'The default format for the file has columns of relative path, '
+                                                    'properties file key, properties file value, whether or not the '
+                                                    'key should be deleted, and commit id for how recent these updates '
+                                                    'are. If the key should be deleted, the deletion row should be '
+                                                    '\'DELETION.\' A header row is expected by default and the '
+                                                    'commit id, if specified, should only be in the first row. The'
+                                                    ' input path should be specified as a relative path with the '
+                                                    'dot slash notation (i.e. `./inputpath.csv`) or an absolute '
+                                                    'path.')
 
     parser.add_argument('-r', '--repo', dest='repo_path', type=str, required=False,
                         help='The path to the repo.  If not specified, parent repo of path of script is used.')
+
     parser.add_argument('-p', '--path-idx', dest='path_idx', action='store', type=int, default=0, required=False,
                         help='The column index in the csv file providing the relative path to the properties file.')
     parser.add_argument('-k', '--key-idx', dest='key_idx', action='store', type=int, default=1, required=False,
                         help='The column index in the csv file providing the key within the properties file.')
     parser.add_argument('-v', '--value-idx', dest='value_idx', action='store', type=int, default=2, required=False,
                         help='The column index in the csv file providing the value within the properties file.')
-    parser.add_argument('-d', '--should-delete-idx', dest='should_delete_idx', action='store', type=int, default=3,
-                        required=False, help='The column index in the csv file providing whether or not the file '
-                                             'should be deleted.  Any non-blank content will be treated as True.')
     parser.add_argument('-c', '--commit-idx', dest='latest_commit_idx', action='store', type=int, default=4,
                         required=False, help='The column index in the csv file providing the commit for which this '
                                              'update applies. The commit should be located in the header row.  ')
+    parser.add_argument('-rs', '--results-sheet', dest='results_sheet', action='store', type=str,
+                        default='results', required=False, help='In an excel workbook, the sheet that indicates '
+                                                                'results items.  This is only used for xlsx files.')
+    parser.add_argument('-ds', '--deleted-sheet', dest='deleted_sheet', action='store', type=str,
+                        default='deleted', required=False, help='In an excel workbook, the sheet that indicates '
+                                                                'deleted items.  This is only used for xlsx files.')
+    parser.add_argument('-di', '--should-delete-idx', dest='should_delete_idx', action='store', type=int, default=-1,
+                        required=False, help='The column index in the csv file providing whether or not the file '
+                                             'should be deleted.  Any non-blank content will be treated as True.')
+
+    parser.add_argument('-z', '--has-no-header', dest='has_no_header', action='store_true', default=False,
+                        required=False, help='Specify whether or not there is a header within the csv file.')
 
     parser.add_argument('-f', '--file-rename', dest='file_rename', action='store', type=str, default=None,
                         required=False, help='If specified, the properties file will be renamed to the argument'
                                              ' preserving the specified relative path.')
-    parser.add_argument('-z', '--has-no-header', dest='has_no_header', action='store_true', default=False,
-                        required=False, help='Specify whether or not there is a header within the csv file.')
+
     parser.add_argument('-o', '--should-overwrite', dest='should_overwrite', action='store_true', default=False,
                         required=False, help="Whether or not to overwrite the previously existing properties files"
                                              " ignoring previously existing values.")
@@ -217,15 +309,18 @@ def main():
     args = parser.parse_args()
 
     repo_path = args.repo_path if args.repo_path is not None else get_git_root(get_proj_dir())
-    input_path = args.csv_file
+
+    input_path = args.file
     path_idx = args.path_idx
     key_idx = args.key_idx
     value_idx = args.value_idx
     has_header = not args.has_no_header
     overwrite = args.should_overwrite
+    deleted_sheet = args.deleted_sheet
+    results_sheet = args.result_sheet
 
     # means of determining if a key should be deleted from a file
-    if args.should_delete_idx is None:
+    if args.should_delete_idx is None and args.should_delete_idx >= 0:
         should_delete_converter = None
     else:
         def should_delete_converter(row_items: List[str]):
@@ -241,9 +336,19 @@ def main():
     else:
         path_converter = None
 
-    # retrieve records from csv
-    all_items, header = list(csv_to_records(input_path, has_header))
-    prop_entries = get_prop_entries(all_items, path_idx, key_idx, value_idx, should_delete_converter, path_converter)
+    # retrieve records from file
+    ext = get_path_pieces(input_path)[2]
+    if ext == 'xlsx':
+        data_rows = get_xlsx_rows(input_path, has_header, results_sheet, deleted_sheet)
+    elif ext == 'csv':
+        data_rows = get_csv_rows(input_path, has_header)
+    else:
+        raise ValueError('Expected either a csv file or xlsx file for input.')
+
+    # convert to PropEntry objects
+    prop_entries = get_prop_entries_from_data(data_rows, path_idx, key_idx, value_idx,
+                                              should_delete_converter, path_converter)
+    header = data_rows['header']
 
     # write to files
     if overwrite: