4  Renaming raw files and uploading to AWS S3

4.1 Introduction

In an earlier workflow, we did not change the name of raw files when they were copied from the game camera SD card. Thus, for every camera deployment, there was a deployment folder in the appropriate Raw directory, within which all of the images had the original names assigned by the game camera, generally beginning with IMG_0001.JPG and increasing.

However, because we are uploading the full-size, raw files to AWS, it is important that each file have a unique name. It is also theoretically possible that images could become confused on the hard drive when they lack unique file names.

For these reasons, we need the ability to retroactively go back to the various deployment folders in the various year Raw directories and rename the images. We also want the option, at the same time, to upload the images to s3.

Our python script rename_raw_files_and_upload_to_s3.py takes care of these steps.

4.2 The script and its arguments

The script rename_raw_files_and_upload_to_s3.py does not have any subcommands. It has several arguments, as follows:

Table 4.1: Arguments required by rename_raw_files_and_upload_to_s3 script

argument	status	explanation
--path-to-raw-directory	Required	Full path to folder with raw files with old naming convention (e.g., IMG_0001.JPG)
--camera-number	Optional	Camera number, as an integer, not exceeding 3 digits. If not specified, script reads through Raw directory and gets list of subdirectories.
--sd-card-number	Optional	SD card number, as an integer, not exceeding 3 digits. If not specified, script reads through Raw directory and gets list of subdirectories.
--path-to-uploaded-files-file	Required	Provide the path to a file called files_uploaded_to_s3.txt, which is located in the misc_files directory one level up from the PythonScripts directory in this repository and which keeps the list of files that have been uploaded to s3
--bucket-name	Required	provide the name for the s3 “bucket” on AWS to which the images will be uploaded. Generally, this argument should be north-country-wild-images

A few things need attention here with regard to working this script.

• The script is recursive. In other words, if you point --path-to-raw-directory to a directory that contains correctly-named subdirectories for a subset of camera deployments (i.e., no subdirectories that vary from the CXXX_SDXXX naming format), the script should work iteratively through each subdirectory in turn.

  • If there are subdirectories in Raw that identify, for example, project phases or sites rather than camera deployments, each then with subdirectories of camera deployments, you can point --path-to-raw-directory to one of those improperly named directories and work through all the subdirectories with it at a time.

  • In other words, the “raw directory” does not have to be called Raw

• If you are working with a single camera deployment, you can specify the --camera-number and --sd-card-number arguments. If, however, you are wishing to iterate across a number of deployments, you can omit these arguments and the script will read the list of subdirectories in the designated Raw directory.

4.3 Normal Workflow

1. Open a terminal window. CD into the PythonScripts folder of this repository.

Make sure script is set for files to go to deep archive

Approximately line 156 of the rename_raw_files_and_upload_to_s3.py reads extra_args = {"StorageClass": "DEEP_ARCHIVE"}. If this line is commented out (i.e. #extra_args = {"StorageClass": "DEEP_ARCHIVE"} and line 157 is active (extra_args = NONE) the files will be pushed to the more expensive but quickly accessible region of s3. We want to be able to use the more expensive storage as an option for testing, but in general the line of code should Not be commented out in practice and instead line 157 should be commented out.

2. For us to keep track of the files we are pushing up to s3, we begin this process by creating and switching to a new branch in the project directory. Generally, we use a branch name such as DD-MON-YYYY-AWS-upload to make clear what we are doing. For example:

git checkout -b 15-Aug-2025-AWS-rename-upload

3. You will use the rename_raw_files_and_upload_to_s3.py script, with arguments as specified in Table 4.1 above.

   • Again, note that --camera-number and --sd-card-number are optional.

4. Run the code in the terminal by stringing together the following lines code, all as one continuous command in the terminal, but replace each case of FILEPATH, CAMERA-NUMBER, SD-CARD-NUMBER or BUCKET-NAME with either the requested file path or the correct argument.

python3 rename_raw_files_and_upload_to_s3.py --path-to-raw-directory FILEPATH --camera-number CAMERA-NUMBER --sd-card-number SD-CARD-NUMBER —-path-to-uploaded-files-file FILEPATH —-bucket-name BUCKET-NAME

5. As the script processes:

   • If you do not provide --camera-number or --sd-card-number the script will output No camera or sd card number provided. Will check for all folders in raw directory.

   • For the first subdirectory of images that the script encounters, it will tell you the number of files in the directory, the directory name, and will give several examples of old file names and new file names as output in the terminal:

     Found XXX files in /START-OF-PATH/CXXX_SDXXX

     Example old file names:

     ['/START-OF-PATH/CXXX_SDXXX/IMG_0017.JPG',``'/START-OF-PATH/CXXX_SDXXX/IMG_0003.JPG', '/START-OF-PATH/CXXX_SDXXX/IMG_0005.JPG'``]

     Example new file names:

     ['/START-OF-PATH/CXXX_SDXXX/CXXX_SDXXX_20250628143219_IMG_0017.JPG', '/START-OF-PATH/CXXX_SDXXX/CXXX_SDXXX_20250628142637_IMG_0003.JPG', '/START-OF-PATH/CXXX_SDXXX/CXXX_SDXXX_20250628142640_IMG_0005.JPG']

   • Next the script will ask if you wish to go ahead and rename the raw files:

     Rename raw files to new renaming convention? (y/n)

   • Select y to continue.

   • The script will rename the images in the first subdirectory and let you know by printing Added XXX from folder CXXX_SDXXX

   • If there is more than one folder to be processed, the script will then move on to the next folder and will again tell you the number of files in the directory, the directory name, and will give several examples of old file names and new file names as output in the terminal. It will again ask if you would like to continue renaming the raw files to the new naming convention and wait for your input.

   • When the script is finished renaming files, it will then check to see if any of these files have already been uploaded to s3, and will print the result. Then it will tell you the total number of files it is planning to upload to s3 (equal to the sum of images in each of the subdirectories it has processed), and will give you a few examples of the uploads it is planning to undertake. Then it will ask if you wish to continue with the upload:

     0 files in year directory already uploaded to s3

     Will upload XXX files to BUCKET-NAME s3 bucket

     Example uploads:

     /START-OF-PATH/CXXX_SDXXX/CXXX_SDXXX_20250628143219_IMG_0017.JPG -> CXXX_SDXXX_20250628143219_IMG_0017

     /START-OF-PATH/CXXX_SDXXX/CXXX_SDXXX_20250628142640_IMG_0005.JPG -> CXXX_SDXXX_20250628142640_IMG_0005.JPG

     ...etc

     Do you want to start upload of XXX files? (y/n)

   • When you select y, the script will report its progress to the terminal, recording Upload successful! each time it uploads a file.

   • Finally, the script will ask if you wish to overwrite the uploaded files file. If you choose y, it will add the names of the files that it just uploaded to /North-Country-Wild/misc_files/files_uploaded_to_s3.txt. If you choose n, it will add the names of the files that it did NOT upload to /North-Country-Wild/PythonScripts/uploaded_files.txt

   • And that’s it!

6. Once the script has processed, you need to create a pull request from the branch you are working on in the terminal:

   • use git add . to stage any files that were changed (which should just be /North-Country-Wild/misc_files/files_uploaded_to_s3.txt if everything went smoothly)
   • use git commit -m "commit message here" to commit your changes
   • Assuming that the branch you just created does not exist on github, then to push the changes to a new branch on git hub for making a pull request, use git push --set-upstream origin name-of-branch-you-are-working-from
   • if you read along in the terminal output, you will see a link to github where you can make the pull request.
   • If you are finished uploading for that session, go ahead and make the pull request on github. Once it is complete, you can merge your branch into main locally and then delete the branch.
   • If you have more Raw image files to rename and upload, continue to process them, using the script in the way we have described, and when you are completely finished, then follow the instructions here in step 6.

7. If you are going back and retroactively renaming files and uploading them to s3, be sure to make a note that files for each deployment were uploaded on the master spreadsheet for tracking game camera deployments (privacy protected - link may not work).

You may wish to check and see which deployments have been stored on s3. For instructions on how to do so, see Chapter 5 of this documentation.

4.4 Atypical Workflows

Coming soon!