Batch processing support

TRex and TGrabs both offer full batch processing support. All parameters that can be setup via the settings box (and even some that are read-only when the program is already started), can be appended to the command-line – as mentioned above. For batch processing, special parameters are available:

Parameter	Description
auto_quit	automatically saves all requested data to the output folder and quits the app
auto_train	automatically attempts to train the visual identification if successfully tracked
auto_apply	automatically attempts to load weights from a previous training and auto correct the video
auto_no_results	do not save a .results file
auto_no_tracking_data	do not save the data/file_fishX.npz files
auto_no_memory_stats	(enabled by default) do not save memory statistics

Examples

Finding the right folder structure is always a good first step. See here for some related examples posted by GitHub user roaldarbol. There he suggests using a folder structure like so:

├── experiments
│   ├── 2021-11-29
│   │   ├── environmental_data
│   │   ├── results
│   │   │   ├── ants-1.settings
│   │   │   ├── data
│   │   │   │   ├── ants-1_beetle0.csv
│   │   │   │   ├── [...]
│   │   │   ├── track_ants-1.log
│   │   ├── videos_pv
│   │   │   ├── ants-1.pv
│   │   │   ├── average_ants-1.png
│   │   │   ├── convert_ants-1.log
│   │   └── videos_raw
│   │       ├── ants-1.mp4
├── trex_batch.sh
├── default_track.settings
└── default_convert.settings

Here, trex_beetles.settings is the main settings file, used for all videos ants-X.pv. This can be achieved by appending -s default_track.settings to the trex command line (and -s default_convert.settings to every tgrabs command line). trex_batch.sh is a script that automatically handles the addition of new files and converts/tracks the ones that have not been converted/tracked yet. It can be launched via:

/bin/bash trex_batch.sh

A modified version of his script (reference above) follows, which additionally checks for the existence of converted videos/tracking data before calling.

Note

Automation is very useful, but unsupervised programs can also be dangerous. Always back your data up in separate places. While the following script is based on much experience with the tools in question, it is largely untested and I cannot guarantee that it is safe to use. It is merely meant as an example of what such bash code could look like/is similar to what I use myself. I do not take responsibility for lost data and/or similar as the result of using it.

EXTENSION=".mp4"
ROOT=$(pwd)

ls -l $ROOT/experiments
read -p 'Date of the experiment: ' DATE

EWD=$ROOT/experiments/$DATE
CONV_SETTINGS=$ROOT/default_convert.settings
TRACK_SETTINGS=$ROOT/default_track.settings
FILES=$(find "$ROOT/experiments/$DATE/videos_raw" -type f -name '*'${EXTENSION})

echo "Found $(echo $FILES | wc -l) videos."
echo "Using ${CONV_SETTINGS} to convert and ${TRACK_SETTINGS} to track."

function convert_video() {
  f="$1"
  BASE="$2"

  if [ ! -f "${EWD}/videos_pv/${BASE}.pv" ]; then
     CMD="tgrabs \
          -d ${EWD}/videos_pv \
          -i ${f} \
          -s ${CONV_SETTINGS} \
          -o tmp"

     echo -e "\t${CMD} ..."
     if { eval $CMD 2>&1; } > convert_${BASE}.log; then
        echo -e "\tConverted ${EWD}."

        # moving post-hoc ensures that the process was successful
        # if we find a .pv file with the correct name.
        mv average_tmp.png average_${BASE}.png
        mv tmp.pv ${BASE}.pv

        return 0
     else
       echo -e "\tFailed to convert ${EWD}."

       # delete temporary files if they exist
       rm -f average_tmp.png
       rm -f tmp.pv
       return 1
     fi

  else # found video already
    echo -e "\tSkipping conversion of ${EWD}."
    return 0
  fi
}

function track_video() {
  f="$1"
  BASE="$2"

  if [ ! -f "${EWD}/videos_pv/${BASE}.results" ] \
    && [ ! -f "${EWD}/videos_pv/${BASE}.results.meta" ];
  then
    CMD="trex \
         -d ${EWD}/results \
         -i ../videos_pv/${BASE}.pv \
         -s ${TRACK_SETTINGS} \
         -auto_quit"

      if { eval $CMD 2>&1; } > track_${BASE}.log; then
         echo -e "\tTracked ${EWD}."
      else
       echo -e "\tFailed to track ${EWD}."
       return 1
     fi

   else
     echo -e "\tSkipping tracking of ${EWD}."
   fi
}

for f in ${FILES}; do
   BASE=$(basename $f ${EXTENSION})
   echo "checking raw video ${BASE}..."

   if convert_video "$f" "$BASE"; then
      track_video "$f" "$BASE"
   fi
done