This is the story of how I created a million+ image timelapse with absolutely no knowledge on how to do it correctly.
One day in the late 2000s I’m sitting in the dungeons of Bethel University with no windows. A new building is being constructed next door. I can hear the construction happening and I want a window.
“What are the odds they made a webcam?”
They did! Cool.
“Can I cron this?”
I sure could.
Over the course of a couple of years I saved millions of jpgs from the construction, and then needed to figure out how to put them into a timelapse.
It wasn’t as easy as just stringing them altogether because when it got dark at night you’d end up with a long black spell in the video. How to get around that?
Simple time comparisons (i.e.m 8-5) wouldn’t work, esp. in MN where the day length changes dramatically.
Solution: imagemagick.
IM would give me the darkness/lightness of an image. So for months my workflow was to compute the relative brightness of each and every image everytime I wanted to update the timelapse.
Something like:
#!/bin/bash
mkdir -p frames
i=0
# Define the brightness threshold (10% of the maximum brightness value, which is 1.0)
BRIGHTNESS_THRESHOLD=0.1
find . -maxdepth 1 -type f -name "*.jpg" | while read -r image; do
# Get brightness (mean value, typically 0.0 to 1.0)
brightness=$(identify -format "%[fx:mean]" "$image")
# Perform the brightness comparison using bc for floating-point arithmetic
# Check if brightness is greater than the threshold
if (( $(echo "$brightness > $BRIGHTNESS_THRESHOLD" | bc -l) )); then
i_filename=$(printf "%04d.jpg" "$i")
ln -s "$(readlink -f "$image")" "frames/$i_filename"
((i++))
fi
done
# Check if any frames were linked before attempting to create the video
if [ "$i" -gt 0 ]; then
ffmpeg -r 25 -i frames/%04d.jpg -c:v libx264 -vf "fps=25,format=yuv420p" output.mp4
echo "Processing complete. Symbolic links created in 'frames/' for images over ${BRIGHTNESS_THRESHOLD} brightness."
echo "Video 'output.mp4' generated from ${i} selected frames."
else
echo "No images met the brightness threshold of ${BRIGHTNESS_THRESHOLD}. No symbolic links created or video generated."
fiExcept far less pretty, I had Gemini clean that up for me.
So every time I generated a new video, recompute every frame based on brightness (I forgot the comparison in there, you can add it in your imagination) and used symlinks to give frame numbers for ffmpeg.
Yes, I was creating tens of thousands and then hundreds of thousands of symlinks to get ffmpeg to pick up on them as individual frames.
Eventually I figured out how to not re-process everything, I think by moving processed images to a different folder. Something very high tech like that.
Was going great until some idiot started leaving a light on overnight.
Completely threw my heuristic out the window.
BUT, I soon found the counting the number of unique colors in the image was even better than the overall lightness. So same loop, but get the count of unique colors.
Problem: we were now at millions of images and still in the Pentium age.
What to do?
What any self-respecting bash guy does: get more computers, then write bash scripts that create a mysql database and load up every image into the database.
I created a job queueing system — of sorts — that required bash scripts to loop over mysql SELECT statements and write multiple imagemagick commands to a single sh file, and execute that.
Something like:
brightness=$(identify -format "%[fx:mean]" "$image")
sql = "INSERT INTO images_and_brightness (image_path, brightness_value) VALUES ('%s', %.4f);"
echo $sql | mysql -h "$MYSQL_HOST" -u "$MYSQL_USER" -p"$MYSQL_PASSWORD" "$MYSQL_DB"But put ten of those into a sh file at a time, and create thousands of sh files.
The main server wrote those and the “workers” picked up a file over an sshfs filesystem and ran it locally. When they were done, they deleted the file and that’s how it was removed from the “queue.”
So lots and lots and lots and of bash scripts now running in a distributing computing environment over ssh on old Pentium 3s my employer had no use for.
I don’t know if I spent more time figuring this out than just letting my Pentium run it.
And still symlink to the original images. Ain’t nobody got enough space for two copies of those jpegs.
At least nobody still eating ramen twice a day.
To this day, the only distributed computing system I every made, and, I believe, one of the more unique systems a person could have come up with.
