Digital Media Storage Methodologies

[Nuluvius]

I have a mess of digital media to sort out, photos and videos, and I want a robust system to move forward with. I've been doing some research and want to offer up what I've found so far for discussion. I think I like the three tier system but need a final shove to decide. To be clear the top most root would be Pictures or Videos, these types are separate.

Three Tier
YYYY->YYYY-MM-DD Event->Photos

• Flattest, most denormalized methodology
• Feels like year directories could get very bloated
• Special events i.e. holidays, that span monthly boundaries would be visually contiguous
• Complex day events could have further sub directories:
  • Those with many sources such as other people and camera types
  • Events that are long like graduations may be divided into sections

Four Tier

1. YYYY->YYYY-MM->YYYY-MM-DD Event->Photos
2. YYYY->MM->DD Event->Photos
3. YYYY->MM Name->DD Event->Photos

• Breaks things down further by month under the year level
• More nested than Three Tier methodology
• Holiday events that span month boundaries would be broken up by month so not visually contiguous
• Same rules possible for complex events and multiple sources as Three Tier

Five Tier

1. YYYY->YYYY-MM->YYYY-MM-DD->YYYY-MM-DD Event->Photos
2. Variants as above

• Plus, all the naming variants as in Four Tier
• Deep nesting
• Could potentially have one or two directories in days
  • These could contain many or few photos
• Same rules possible for complex events and multiple sources as Three Tier

General Comments

• With the Three Tier methodology the concern of ‘keys’ is not there because everything is flat under the year directory.
• > Three tiers then we might start thinking in terms of a relational database i.e. primary and secondary keys:
  • YYYY is a primary key [PK] at level one but is a foreign key at level two and so on
  • YYYY[FK]-MM[PK]
  • YYYY[FK]-MM[FK]-DD[PK]
• Do we even want such a bloated naming convention with more than three tiers?
• It might be unwieldy to deal with anything beyond four tiers, we are humans and not computers after all
• It is undesirable to rely on additional software such as Lightroom
• The system must be as future proof as possible
• It should be intuitive for all that may encounter it – this will be handed down and added to through the generations
• Metadata is nice but it should be an independent concern that can be dealt with later/separately

I would greatly appreciate your thoughts and input.

 

[KIPAX]

YYYY - > YYYY-MM -> YYYY-MM-EVENT -> Photos
(No DD)

I have half a million pics over 20 years ..With an average of 30 events in each month and i find it simple and quick to find what I need..

 

[AndrewFlannigan]

I'm gradually moving from a year-day filing system (which currently holds 200,000 images) to one based on a "best practical" textual naming system. As my operating systems are all Unix derived I'll then build a simple shell script around "ls" and "grep" that should let me get to the files I want quite quickly.

My use is personal, so I'm not proposing this for commercial purposes.

 

[Snapsh0t]

I use the three tier system to store unprocessed originals but finished days/events are stored in a tree structure that matches my web site.

 

[Major Eazy]

@Nuluvius



Three Tier system is good.

Under the Four Tier system…

I’m not sure about Option 1. Even folders on their own would take up some memory storage, granted it is only a very tiny amount like barely 1KB, so that may sounds peanuts, but the more folders you have, the more they add up. So I figure, remove the YYYY-MM subfolder, just have the YYYY-MM-DD Event subfolders directly under the YYYY folders.

But then again, if you’re shooting EVERY day, I guess 365 sub-folders under the YYYY folder is too much to search for, so it does make sense to break it up into month subfolders.

So option 2 and 3 sounds better.

Under the Five Tier system…

Option 1 is a bit of a mouthful. Every little characters you use in the folder names, does count towards storage space.

YYYY -> MM -> DD would be better than YYYY -> YYYY-MM -> YYYY-MM-DD.

People often thinks it is the file size of their photos in the folder that eats up the disk’s memory storage space, not realising that even a lot of folders and long folder names, while being very tiny, does add up. You could have 10,000 folders with long folder names, but that could use up maybe 10MB. Does not sound much, but it could be worth a couple of extra photos (depending on camera). But then again, that's fine for the old days when HDD used to be measured in MBs and something like up to 500GB, but I guess it's now a moot point considering that HDD are nowadays measured in TBs.

You said something about Pictures or Videos. I assume that imply you want to save some photos and some videos. Maybe it could help if you have something like…

Photos
¦---YYYY

Videos
¦---YYYY

Instead of

YYYY
¦---YYYY-MM-DD Event
----¦------Photos
----¦------Videos

You move your mouse and clicked on Photos then said “Opps! I meant to click on Videos!”

Well, that’s your choice if you want Photos and Videos under YYYY folders, or if you would consider Photos having its own main folder, and Video having its own.

I use that. I have Photos -> YYYY

And videos have its own folder, since I have less videos than photos, plus it avoid accidently clicking on Photos when I want to click on Videos.

You do seems to have a good way of analysing your folder options, and finding better options. Hope you found your dream filing system.

 

[Major Eazy]

I'm gradually moving from a year-day filing system (which currently holds 200,000 images) to one based on a "best practical" textual naming system. As my operating systems are all Unix derived I'll then build a simple shell script around "ls" and "grep" that should let me get to the files I want quite quickly.

My use is personal, so I'm not proposing this for commercial purposes.

View attachment 279169

200,000 photos on that Olivetti machine? MY FOOT!! You're lucky enough to get maybe a couple of pixel art images!

 

[andrewc]

I use YYYY->YYYY-MM-DD Event HI and YYYY->YYYY-MM-DD Event LO

I'll have a subdirectory for edits over and above my standard export recipe.

I usually shoot with two memory cards with different file sizes on each card, and I usually look at the lo-res images first - as I'm often away on a shoot with limitations on my time and laptop battery - hence looking at the lo-res images. I'll flag various images with colour classes, star ratings and the flag option in lightroom. Once back at a place with more time and mains power I'll look at the hi-res images and will sync the flags, star rating and colour classes from one directory to the other within lightroom using the Syncomatic plug-in.

 

[AndrewFlannigan]

200,000 photos on that Olivetti machine? MY FOOT!! You're lucky enough to get maybe a couple of pixel art images!

I'll have you know that when I bought that M21 in 1983 it was the fastest PC clone on the market!

The following year I fitted it with a 10MB hard drive on an ISA card that gave me the same capacity as the machine we used in the office to support 8 users. What's more I could carry it nearly a quarter of a mile before my shoulder joint started to tear open.

Nowadays my data server is a Mac Mini that weighs about 1/20th of the M21, will fit in the poacher's pockets of my waistcoat and has 3 Terrabytes attached. (and this picture reminds me that I really must dust that shelf more often )

 

[Nuluvius]

@Major Eazy excellent reply and points on data usage, thank you.

Well, that’s your choice if you want Photos and Videos under YYYY folders, or if you would consider Photos having its own main folder, and Video having its own.

I use that. I have Photos -> YYYY

Yes indeed, exactly as I would have it. Sorry I should have been more explicit with that.

You do seems to have a good way of analysing your folder options, and finding better options. Hope you found your dream filing system.

Thank you and I must say at this point, considering that I have asked the same question in a number of different places, the consensus thus far seems to be the three tier system for general use cases:

I use the three tier system to store unprocessed originals...

Three Tier system is good.

I use YYYY->YYYY-MM-DD Event HI and YYYY->YYYY-MM-DD Event LO

While the four tier system may be better suited the heavier use cases:

YYYY - > YYYY-MM -> YYYY-MM-EVENT -> Photos
(No DD)

I have half a million pics over 20 years ..With an average of 30 events in each month and i find it simple and quick to find what I need..

But then again, if you’re shooting EVERY day, I guess 365 sub-folders under the YYYY folder is too much to search for, so it does make sense to break it up into month subfolders.

Potentially many more than 365 considering multiple events per day.

I may trial the three tier system and see how it looks.

 

[admirable]

I store my images Event Name - YYYY

 

[seaodyssey]

I have just been sorting out all my images as well, It has taken weeks to get it more or less done.

I Have
Photos
Photos 1996 to 1999
1996
YYYY-MM-DD
1997
1998
1999
Photos 2000 to 2009
ETC

I have used DIGIKam for Keywords, such as Location( Country, town, place), Subject( car, airplanes, Ships, Boats), People(Names) , Techniques(Focus Stack, B&W), Events(Birthdays, Weddings, Shows)
Digikam can look at the folders or keyword anyway you want it to.
I have a lot more keywords than above, but it gives an Idea

I have also sync the folders between my NAS system and On Local Disk and I will take a backup copy on a portable disk along with the DigiKam Database. I don't use Digikam for Editing.

Pete