Managing / sorting out 8tb of photos in Lightroom

James Blonde

James Blonde

Messages
385
Name
Scott
Edit My Images
Yes
Finding I've dug myself into a hole over the years and made the fundamental mistake of never culling my photos and just throwing hard drives at my local PC when I run out of space. I don't think this is sustainable any more, especially as my PC doesn't have any spare hard drive bays. Sure, I could just delay the inevitable and get a 12, 14, 20tb hard drive, but they're noisier than I'd perhaps want and just feels like storing up problems in the event of a hard drive failure or something. So maybe now is the time to actually start actually managing my photos.

I have one lifeline here - all my photos are star rated based on physical quality of the image. (though not on subject, whether they're duplicates (I tend to over-shoot to compensate for not using a tripod when I should) or even likeability / whether I would even process them) So theoretically at least, I could probably bin the 1 and 2 star rated photos without an issue

I've got my images backed up on at least 1 NAS, plus they're backed up on Amazon Photos. Local PC and NASs are on a cabled 1Gbps network. I'd also need to figure out how to make the changes to NAS and Amazon too (assuming I want to - I think I'd want to at least replicate the changes to the NASs - Amazon is "free" so not sure I care)

They're all in 1 Lightroom catalogue, which I think is also likely to be a mistake. I do have a managed file structure - year-month-date, Location or subject, camera model / whether infrared or colour. I think I need to move (back) to sidecar images for managing metadata and RAW processing changes, rather than having it stored in the Lightroom catalogue - I've nearly had issues here when I thought I'd lost my lightroom catalogue during an upgrade and lost all of the star ratings / RAW changes. I thought I'd made this change already in Lightroom, but it doesn't seem to have taken so think I might have done something wrong.

I think I probably also need to do a mass keywording / general location tagging update - maybe that'll encourage me to go back and look at some of my old stuff that I've never processed!

and finally, I also have a laptop, which I'd love to be able to link in to all of this somehow. It's currently a mish mash of stuff I've saved there as holiday back ups or to work on when I'm on holiday and then never synced in any way, and seems to be in a completely different filing system

How would you approach this?
 
None too sure and glad it is not mine ;)

Having said that..... Excite FOTO might be of help? It is AI driven and when I trialled and the bought an early version (it was not quite the DAM i thought it might be) but IIRC it was very quick displaying images and to sort them using its own ID keywords.

It may not be a whole solution for you but perhaps worth looking at for speed of sorting onto dupes and "similars" for culling as needed to reduce the burden of time sorting such a large volume of images???
 
It doesn't sound too bad to be honest!

How I would approach it is to use your existing star ratings already - go through the 1* photos one year at a time (you can quickly filter in the library module with "\"). Assess if you want to keep the photos, and delete any that you don't need. Repeat for each year - splitting it down by year makes it a more manageable task. Then do the same for the 2* images, and see how you get on. If your NAS isn't a sync of your main machine, I don't think it hurts to keep them on there.

Going forward, you could set up a smart collection to gather any 1* or 2* images older than 1 year, for you to "spring clean".

I would also back up your Lightroom catalog, and set the weekly reminders to back it up when closing Lightroom. There is also the option to write the changes to XMP files, as well as the Lightroom catalog.
 
Last edited:
[sad alert]

I keyworded my entire catalogue from nothing using the following method. Once it's keyworded properly, I can find pretty much anything quickly and easily. My keywording was a mess, so I removed all keywords and started again from scratch. You'll need to adjust this method if you've already half done a job.

1. Come up with a bunch of genres you regularly shoot in. It's pointless giving you a list because if you don't do macro, or portraits for example, it's wasted time. I came up with 9 because I could then create a custom keyword set of those 9 keywords. This is what mine looks like.
Screenshot 2023-09-22 164622.png

2. Create a collection set for keywords. Mine is called "Keywording to do".
3. Within that set, create a Smart collection looking for all photos without a keyword. This is your main "to do list". The good news is that as you add more photos the job just gets bigger so you can work on whatever you want, whenever you want and the smart collection will just pick up what's not been done. You don't need to stay on top of it, you can just trackle it all as and when. You don't need to go through things by year and remember where you were up to, the smart collections handle it all for you.
4. Create another smart collection for each category looking for that keyword and no colour label.
5. Create a final smart collection called "Keywording complete" and have it look for a flag you set, to define the keywording is complete. I use a green label, but you could use a flag, or another colour, or the keyword "complete".
6. Start working through your photos giving each one a category from the list in 1. This isn't too bad of a job because things like portraits or landscapes will tend to be grouped together so you can multi select and add the keyword. As you add them, they will drop from the "no keyword" smart collection, and then start to populate the category collections.
7. Once you've done this, you've then got a bunch of categories to work through, which again, you can do at your own pace, whenever you decide to face it. For keywording, I try and think of a sentence to describe the picture, then pick the three most important words that define it. By going through categories you can set new custom keyword sets that apply to more specific genres. For portraits for example, I have a keyword set with the most common family members. For animals, I have a keyword set for pet names etc etc.
8. When you've finished keywording an image, apply the flag (in my case a green label). This moves it out of the genre collection and into the "keywording complete" collection.

This is an example of my setup today. As you can see, I have 398 photos with no keyword, and 6 that I've given a genre to, but haven't keyworded yet, and 10k that are done. I haven't done any keywording for a while!

Screenshot 2023-09-22 170212.png

You can then set up other smart collections looking for (for example) all your "1* with landscape photography as a keyword", or "family portraits of john and laura". It really does make finding your images so much easier.

[/sad alert]
 
Thanks all, this is great stuff!!!

@Box Brownie thanks, I'll have a look at that software - might give me a bit of a head start at least!

@Craikeybaby, thanks! You give me hope! :D yeh the NAS copy is just a straight duplicate - not synced. Always felt there was too much risk there for some reason!

@Harlequin565 nowt sad about that! I wish I was that organised!!!

I've never gotten into Lightroom - only ever used it for importing until recently, and even then it was just a straight import into the existing filing system. Filing system itself is fine as well. Did toy with breaking the folders up into years, but seemed pointless from a storage perspective. I never liked the lightroom interface to the point of using ACR in Photoshop instead (yeh, I know...) but have at least progressed away from that! Maybe it's time to actually learn it. It's not like I don't work in technology or anything :rolleyes:

​

 
One thing I learnt, don't delete anything.

Start a new system on a new disc/s, and copy what you want to it, what you don't want, leave on the old system.

I did this about 18 months ago, now I gradually go through the "old" system and delete bad copies of what there is a good one (ie if I have 5 photos of the same thing at the same time, but only one is in focus, I delete the other 4)

In time, things are getting in order, and I have everything I could possibly want or need.

I only sort by camera, and within that, year. I don't rate things, that is too subjective, and can change :)
 
Thanks @Sangoma - I've got everything in triplicate (at least) so could be an option to leave "everything" somewhere - probably Amazon as I'm not paying any more than my prime membership for it. I think that's effectively what you're proposing - with the "new" system being me clearing down my live catalogue.

FWIW, my star rating system is based on sharpness so more objective than subjective. 1 is totally black / movement blurred out of existence / totally out of focus and unrecoverable, 2 slightly less so but not worth the effort to recover or unlikely to be recovered, 3 is meh - can probably still be used as a sentimental or record photo, or maybe something I could get creative with, 4 is usable but maybe not tack sharp, or maybe some of it is tack sharp but not necessarily all of it, and 5 is tack sharp exactly where it needs to be. It might be the worst possible picture that I'd never use or look at, but it's sharp - 5 stars! :D

So the intention would be to only keep usable photos, regardless of if they're any good or not! I can be subjective from there, but as you say, probably keep them all.
 
In a similar vein to @Harlequin565, I make use of Smart Collections in Lightroom to help with admin - I add a colour label when I export/print an image, and/or star rating with 5* being portfolio worthy. I then have a filter to get any images without a colour label or <4 rating that are over a year old, i.e. an average photo that I have not shared. I then delete them, on the basis that if I haven't done anything with it in a year, I can't see myself doing anything with it now. I do have everything backed up on my NAS though.
 
Craikeybaby said:
In a similar vein to @Harlequin565, I make use of Smart Collections in Lightroom to help with admin - I add a colour label when I export/print an image, and/or star rating with 5* being portfolio worthy. I then have a filter to get any images without a colour label or <4 rating that are over a year old, i.e. an average photo that I have not shared. I then delete them, on the basis that if I haven't done anything with it in a year, I can't see myself doing anything with it now. I do have everything backed up on my NAS though.
Click to expand...
I think that's likely to be the key - keeping a copy of everything either on my NAS or on Amazon Photos. I'm quite distrusting of cloud service providers though, so maybe that solves the other problem of how to / whether to manage my NAS photos - maybe I just do that lightly
 
I had a similar problem (although not as severe!!!) a few years back. Having (at least) 2 copies of everything helped since I could use one set as a backup and work on the other. Here's how I tackled it...

I broke the job down into manageable chunks - for me, that was about an hour at a time. I then went through them a folder at a time using Windows Photo Viewer (IIRC), keeping everything I still liked the look of and deleting the dross (OOF, too badly exposed etc.). There were a fair few very similar shots (sunrises, sets, waves and similar) which (mainly) got cut down from 10 to 2 or 3 - sometimes even just one. Once I was happy with my selection, I renamed the folder to *foldername*culled so I knew how far I'd got and (since I had an empty HDD handy) saved a copy of it as a backup.

Pretty sure I didn't dispose of any real gems (precious few were ever there!!!) and if I did, I haven't missed them - the very best get printed big anyway. Took a few weeks for me to work through all the folders and probably needs doing again on the more recent shots...
 
Thanks @Nod - makes sense! I'm pretty bad at thinking about the whole problem at once, rather than trying to think of it in manageable chunks :rolleyes:

I guess the only outstanding question at the moment is, it trying to manage all of this in one catalogue the best route still? Or should I consider splitting it? (I dunno, by year or something...?)
 
I wouldn't bother splitting it. Unloading/reloading catalogues is a pain, and when LR updates and you have to recompile all of them that would be tedious.

Pro's I know have separate catalogues for paid work vs personal, and I keep separate catalogues for screenshots and game modding images. But if I'm looking for an image and can't remember what year I took it i.e. "When did we go to Morocco love?" that would be a nightmare for me. Obviously YMMMV.
 
I am also in the 1 catalog camp. Then you can use collections to group things - Lightroom automatically lets you filter by year. My catalog goes back to 2007, when I started using Lightroom v1.
 
For those 1 and 2 star images... you could do what I did which saved a chunk of space (Assuming they are RAW)

Export them into a sub folder as jpegs with a lower quality, imported into LR

Then delete the RAWs

You've got the RAWs on Amazon in case you want to go back in the future and you've a small image on your catalogue to remind you it is there...
 
I can't add any help, but wanted to say this thread has been useful. I've got several hundred gigs of raw files I want to clean up. I intend on deleting the chaff - and I'm now in good habits with regards to importing and culling.

Really good advice!
 
Similar boat really with near identical setup.

I ended up simply culling the unrated or low scoring photos locally. I figured that if I hadn't found a great picture in a folder of 200k images from 11 years ago then I probably wasn't going to miss it!

The Lightroom library filters in the lower right let you filter out specific types so you can say all un picked photos or less than say 3 . I then just selected them and deleted them.

My sync software (freefilesync) mirrors any change between my 8tb photo drive and my backup nas, so anything deleted from or added from the main drive mirrors over.

Amazon is set to just copy everything up from the photodrive (not my storage so don't care that there's a load of old s*** in there) , it's there if anything happens to the stuff in my house.

It was fairly easy to spot problem folders, anything with 1k+ images in it probably hasn't been culled.

re: the laptop, easiest way is to ignore the home based stuff.... I'm guessing you're using it for editing photos when away ?

Best thing to do is invest in an external USB 3 SSD (I use a 2tb Sandisk extreme thing) and set the laptop to use a catalogue on that disk and store the photos on that disk, Keep the folder structure the same as your home based stuff. All you have to do then, is when you get home, attach the external ssd to your home pc and use the import from catalogue function. Job should be a good'un then, all your new photos wil be in your main home based catalogue and file structure and backed up like normal.

Only thing to bare in mind with that, is that if you use the flickr connection it wont sync over any photos uploaded.
 
Last edited:
Box Brownie said:
None too sure and glad it is not mine ;)

Having said that..... Excite FOTO might be of help? It is AI driven and when I trialled and the bought an early version (it was not quite the DAM i thought it might be) but IIRC it was very quick displaying images and to sort them using its own ID keywords.

It may not be a whole solution for you but perhaps worth looking at for speed of sorting onto dupes and "similars" for culling as needed to reduce the burden of time sorting such a large volume of images???
Click to expand...
Can you remember how well it identified the images? My photos go back 20 years now and as time goes on I'm finding it hard to remember which photos are where so I've been looking at a simple platform independent process that could catalogue them, since this software appears to be able to add the keywords to the photo meta files it looks like it would be worth the money.
 
Funnily enough I was talking this over with another wildlife togger last week and we both came to the same conclusion there’s virtually no point in keeping files that your never likely to look at again let alone process .. maybe one or two outstanding shots but that’s it in the rest
 
When you have keyworded, or if you have keyworded - it's not a bad idea to select all of one keyword, and go through them. For example 'Blue Tit' - I seem to have hundreds over the years - and the later ones are better than the early ones, so I'm going to delete most of the early ones even though they were the 'best' ones when I actually took them.
 
I see it like this... what do I take photos for? For me, a photo has to trigger a vivid memory, waymark a place, person or event, stir an emotion, or just look damned interesting from thumbnail to image. I keep everything for a year or so, and then have blasts of going through my stuff. If it doesn't do any of those things, it can go. I don't need a hundred different shots of one thing, so I choose one or two. The exception is my kids, where I back up every shot [cloud and HDD] because I can't bear to delete.
 
Last edited:
the black fox said:
Funnily enough I was talking this over with another wildlife togger last week and we both came to the same conclusion there’s virtually no point in keeping files that your never likely to look at again let alone process .. maybe one or two outstanding shots but that’s it in the rest
Click to expand...
You're probably / almost certainly right, but it would be a SERIOUS mind shift for me personally - one I'm not sure I am capable of taking :/ BUT one I will need to take at some point in my life....

Mind you, if I did manage that, there's no telling what else in my life I might decide I no longer need!!!! :eek::p
 
the black fox said:
Funnily enough I was talking this over with another wildlife togger last week and we both came to the same conclusion there’s virtually no point in keeping files that your never likely to look at again let alone process .. maybe one or two outstanding shots but that’s it in the rest
Click to expand...
This is more my approach now, but 5 years ago I was more... Er... Generous with what I kept. Now I take fewer and cull before import based on the previews - I've never had a photo look bad in the preview but good in full size, you can see where focus is missed or light is bad or the composition just plain doesn't work.

And you can see when a photo has all of those issues, but you still like the memory and keep it anyway.
 
The thought of it gives me palpitations.

I'd certainly think about chunking it up, but the real question is how do you do that?

By year and/or month? this is pretty much how I manage my shots. I have a folder for each year, then each of those is broken down to month although I also give titles to holidays or special events. Sort of like this

1697032097189.png

I just find it easier to manage it by month.
 
Interesting subject. Culling is a very subjective activity. When I go back to pics shot a couple of years ago, I really wonder why I'm keeping many of them. Yet there are a few that become more interesting with passing time.

Wouldn't it be wonderful if Lightroom C had focus peaking. I shoot a lot of bugs and birds, often in less than ideal circumstances, and focus can be off. So I take lots of shots, and then try to select the sharpest afterwards in Lr. It is very tedious and time-consuming to pick the best of a bunch. If the software would highlight the sharpest areas, that would speed things up.
 
That would be a good feature! But alas it seems like they are keener for us to add artificial blur to our images...
 
You must log in or register to reply here.
Back
Top