Working with multiple LR catalogs

[Livin The Dream]

I nearly asked similar on another recent thread below and so save gatecrashing just wanted to clear a couple of points up.

Some of my older stuff I have archived off as a catalog on it's own, so for example a yearly catalog. Is there an easier way of just deleting all the other years and saving?

My current catalog location is referred to the one that I last opened (a few months ago) and each time I exit and save, it creates an additional backup file. Is LR still referencing the file from several months ago? I've deleted back ups in the past and accidentally deleted the file that it references which created a mini headache.

I have come across problems in the past moving from one catalog to another. Simply you open the zip file and open in LR, correct? Is it a simple as that?

 

[Livin The Dream]

I've been trying to suss whether it's any better working with smaller catalogs. My catalog spans the last 5 years, has 30,000 files, is 8.6GB, with each backup file being 360MB. It's generally personal work.

 

[Nawty]

Yep, just unzip and open.

I need to give some serious consideration to my backups. At the moment I have all the previous year's raw files on an external drive and that is indexed to the current catalogue, I also have a time capsule backup and cloud storage (iCloud and Google Photos) of all my exported JPGs.

Having had a few catalogue near-disasters when trying to sync laptop and desktop across iCloud (don't bother) I wonder if there is a better way. Thing is, I never go back and re-use the raws so I wonder if I could ave a whole lot of space by either deleting or converting to JPG, then I wonder if it's worth the bother as storage is cheap...

 

[PhilH04]

I've been trying to suss whether it's any better working with smaller catalogs. My catalog spans the last 5 years, has 30,000 files, is 8.6GB, with each backup file being 360MB. It's generally personal work.

Thats not a huge catalogue,so I wouldn't worry too much. Different people have different ways of working, if you feel more comfortable having smaller or more generalised multiple catalogues you can do so and have LR present a dialogue to choose one when opening, just make sure you give each catalogue a meaningful name.

To your other post, not quite sure what you are getting at, catalogues have the extension .lrcat, so don't touch anything with that extension, your backups will have the .zip extension appended, you can delete probably all but the last two created.

If LR is backing up the catalogue each time you exit then that is the way you have it set. you can alter the frequency of this backup in LR preferences to lets say once a week, it really depends on how often you are using the app.

The .zip files are not different catalogues, they are only the backup files so you really do not need to touch them unless your .lrcat file has become corrupt.

 

[PhilH04]

Yep, just unzip and open.

I need to give some serious consideration to my backups. At the moment I have all the previous year's raw files on an external drive and that is indexed to the current catalogue, I also have a time capsule backup and cloud storage (iCloud and Google Photos) of all my exported JPGs.

Having had a few catalogue near-disasters when trying to sync laptop and desktop across iCloud (don't bother) I wonder if there is a better way. Thing is, I never go back and re-use the raws so I wonder if I could ave a whole lot of space by either deleting or converting to JPG, then I wonder if it's worth the bother as storage is cheap...

Yes there is a better way... I work on the same catalogue on both laptop and desktop, the main catalogue and the preview files are on a separate external hard drive, it doesn't need to be huge and I just move that and another drive with the main image files between the two computers... everything synched up perfectly. Just make sure that the backups go to a different drive.

Oh' you shouldn't need to open a zipped backup unless something has gone wrong with you main catalogue file.

 

[Nawty]

Yes there is a better way... I work on the same catalogue on both laptop and desktop, the main catalogue and the preview files are on a separate external hard drive, it doesn't need to be huge and I just move that and another drive with the main image files between the two computers... everything synched up perfectly. Just make sure that the backups go to a different drive.

Oh ad you shouldn't need to open a zipped backup unless something has gone wrong with you main catalogue file.

Problem there is everything being on an external drive and having to carry that drive, when I'm using the laptop it is because I am travelling and I travel light so an extra drive seems like hassle.

What I tend to do is just export catalogues and import to the desktop iMac. Actually more and more I just edit on the laptop so the iMac is more or less just a backup station.

I'm planning to try using an iPad for an upcoming trip and see how LR CC works - supposedly it syncs everything for you and having had a play it will do everything I want except photomerge (panos and HDR)...

 

[Livin The Dream]

Yep, just unzip and open.

I need to give some serious consideration to my backups. At the moment I have all the previous year's raw files on an external drive and that is indexed to the current catalogue, I also have a time capsule backup and cloud storage (iCloud and Google Photos) of all my exported JPGs.

Having had a few catalogue near-disasters when trying to sync laptop and desktop across iCloud (don't bother) I wonder if there is a better way. Thing is, I never go back and re-use the raws so I wonder if I could ave a whole lot of space by either deleting or converting to JPG, then I wonder if it's worth the bother as storage is cheap...

I have my backups in the cloud. Know exactly what you mean trying to sync that way, was a disaster too.

Thats not a huge catalogue,so I wouldn't worry too much. Different people have different ways of working, if you feel more comfortable having smaller or more generalised multiple catalogues you can do so and have LR present a dialogue to choose one when opening, just make sure you give each catalogue a meaningful name.

To your other post, not quite sure what you are getting at, catalogues have the extension .lrcat, so don't touch anything with that extension, your backups will have the .zip extension appended, you can delete probably all but the last two created.

If LR is backing up the catalogue each time you exit then that is the way you have it set. you can alter the frequency of this backup in LR preferences to lets say once a week, it really depends on how often you are using the app.

The .zip files are not different catalogues, they are only the backup files so you really do not need to touch them unless your .lrcat file has become corrupt.

Thanks Phil. LR updates your catalog a little as you go as I understand it, so do you not need to backup every time on exit? If I import a new job then is it best practice to backup afterwards for example and if I'm just working on a few files from previous not?

Is it better practice to have your .lrcat separated from the zip back ups? I've been saving a monthly backup when I go back and clean them up.

 

[gremlin16]

My personal photos are in one big catalogue.

Each paid job gets a catalogue of its own, on a disk purely for paid work. A backup of the master races goes on a separate disk too, and a third set onto a cloud.

 

[User.82148]

I thought most advice was not to use multiple catalogues. Have a look at Abode LR help and support and websites like Lightroom Queen.

 

[PhilH04]

Problem there is everything being on an external drive and having to carry that drive, when I'm using the laptop it is because I am travelling and I travel light so an extra drive seems like hassle.

What I tend to do is just export catalogues and import to the desktop iMac. Actually more and more I just edit on the laptop so the iMac is more or less just a backup station.

I'm planning to try using an iPad for an upcoming trip and see how LR CC works - supposedly it syncs everything for you and having had a play it will do everything I want except photomerge (panos and HDR)...

I am looking at getting an external SSD for my catalogue and previews as I tend to take the laptop when away,they are obviously a lot smaller, the other thing I am going to try is using an SD card, both computers have SD slots.

I believe LR CC isn't compatible with iOS, I think you need LR Mobile for that.

 

[drdocmatt]

Multiple catalogues is going away in LR Classic CC, so don't get used to it. They envisage you moving to the one master catalogue that sits in the cloud.

For standalone users unless you start reaching hundreds and hundreds of thousands of photos I wouldn't worry about archiving parts of the catalogue off, for personal users anyway. For professionals once a job is complete I can fully understand the desire to archive out the RAWS and the catalogue to long term archive.

 

[PhilH04]

Thanks Phil. LR updates your catalog a little as you go as I understand it, so do you not need to backup every time on exit? If I import a new job then is it best practice to backup afterwards for example and if I'm just working on a few files from previous not?

Is it better practice to have your .lrcat separated from the zip back ups? I've been saving a monthly backup when I go back and clean them up.

Yes to your first sentence, simply set the frequency in preferences, mine is set to backup each time on exit, but it is a large part of my living.

Second sentence leave your .lrcat file where it is but make sure your backups are saved to a different hard drive.

 

[PhilH04]

I thought most advice was not to use multiple catalogues. Have a look at Abode LR help and support and websites like Lightroom Queen.

Yes and no... there are so many different ways of working and none is right or wrong, I know of wedding photographers who create a separate catalogue for each wedding. I always advise to use whatever system you are most comfortable with.

 

[PhilH04]

Multiple catalogues is going away in LR Classic CC, so don't get used to it. They envisage you moving to the one master catalogue that sits in the cloud.

No its not... Classic is the same as any previous version of lightroom, it is the new LR CC that is cloud centric.

 

[Livin The Dream]

Yes to your first sentence, simply set the frequency in preferences, mine is set to backup each time on exit, but it is a large part of my living.

Second sentence leave your .lrcat file where it is but make sure your backups are saved to a different hard drive.

I keep my backups on my local Mac drive at present, which backs up to the cloud, but they are in the folder as my .lrcat. My files are on a mirrored RAID setup and both are additionally added to a NAS too (just because there's additional space there and wifi isn't quick enough to work on them, otherwise I'd just use the NAS)

 

[Byker28i]

Multiple catalogues have their uses for separating collections, i.e. work and personal, but personally I have all mine in a single large catalogue, because then you get to use the power of the library module for searching images.

Problem there is everything being on an external drive and having to carry that drive, when I'm using the laptop it is because I am travelling and I travel light so an extra drive seems like hassle.

In this case there's a valid reason to use a new catalogue for the trip when traveling to work on just those images and to keep the catalogue small. Then merge the catalogue into your main on return

Julieanne kost has some great tutorials on this
http://blogs.adobe.com/jkost/2015/0...lightroom-catalogs-into-a-master-catalog.html

 

[Byker28i]

I keep my backups on my local Mac drive at present, which backs up to the cloud, but they are in the folder as my .lrcat. My files are on a mirrored RAID setup and both are additionally added to a NAS too (just because there's additional space there and wifi isn't quick enough to work on them, otherwise I'd just use the NAS)

I have this years raws on a SSD, catalogue and cache on another SSD, OS and programs on another SSD, then have 20Tb of raided drives that store the previous years raws and exports. Ok I've been lazy this year and not moved images as I've shot fewer. All mine are stored in year folders, and by data underneath (done on import).


Backs are run weekly to an external Nas (HP miniserver running robocopy or Syncback) and to external drives, rotated to my desk at work.

 

[drounding]

I use a catalog for each event. I then archive the originals and exported files all together in one folder to external drives.

 

[User.82148]

I have this years raws on a SSD, catalogue and cache on another SSD, OS and programs on another SSD, then have 20Tb of raided drives that store the previous years raws and exports. Ok I've been lazy this year and not moved images as I've shot fewer. All mine are stored in year folders, and by data underneath (done on import).
View attachment 113015

Backs are run weekly to an external Nas (HP miniserver running robocopy or Syncback) and to external drives, rotated to my desk at work.

Very organised. Do you find the SSDs much quicker for loading files?

 

[drdocmatt]

RAWs on SSD? Hell yes, the low latency helps a great deal when moving around the library. Not so much when working in develop, but a big help in general responsiveness.

 

[Byker28i]

Very organised. Do you find the SSDs much quicker for loading files?

Yes. I have an Evo 950M2 ssd for the OS which is rapid, then 850SSD's for this raws and the catalogue and cache. All 500Gb SSD's which is a little overkill for the cache and catalogue but it all makes lightroom fly.
The PC was deliberately built for photography

 

[Livin The Dream]

I have this years raws on a SSD, catalogue and cache on another SSD, OS and programs on another SSD, then have 20Tb of raided drives that store the previous years raws and exports. Ok I've been lazy this year and not moved images as I've shot fewer. All mine are stored in year folders, and by data underneath (done on import).
View attachment 113015

Backs are run weekly to an external Nas (HP miniserver running robocopy or Syncback) and to external drives, rotated to my desk at work.

I'm probably not a million miles away. Assume you're more than an amateur, not that that makes too much difference, files are important either way. For myself, which is mostly personal, I think a smaller catalog is probably not needed. The main difference you have is separating the OS and the .lrcat which is easier if you're using a desktop. I've gone the MPB route and so I think keeping the .lrcat on the Mac SSD makes sense (providing I backup elsewhere).

Thanks everyone for your input, interesting to hear workflows.

 

[ABTog]

I use smaller LR catalogues. This is my preference as it has always allowed my computer (which has always been of older/lower spec) to run LR better. It also has the benefit of being easier to back up.
I mostly shoot for myself with the occassional paid event thrown in. So generally I have a fresh LR catalogue for each month (or each paid event, keeping them separate from personal stuff).
At the start of each month, I create a fresh catalogue and the previous months catalogue (and all RAW and edited images) is backed up to my external drives. All data on my main drives is backed up to Backblaze as I go.
If you use one big catalogue, if you suffer a corruption of data or the disk drive has issues, then it's going to be a bigger issue.

All my images (RAW and edited jpg) are stored by year, month and then event/trip. Nice and neat.

 

[Byker28i]

Re one big catalogue and corruption, that's why Lightroom creates backups and you have a backup routine.
Having multiple catalogues loses any benefits of the library module as a very powerful way of cataloging all your images. You'd have to know which of your many catalogues to open to find a desired image.

There was a thread reopened on here from 7 years ago of an image I'd taken in Brighton. I found it within 10 secs of opening Lightroom.
There's no real speed difference between small and large catalogues, although maybe if you have old, slow, drives, perhaps on lower spec laptops.

 

[Livin The Dream]

Re one big catalogue and corruption, that's why Lightroom creates backups and you have a backup routine.
Having multiple catalogues loses any benefits of the library module as a very powerful way of cataloging all your images. You'd have to know which of your many catalogues to open to find a desired image.

There was a thread reopened on here from 7 years ago of an image I'd taken in Brighton. I found it within 10 secs of opening Lightroom.
There's no real speed difference between small and large catalogues, although maybe if you have old, slow, drives, perhaps on lower spec laptops.

Completely agree, you'd never find anything.

This has made me look at a few things. For starters, I've still got my phone pictures/videos locally so in the process of stripping all documents out and dumping on the NAS.

 

[ABTog]

Re one big catalogue and corruption, that's why Lightroom creates backups and you have a backup routine.
Having multiple catalogues loses any benefits of the library module as a very powerful way of cataloging all your images. You'd have to know which of your many catalogues to open to find a desired image.

There was a thread reopened on here from 7 years ago of an image I'd taken in Brighton. I found it within 10 secs of opening Lightroom.
There's no real speed difference between small and large catalogues, although maybe if you have old, slow, drives, perhaps on lower spec laptops.

I don't use the LR catalogue to search for images. I've never bothered using keywords and so on, so there's not much to search. I taught myself LR so I never had an explanation of how or why to use the catalogue/library function like that. And as I said, loading large numbers of images/previews/etc made LR slow on my lower spec computer. As I've learnt more of Lightroom's functions I've adjusted my methods, but I've yet to see a reason for me (personally) to change the way I deal with catalogues.
I can see that for some people, the ability to search the library is useful, but it's not something I feel I'm missing.
As for the OP, they'll have to decide for themselves if that's something they need to keep.

 

[Byker28i]

There's another thread on her where someone realised they imported jpegs and raw files and wanted to delete all jpegs. Easily done in the library module if you have one catalogue, time consuming if you have many.

As said, it's your workflow, but many assume Lightroom is an editing app, rather than a full workflow tool. Many also haven't spent the time learning the full features of the tool, so threads like this are useful in discussing the features and functionality that some are unaware of.

 

[drdocmatt]

Fwiw catalogue backups are automatically compressed by default from LR6. Prior to that it was time consuming but it's a lot quicker to write the backups out now.

 

[Livin The Dream]

It doesn't hurt once in a while to think about your strategy, with time, everyone gets a little sloppy at times.

The last 5 years I have a good directory with keywords for the important things, but I keep meaning to go back and tidy up the older stuff. That said, there's not too much there that's as precious as these last few years. It's a Christmas project!

 

[User.82148]

I bought LR principally for the library function as my previous software was not that user friendly. If you set up keywords in a logical manner, I find the library function to be very good and it suits my purpose. Just wish I keyworded more frequently. The development module was a bonus but in general it is all I need. Appreciate I am just an amateur and others have much different requirements.

 

[Byker28i]

So I import by date, standard lightroom function but with a top level folder of the year.
I use keyword tags on import, occaisionally it's nice to search through them by event, place.
Sometimes, not always I'll use the map function and tag photos for places as I don't have gps built in my camera so it's a manual process.
Plenty of ways to search for images

I also heavily use collections and collection sets, these are my go to place for finding images. Top level folders for things like cars, sport, family etc, then subdivided under that. For family it's then year, then event, for others it's usually just event for the collection title.


So many ways to find things.
Be a pain for me if these were all in separate catalogues, but as said, it's all about your personal workflow

 

[Livin The Dream]

So I import by date, standard lightroom function but with a top level folder of the year.
I use keyword tags on import, occaisionally it's nice to search through them by event, place.
Sometimes, not always I'll use the map function and tag photos for places as I don't have gps built in my camera so it's a manual process.
Plenty of ways to search for images

I also heavily use collections and collection sets, these are my go to place for finding images. Top level folders for things like cars, sport, family etc, then subdivided under that. For family it's then year, then event, for others it's usually just event for the collection title.
View attachment 113060

So many ways to find things.
Be a pain for me if these were all in separate catalogues, but as said, it's all about your personal workflow

I’ve started cleaning my early years up today. I file similar but only 18 months ago did I start adding the date in the file, which has helped more than I thought it would.

I’m going to tidy up my catalogue and clean my collections which have been a little random. Moving forward, I’m going to separate paid shoots with their own catalogue, keeping the personal catalogue clean. It’s been a good exercise.

 

[Bollygum]

I'm new here, but this is an interesting thread. I have something like 500,000 photos in total. The main reason for this is that I do time lapse and that collects a lot of photos. As far as I can tell, that many photos is too much even for lightroom, so I have separate catalogues for current and archived folders. One problem I have is that I only include backup photos into a backup catalogue as I reference them. This is fine for normal photos as I usually don't have much in the way of modifications, but for time lapse this isn't always so. If I remember to save metadata it is ok, but I often don't. I'm trying to work out a quick and simple way to move the photos to a backup and at the same time move the catalogue entries.

 

[Livin The Dream]

I'm new here, but this is an interesting thread. I have something like 500,000 photos in total. The main reason for this is that I do time lapse and that collects a lot of photos. As far as I can tell, that many photos is too much even for lightroom, so I have separate catalogues for current and archived folders. One problem I have is that I only include backup photos into a backup catalogue as I reference them. This is fine for normal photos as I usually don't have much in the way of modifications, but for time lapse this isn't always so. If I remember to save metadata it is ok, but I often don't. I'm trying to work out a quick and simple way to move the photos to a backup and at the same time move the catalogue entries.

That sounds an awful lot of files. Are they generally RAW/DNG files or do you convert and save a jpeg edited? I bet you’ve got through a few shutters? What time period have you amassed that amount?

Was reading a thread elsewhere yesterday where someone who wasn’t a pro were shooting 80TB a year. Now that sounded like a proper habit!

 

[Byker28i]

I'm new here, but this is an interesting thread. I have something like 500,000 photos in total. The main reason for this is that I do time lapse and that collects a lot of photos. As far as I can tell, that many photos is too much even for lightroom, so I have separate catalogues for current and archived folders. One problem I have is that I only include backup photos into a backup catalogue as I reference them. This is fine for normal photos as I usually don't have much in the way of modifications, but for time lapse this isn't always so. If I remember to save metadata it is ok, but I often don't. I'm trying to work out a quick and simple way to move the photos to a backup and at the same time move the catalogue entries.

That's an interesting issue to have and depending on the spec of your machine, There is no maximum number of images in lightroom catalogue, but it's just a performance thing and depending on your setup, a very large catalogue will affect performance. After all it is a disk based OS. I followed this with my last PC and built my new one to take all of this into account
https://helpx.adobe.com/lightroom/kb/optimize-performance-lightroom.html

I think this may be a case where I can see the need here for different catalogues just to separate out the timelapse from other work.

At my last company where I shot all the marketing/publicity shots I had around 600,000 images, shot over 4 years in a single lightroom 5 catalogue, but that was stored on a large dedicated San based storage system with 15K rpm disks, connected over 10Gb network. One of the advantages of working for a Broadcast/post production manufacturing company is there was high end kit to be had.

One of the things I did find Lightroom had a file size limit of an image 65,000 pixels long or wide or up to 512 megapixels (Stitched large image when we had access to a gigapixel for a day)

 

[drdocmatt]

When I've done time lapse I've usually processed the originals, done jpg exports and deleted all the RAWS pretty sharpish. I've not seen any need to keep them in LR when I can manipulate the output video file for most use scenarios without having to go back to the original.

 

[Bollygum]

That sounds an awful lot of files. Are they generally RAW/DNG files or do you convert and save a jpeg edited? I bet you’ve got through a few shutters? What time period have you amassed that amount?

Was reading a thread elsewhere yesterday where someone who wasn’t a pro were shooting 80TB a year. Now that sounded like a proper habit!

It is a lot of files. I use 4 cameras for timelapse, a Canon 1Ds Mk3, a Canon 5D3, a Canon 7D2 and a Sony A7R2. At times I will have all of these in operation at the same time and they can generate a very large number of images. Still no shutter failures, but the cameras do get stressed at times. I keep all the files in RAW and lately the jpeg 4k timelapse file plus the videos. I do this because some customers want the RAW files (eg the BBC). I have tried bigger LR cats, but if something goes wrong it is almost impossible to fix if it is very large. I use solid state disks for current work but archives are onto 8TB drives which aren't all that quick.

I'm waiting for a little while to get a new machine as I like to double the speed with each upgrade and I can barely do that yet.

I don't think I'll hit the 65,000 pixel limit for a single file.

 

[drdocmatt]

Yeah there is no doubling of speed any more. CPU core instruction throughput has increased by barely 10% in the last five CPU generations (there's just more of them, and LR is notorious for not taking advantage of more than about 3). Memory performance has improved a couple of times and obviously the move to SSD from spinning rust has been a huge benefit for certain workflows but there's no sea change coming again.

 

[Bollygum]

I can use the extra processors as I do multitask as much as I can, but there are limits. If I go for the latest AMD processor it is actually slower for each thread than my i7. If I use the latest intel, I have to pay a fortune. I'll wait a while. SSD's are good, but limited in space, unless you have no price constraints. Would you believe I bought (for my employer) my first SSD in 1984. It was 256MB and was the size of a large double refrigerator, and probably cost about the same as a block of flats. Now they are cheaper.

 

[drdocmatt]

Heh, no doubt.

Multitasking does benefit from more cores, but the issue is that LR on its own saturates the memory bandwidth on the CPU, it's not cores that restrict performance it's always memory bandwidth. Unfortunately this means you don't get a *lot* of benefit from those extra cores. Such is life. A lot of hpc workloads go for fewer cores and really count memory buses as the bottleneck. This is one of the key benefits of workstation class hardware compared to the consumer orientated chipsets.