Further experiments in JPEG compression

This is an expansion of the experiments on JPEG that I reported in this post. It's worth reading that before continuing here.

This time I was looking to expand the idea to the limit - just how many iterations would it take to cause the image quality to break down?

I took the same image, working with Lightroom JPEG quality 75 and continued to successively expot the full size image. At the 35th iteration I found the first artefacts - pixel blocks, about 3x3 in size, in one of the dark patches. Nothing that would show up in print. In fact, it took a while to find it at all on screen at 100%. I then continued. At 80 iterations I got bored, happy that this is well past the number of successive saves I'd ever need.

I also tried an image that had been sharpened in the first conversion from uncompressed TIFF, using Lightroom's heavy screen sharpening. After 20 iterations there was no difference, apart from the sharpening - no artefacts, halos etc. I'd expect it to go many more iterations without problem.

JPEG torture test

I also devised a torture test, creating a multi-patch image with random detail running across it. This is a graphic image better suited to GIF than JPEG. Result: noticeable degradation around the wiggly line after 10 images at quality 75, but only against pure colour (one of the white patches and the pure green bottom left). The rest was intact, including edges between patches. The file size is small, reflecting the large colour patches: 340kB for a 6MP image, much smaller than the 1.2MB for the other, 10MP image.

Torture test after 10 iterations, Lightroom quality 75
Click for 100% view

Final conclusion on using JPEG for presentation and book making: I'm sticking to lower quality levels than before, using JPEG exclusively for creating photobooks and not getting too worried about compression settings.

Experiments in JPEG compression

I wrote before about optimising jpeg compression for various output, mainly aimed at single conversion for on-screen view. Then a couple of comments on this post by Colin Jago had me thinking more about using jpeg images.

It was suggested, in reference to making photobooks, that TIFF (by inference, uncompressed images) are needed to retain image quality. I don't do that. when using Scribus I import all images as resized jpegs exported with a quality of 9 from Photoshop or 90 from Lightroom. By working with jpegs I minimise memory requirements and speed up software response. Am I losing quality in doing this?

So I've run a little experiment. I took the image above (selected for a mix of fine detail and smooth tonal areas), from its 16-bit uncopressed TIFF format and successively converted it to jpeg. that is, I converted to jpeg, took that jpeg and converted and so on. I ran 10 iterations. I tried 3 different qualities (90, 75, 50 from Lightroom), no sharpening applied in conversion. This is what I found.

The 10th quality 90 (jpeg 90-10) showed almost no discernible difference to the first jpeg (jpeg 90) which is indistinguishable from the original. That at 100% on-screen view. The one slight difference between jpeg 90-10 and jpeg 90 was slight posterization in the darkest shadows. By slight, I mean peering closely and changing my angle of view a lot. At full-screen view it is unnoticeable. It would not show in a print. Each jpeg in this series is 2.6Mb in size for a 10MP image.

jpeg 90-10 crop, click for 100%

jpeg 90 crop, click for 100%

The same story for jpeg 75-10 compared to jpeg 75. These files are 1.2MB for the same image.

For jpeg 50-10 there is definite loss of detail and a number of strange block artefacts in smooth tone areas. I had to go all the way back to jpeg 50-4 (4th in series) before it was indistinguishable from the original. The files in this series are a mere 581kB.

jpeg 50-10 crop, click for 100%. Notice the weird blocks in the smooth areas

jpeg 50 crop, click for 100%

Conclusion: it is quite feasible to use jpeg for successive output operations without losing detail. For short runs (3 or 4 successive conversions) it is feasible to use quite low quality (high compression) and not lose detail. Higher quality will support longer runs. I haven't yet run the 75 and 90 series to the point that they start to lose quality, that's a job for another day.

Scribus output, pdf and jpeg

This is aimed squarely at the SoFoBoMo crowd but should be useful for others putting together pdf books for the web. I know several will be using Scribus as their layout software.

I'm using Scribus as my layout software, as I have done in the past. Normally when I'm playing around with layouts, I export a partial book to pdf to see how it stacks up. Last night I made a minor error in the options I chose and felt it worth pointing it out.

There is an important option in the pdf output dialog (below) where the type of image embedded can be selected. I accidentally chose "Automatic". Running a layout with just 10 images in it gave me a 7MB pdf file, even using the "medium" compression setting. I then re-ran the export but with jpeg selected, suddenly the same material was only 1MB. An issue to note. I don't know what image format Scribus chose under "Automatic" but it clearly wasn't as compact as jpeg. Visually there was nothing different in the output.

Scribus pdf generation: several options for images - chose JPEG

It might be that other pdf generating software has similar options around the format for image embedding, I don't know as I don't have access to other options. If anyone else knows of similar options in other software, post a comment.

More on jpeg settings

Walcott beach, Norfolk, May 2009
From Lightroom, jpeg quality 50, 1200x800
click for full size

Ed Richards had comments on my suggestions on jpeg over at Paul Butzi's. So, as is my wont, I went off to test his points (and mine for that matter).

A few points to start: if one is creating jpeg images for displaying on a screen, then a lot of the original data is thrown away anyway. For a typical 10MP original, going to 1200x800 (I'll work with Ed's big image assumption) throws away about 90% of the original pixels. If you're an LF photog, it's going to be way more than that. So we can forget about on-screen versions ever getting to the fine detail and nuance of a really nice print from a really nice image.
The next idea is that higher quality settings really have an impact on the on-screen viewing quality. If I want to show the very best, I need the higher quality. But how true is that?

I ran a bunch of tests on a series of images with different jpeg quality settings. For this I used Lightroom but any jpeg generator would yield similar results. I ran film & digital originals. I tried detailed & mixed images (with some expanses of limited detail). I tried jpeg quality settings from 10 to 95. I visually compared the results versus quality setting and file size.

Lightroom jpeg quality 10, 1200x800

What did I find? Obviously, at low settings there were problems. Lots of artefacts around edges, detail blurring, pixelation in large, continuous areas. At quality of 30-40 most problems were gone. Some fine details started to go. At 50 most images were indistinguishable from the higher settings. The few differences needed a careful look and hopping between versions. Going from 70-95 showed no improvement on any image.
How about file sizes? At 1200x800, quality of 50 gave about 220kB per image. At 70 that was 350-400kB and at 90 all the way up to 1.5MB. Yet no visible difference between them (and I was looking carefully on a decent, calibrated monitor). 1000x667 images were proportionally (by area) smaller.

Of the hundreds of images I've posted, the average file size is around 250kB and I can't recall seeing compression artefacts or loss of detail in any of them at web sizes.

Like I say, for good, on-screen display, target a file size of around 150kB for 1000x800, up to 200kB-ish for 1200x800. The reason I've been suggesting 1000x800 is that's a 10"x8" at 100ppi, which is a decent size for a book image, working on the basis that we're talking embedded jpegs in a pdf.

Gorse, Cumbria, May 2009
Lightroom jpeg quality 50, 1200x800

Ed also made a point that he wants the highest quality to wow the guys with the huge, high quality monitors. Thus he ends up with a big pdf. Personally, I want a bunch of people to download my SoFoBoMo book, so I'd rather a smaller file size to encourage them to do so. And based on the above tests, I reckon a 5MB book file would be visually indistinguishable from a 15MB one anyway, on any screen.

A photographer's practical guide to jpeg

Stemming from the question on the last post about sizes of pdf books, I thought it was worth writing about JPEG images: sizes, resolutions, compression etc. This is based on a little technical reading and a lot of practical experimentation. Technically, I may be a little off in some areas but this will be good enough for the masses.

The aim of a JPEG image is to preserve as much detail as possible while compressing the file size as much as possible. By setting the quality/compression (these 2 are the reverse of one another - high compression gives low quality) value you are determining the trade-off between quality and file size. However,there is a general misconception as to how this works.

For highly detailed images (think lots of tree branches and small leaves) there is automatically less compression as the JPEG tries to keep the detail. If your image is highly detailed then you can actually use lower quality setting and still get an excellent image. In the other direction, large expanses of nothing compress very well. If you shoot a picture of a white wall even the highest quality setting will give a small file. JPEG is clever like that. The tricky stuff comes when there is a mixture of detail and even tones (e.g. trees against a blue sky). The even part compresses well, the detail is kept in the main but at the edges JPEG gets confused, throwing up those nasty artefacts. This is where judicious use of the quality level is needed. I'll come onto advice on values later.

The next part is the resolution or pixel size of an image. If you are printing, you'll want high quality files with lots of pixels. Typically 240-300ppi for the given print size. That means the total pixel size will be the paper size multiplied by 240 or 300 (e.g. if I print a 10"x8" at 300ppi, that'll be 3000x2400 pixels). Nice and easy.
The tricky part is on-screen display. To display on the majority of monitors, you only need an image about 1000pixels wide or 600 pixels tall. Maybe a bit more to cover larger displays. If your software works as a size & resolution then the size multiplied by resolution shouldn't be more than about 1000x600pixels (e.g. if I have a 10"x8" page for on-screen display, I need a resolution of only 100ppi to give 1000x800pixels). This is important for pdf generation, as pdf is designed to work in physical size and resolution. To cover most monitors today, you don't need more than 100ppi resolution (nor page size more than about 12"x8").

So what about recommendations for compression values? JPEG generation generally has one of three ways of setting quality: a high/medium/low scale; a JPEG compression factor (typically a number up to 100) or another numerical scle of some sort (e.g. Photoshop's 1-12).

For printing, go high. Photoshop 9 or 10 (11 and 12 really are pointless and don't practically give better results), JPEG factor 90 or 95. NOTE: JPEG compression factors are NOT percentages. the maximum value (from the JPEG standard) is 95 - any scale that goes to 100 is distracting you.

For on-screen display, you only need a medium level. Photoshop 7 or 8 (sometimes lower but go careful); JPEG factor 70-75. This will yield a files size about half that of the higher qualities and still give excellent on-screen display. Use the lower values unless there is a big mix of detail and even tones.

A final note on setting resolution versus compression, especially in pdf generation. The biggest win in file size (small being good) is a lower resolution. 300ppi has 9 times the pixels of 100ppi. To get that kind of compression with quality you need to go down to about 20-25 quality factor, which is barely recognisable. You could probably go down to 90ppi and give good display quality even on large monitors (a factor 11 smaller than 300ppi).

As for pdf books for the web (aimed squarely at the SoFoBoMo crowd) - aim for file size that gives around 50-60kB per page or 100-150kB per image overall (non-image pages generally take very little space in pdf). If you've gt 40 images in 55 pages (typical SoFoBoMo fare), that's 2.7-5.8MB total filesize.

Lightroom: nearly but not quite

Regular readers will probably be aware that I've been a long-term Lightzone user but recently have been writing about Lightroom.

Lightroom has almost completely taken over my photo-editing and management duties. It has many features to commend it - simple to use, pretty quick for most tasks, great organising features and solid printing system. However, it is just a few notches short of being quite all there.

First, as Kjell has noticed, are the problems with local adjustments. I'm sure that'll get fixed in time. Most of the time I only want small amounts of local adjustment for which it's great.
Next, and a big one for me, is the handling of sharpening. I use high radius, low amount (hiraloam) sharpening on a lot (if not, most) of my work for local contrast enhancement. I find Lightroom is not so good for that - clarity is a bit heavy-handed and I can't fine tune its effect. And I'm not enamoured of the sharpening tool in the details section. Again, not the control I'm used to with USM (or PWP's excellent advanced sharpen). This is particularly bad for scans (which usually take much more aggressive sharpening). I'm finding I have to take most images elsewhere for sharpening adjustments.
I'd also like to have support for advanced plug-in in the way Photoshop does. I've tweaked my workflow to deal with this but it does mean some back-and-forth.

I could add a list of other things I'd like to see, but then it would move Lightroom away from the things that make it good.

So what about Lightzone? Well, I do still use it but for limited applications. It is still has the best shadow recovery tools anywhere. The ability to do selective white balance is invaluable as is the ability to white balance processed files. I still it has the best masking system around especially after I discovered the clunk that is Lightroom masking. And Lightzone is still my favourite black and white converter: selective masking by area, colour, tone means I can apply multiple filter effects in lots of subtle ways. There are a couple of B&W things I do that I can't do any other place.
But Lightzone is quite rough around the edges: generally slow, memory inefficient and a host of other niggles. Development doesn't seem to be going anywhere and they are lagging behind the competition.

I think my perfect editing tool would have the layering and tools of Lightzone, integrated into the Lightroom workflow. Now that would be a powerful product. No, I don't think using Lightzone as an external editor from Lightroom fixes that. It would be nice to have a develop panel in Lightroom for Lightzone, running it as an engine underneath, controlled from the same xml file format that Lightroom uses. That would get me to the core differentiator in Lightzone (the adjustments engine) without having to suffer the niggles.

Lightroom tip: selective masking

Ears up, Tanzania, January 2009

Lightroom 2.2, Windows XP
Read my tips intro if you're new.

Why is the brush tool so slow?

I've been doing quite a bit of brush work with Lightroom recently, especially playing with selective desaturation. I was getting frustrated by system slowdown and hang-ups so I went off to find out what's going on. what follows are some of the details, skip down to the Advice section for the what-to-do part.

In general, for small curves, limited regions and editing only a few images with the brush in a session, Lightroom 2.2 is very responsive. If, however, I start running large areas, many images in sessions or lots of changes, problems jump up.

How does Lightroom create the mask?

This is the central part of the problem, I think. When a mask is created, each stroke of the brush is saved as a separate multi-point curve. you can check this by saving the metadata to and .xmp file and opening it. the Mask has the form:

[crs:PaintBasedCorrections]
[rdf:Seq]
[rdf:li rdf:parseType="Resource"]
[crs:What]Correction][/crs:What]
[crs:CorrectionAmount]1.000000[/crs:CorrectionAmount]
[crs:CorrectionActive]true[/crs:CorrectionActive]
[crs:LocalExposure]0.000000[/crs:LocalExposure]
[crs:LocalSaturation]0.600000[/crs:LocalSaturation]
[crs:LocalContrast]0.000000[/crs:LocalContrast]
[crs:LocalClarity]0.200000[/crs:LocalClarity]
[crs:LocalSharpness]0.000000[/crs:LocalSharpness]
[crs:LocalBrightness]0.000000[/crs:LocalBrightness]
[crs:LocalToningHue]356.000000[/crs:LocalToningHue]
[crs:LocalToningSaturation]0.000000[/crs:LocalToningSaturation]
[crs:CorrectionMasks]
[rdf:Seq]
[rdf:li rdf:parseType="Resource"]
[crs:WhatMask/Paint][/crs:What]
[crs:MaskValue]0.900000][/crs:MaskValue]
[crs:Radius]0.023269[/crs:Radius]
[crs:Flow]1.000000[/crs:Flow]
[crs:CenterWeight]0.000000[/crs:CenterWeight]
[crs:Dabs]
[rdf:Seq]
[rdf:li]d 0.321572 0.469314[/rdf:li]
[rdf:li]d 0.326572 0.462008[/rdf:li]
[/rdf:Seq]
[/crs:Dabs]
[/rdf:li]
[rdf:li rdf:parseType="Resource"]
[crs:WhatMask/Paint][/crs:What]
[crs:MaskValue]0.900000[/crs:MaskValue]
[crs:Radius]0.023269[/crs:Radius]
[crs:Flow]1.000000[/crs:Flow]
[crs:CenterWeigh]t0.000000[/crs:CenterWeight]
[crs:Dabs]
[rdf:Seq]
[rdf:li]M 0.891740 0.296029[/rdf:li]
[rdf:li]M 0.894016 0.305266[/rdf:li]
[/rdf:Seq]
[/crs:Dabs]
[/rdf:li]
[/rdf:Seq]
[/crs:CorrectionMasks]
[/rdf:li]
[/rdf:Seq]
[/crs:PaintBasedCorrections]

the bit in bold shows two sets of points for two strokes but for a single region. Here I've replaced all the xmp <> with [] so it doesn't get screwed up by Blogger.

If you create a large mask (as I did for the whole leopard in the shot above) then you can end up with many lines, each with thousands of points. If you really want to get anal, these can be edited down with a text editor. Easily half the points can be removed (alternate points). tedious, and not recommended.

The more you edit, the more points are generated. If you have to go back over sections, you end up with lots of overlapping lines, and repeated work being done. I think, by experiment, that Lightroom is set up to interpret each line independently, which then goes to defining the area for applying the adjustments for a give brush mask. If you've got thousands of points, that's going to take time. So I suggest using some care, and good painting technique to improve things.

Advice

So here is my advice, particularly for large areas, in this order:

1. Use the mask overlay to show the mask as you paint (key O)
2. Start with a large brush and smooth continuous strokes for large areas.
3. Use the lowest flow you can get away with (cuts the number of points generated). Faster brush work needs a higher flow.
3. Avoid reworking areas
4. Zoom in and use a smaller brush for details and edges (generally with a lower flow, and slower brush strokes).

That should help cut down the number of points generated, and the number of line segments, which should maximise performance.

Scribus and POD output

I've just received my first Blurb book, formatted in Scribus and output in jpeg for inclusion in Blurb's Booksmart software. The book is a 7" square soft back, premium paper, printed in the Netherlands.

I came across an issue that is worth mentioning for others doing POD books.

My process was this: create all the layout in Scribus to correct sizes, export all pages (including text) as high quality jpegs, import these jpegs into Booksmart using full-bleed image layout. This way I get all my layout & fonts without having to worry about Blooksmart options. Simple and hopefully effective.

What happened? I received the book pretty quickly. First inspection is that this is a quality product: it's nicely bound, the paper has a good weight to it and the black and white is truly neutral. So far, so good. But closer inspection revealed very poor printing - jaggies and aliasing on all the diagonals and details. Ugh!

I trawled the Blurb forums for evidence of similar problem, nothing doing. However, before racing off to complain at Blurb I decided to double check the input files for a comparison. It turns out this problem is entirely user generated (i.e. my fault).

The problem is this: the output jpegs from Scribus are of pretty low quality. Despite setting high quality compression and 300dpi, the base jpegs look like 100dpi (or lower) versions. Not good. I wondered what had happened, so I did a few trials.

It turns out that when exporting pages from Scribus as jpegs, it uses the preview quality as the basis for generating the output, not the base image quality. I had been using medium preview quality for program speed throughout the process of generating the layout. Thus, the jpegs that were output had very low resolution quality. On switching to full quality previews, the jpeg output is as it should be. This is entirely different behaviour to the pdf output, which always uses original image quality for output.

Needless to say, I'll be reordering but I'm pretty confident results will be very good this time around.

Lightroom tip: advanced custom curves

English Wine & Beer Shop, India, August 2008

Lightroom 2.0, Windows XP
Read my tips intro if you're new.

How do I create my own curve profiles?

I previously posted on setting a Lightroom preset to do a basic tone curve inversion. You'll need to understand that, before reading on. I then wanted to go further, introducing my own custom curves, especially for my 35mm black and white scans.

I always scan my black and white as a positive and then invert in software. For a good explanation why you'd ever want to do this, read the post Matt Alofs wrote on the subject. When a negative is scanned as positive it usually compresses all the tones into the lower half of the spectrum, so on inversion it all ends up at the top. For me, the original positive scan rarely goes to more than 128 on a 255 scale (of course I'm scanning to 16 bit output files, not 8 bit).

After inversion, the first thing I want to do is apply a levels adjustment to bring the shadows back down into the lower tones. That meant a trip to Photoshop after importing into Lightroom to get a levels adjustment. I can, in fact, combine the 2 into one Lightroom preset. Here's how. The key is to set the "black point" of the inversion to a different point, close to the maximum scan value. So, instead of my black point origin being at 255,0, I want it at 130,0 so that the maximum scan value of 128 is just above the black point (so I can tweak later).

Here's what the preset file looks like with the change.

s = {
id = "498D92B2-EB89-4054-AAA2-46D3207AB652",
internalName = "Tone test",
title = "Invert 2",
type = "Develop",
value = {
settings = {
ConvertToGrayscale = false,
EnableGrayscaleMix = true,
ParametricDarks = 0,
ParametricHighlightSplit = 75,
ParametricHighlights = 0,
ParametricLights = 0,
ParametricMidtoneSplit = 50,
ParametricShadowSplit = 25,
ParametricShadows = 0,
ToneCurve = {
0,
255,
130,
0,
},
ToneCurveName = "Linear",
},
uuid = "8422BC3B-D8AA-4981-9F69-C350D56D935E",
},
version = 0,
}

I can now import into Lightroom applying the preset to the entire batch and get pretty good results from the off. This minimises the amount of work I need to do in other software. Of course, results may vary with film, scanner and exposure technique, so you might need to tweak the values for you own set-up.

If you're not good at reading negatives (and thus scan everything to check the best ones) this should speed things up considerably for you.

But there's more. What if I want a custom conversion curve to cover standard contrast or gamma adjustments? Well that can also be done (for positive or negative), by including extra points into the ToneCurve section of the preference. Here's a (rather stupid) example:

s = { id = "498D92B2-EB89-4054-AAA2-46D3207AB652", internalName = "Tone test", title = "Curve weird", type = "Develop", value = { settings = { ConvertToGrayscale = false, EnableGrayscaleMix = true, ParametricDarks = 0, ParametricHighlightSplit = 0, ParametricHighlights = 0, ParametricLights = 0, ParametricMidtoneSplit = 0, ParametricShadowSplit = 0, ParametricShadows = 0, ToneCurve = { 0, 0, 32, 130, 65, 100, 97, 60, 255, 255, }, ToneCurveName = "Linear", }, uuid = "8422BC3B-D8AA-4981-9F69-C350D56D935E", }, version = 0, }

The same shot as above with the strange curve applied

In fact, Lightroom curves can take any number of arbitrary x,y points to define the curve, you just have to open a text editor to get at the function.

So now, if you can translate multiple curve effects into a single curve they can be applied in Lightroom. A bit fiddly, but possible. Once the technique is known, the possibilities are endless.

Rather begs the question, if this ability exists in the software, why not expose it in the GUI?

Lightroom tip: invert tone curve

Lightroom 2.0, Windows XP.
read my tips intro first.

How do I invert the tone curve?

I've seen a few utilities available online and a lot of questions on how to, but nowhere that showed you how. So here it is. I'm assuming you know how to create a development preset.

First you need to know a bit more about Lightroom presets. They work as formatted text files, which makes them very editable. In Windows XP they're stored at: C:\Documents and Settings\%username%\Application Data\Adobe\Lightroom regardless of software install directory (for example, I've got Lightroom installed on the G:\ drive). For this you'll need to get into the \Develop Presets\User Presets sub-folder.

But first, it's easiest to create a blank tone curve and save as preset, with a name you can remember (easier than editing from scratch). If, for eample, you created a preset called invert, go to the prest directory and open invert.lrtemplate. The text should look like this:

s = {
id = "498D92B2-EB89-4054-AAA2-46D3207AB652",
internalName = "Tone test",
title = "Invert",
type = "Develop",
value = {
settings = {
ConvertToGrayscale = false,
EnableGrayscaleMix = true,
ParametricDarks = 0,
ParametricHighlightSplit = 75,
ParametricHighlights = 0,
ParametricLights = 0,
ParametricMidtoneSplit = 50,
ParametricShadowSplit = 25,
ParametricShadows = 0,
ToneCurve = {
0,
255,
250,
0,
},
ToneCurveName = "Linear",
},
uuid = "8422BC3B-D8AA-4981-9F69-C350D56D935E",
},
version = 0,
}

Note the line title, which is what shows up in the presets list in Lightroom. The important bit here, however, is the bit labelled ToneCurve. This lists the end points for the curve. We're going to reverse them from {0,0:255,255} to {0,255:255,0} thus:

ToneCurve = {
0,
255,
255,
0,
},

There it is, a basic straight line inversion curve for Adobe Lightroom. I actually do something a bit different, changing the black point instead, thus:

ToneCurve = {
0,
255,
250,
0,
},

I could also change the white point by setting the first pair to 0,250.

I'm going to come on to a fancier use of this in another post.

Lightroom tips: an introduction

Now that I've been using Adobe Lightroom for a while, I realised that there were quite a few things that I've needed to work out how to do. Apart from learning the new tools and what all the buttons do, there are a few features that seem to be either missing or undocumented.

I come at most software from the angle of asking myself "How do I do...?" and then trying to find out how to do it. I tend not to learn all the features and then work out how to put them into practice.

From this standpoint, I thought it would be useful to others to put together some posts with tips on things I've worked out how to do. These are often questions asked out in web-land without satisfactory answers. Certainly the things I'm going to write about I've not been able to find documented anywhere, including in books.

Each of the posts will start with the basic "how to" question I was asking myself and maybe reveal a few other aspects along the way. Each post will also refer to the Lightroom version and operating system platform. The latter is really related to file locations etc. If you read the posts and know the specifics for other OS, drop a comment which will make the posts more complete.

Back-up for the forgetful

That's me, Kristiansund, June 2008

I'm atrocious at keeping routines over extended periods. I often drift off doing something else and can easily forget to do something, even if it's something I do daily. Backing up my files is no exception. I've been 6 months or more without doing anything and then need a whole weekend to catch up. No longer. Now I've got the NAS box, I've been looking to make things more automated, especially so that I can get back-ups running in idle time due to my slow network. The back-up software I use is AllwaysSync (free for "moderate use" - after about 400GB of back-up it locked me out and asked to pay for a copy, $30 well spent). I'd had it configured for timed back-up, which can be a problem with power management, and also auto-on-connect for my external drives, which was a problem if I connected the wrong device (back-up being based on drive letter assignment). Then I found the magic setting.

Now I'm set for auto-on-idle. The software monitors the folders set in the auto jobs, when new stuff is written and the folders become inactive, it runs the back-up in the background. As I'm backing up over a network, I don't see any hit in normal operation. If I was working on the files to be backed-up, AllwaysSync would just wait for me to finish. In the week I've had this setting active, I've not missed a sync and not had to intervene. Perfect for the forgetful/lazy man.

Black and white shootout: Lightroom and Lightzone

At Checkpoint Charlie, Berlin, October 2008
from Lumix LX3 RAW file converted in Lightzone

As I mentioned at the end of this post, I've been using Lightzone as my main black and white conversion software for sometime. It produces really pleasing results with lots of control and minimum effort. This is where the underlying linear colourspace and the ease of the Zonemapper come into their own.

With the funky new tools, I thought I'd take a crack at using Lightroom in a sort of shoot-off. This won't be a long post because, quite frankly, there was no contest. Lightroom is very clunky and fiddly to use in comparison. Sure, it's got all the fine tuning possibilities in every part of the spectrum and masking etc. But with Lightzone I get that same power, with far fewer clicks. Plus I can stack tonal adjustment layers in Lightzone, which isn't possible in Lightroom (limited to a single curve tool).

Even for tweaking monochrome scans, I find Lightzone better, largely because I'm so tuned into the Zonemapper, but in these cases it's a closer call. I'm using Photoshop for scans mostly, anyway.

So for black and white, Lightzone wields the single stone that slays the Lightroom Goliath

Black and white film: a workflow

Zoo Station, Berlin, October 2008
Delta 400 @ EI800

Now I've been shooting black and white film for a while, I think I've got a smooth workflow down. I thought some might be interested in me sharing. Starting at exposure, I'll run through the process to print.

I tend to under-expose all my black and white film, for various reasons. HP5+ @ 800 (as much as 1600 in very low light), FP4+ at anything from 125 to 200. Delta 400 @ 800 indoors. Key reason for me doing this comes from the development and scanning routine.

Development is all commercial, through my local camera store (who send it off to Fuji's Pro lab). I don't know what chemicals they use, but at least the results are consistent. No pushing or pulling, just straight-up. I've no mind whatsoever to start my own darkroom work.

Scanning is done in bulk on the entire roll. As I'm using an Epson V750 flatbed, this is more efficient in my time. Always on a 2-pass scan, which yields a slightly finer grain output. Always scanned as positive with software inversion.

I've got a development profile for this in Lightroom now, which speeds things up tremendously. I also do rotation and initial cropping in Lightroom (used to be in Lightzone). Inverted images exported to 16 bit TIFFs as a batch. I only invert the ones I'm going to work up further - this is the first screening point.

Final adjustments depend slightly on image and subject. I've got a couple of profiles for Lightroom for quick conversions, especially to web. For anything decent, it's off to Lightzone for a couple of layers of Zonemapping or Photoshop.
Typically the adjustments are for final white and black point setting, shadow depth (my inversion routine means the shadows start open and I burn them in) and mid-tone contrast. Sometimes I run a light noise reduction with Neat Image, normally to tone-down graininess in areas of uniform tone, like expansive skies or large shadow blocks. Hiraloam and output sharpening to close.

I'm now printing through Lightroom because I really like the way their print manager works, plus it means I don't need a specific print version of the file, that I find I always need with other software.

Lumix LX3 and SilkyPix

Lumix LX3 test image, Kuala Lumpur, September 2008 f/2.8, ISO800
I used this for testing as it shows a lot of tricky image properties: highlights, shallow DoF, fine patterns in background, deep shadows

Previous posts on the Lumix LX3 itself here and here.

I said a while back that I'd write some thoughts on using Panasonic's SilkyPix Developer software together with my Lumix LX3, so here it is. It takes quite a lot of work to put these technical posts together, hence the delay. I'm using SilkyPix Developer Studio 3.0 SE: the bundled version that came with my LX3.

I was prompted to finally post this by the review of the LX3 at Luminous Landscape. I agree with pretty much all MR has to say, except 2 things: high ISO noise and use of the SilkyPix software. I'll come onto why later but first a definition (which may seem like a rant).

When I'm evaluating software, I have a clear idea and definition of what constitutes "intuitive". I do a lot of software evaluation, it's part of my day job. For me, if I cannot get useful output from new-to-me software in 2h without a manual, it's not intuitive. It may take you more or less time, you may want a manual or help files but this is my personal measure. Just because I wouldn't consider it intuitive isn't necessarily a Bad Thing but it is not a good sign in general consumer products. A couple of examples: Google is high on the intuitive scale - blindingly obvious what it does and how to use it, if you've ever seen a computer you'll be off with it. Adobe Photoshop, on the other hand, is as close to non-intuitive as possible; verging on the unusable. It's the only software in a long time (many years) I've needed instruction on how to even get going.
Just because a new piece of software doesn't work like similar products doesn't make in unintuitive, just as several products that work the same with similar initerfaces are not necessarily intuitive - copying bad practices just makes your product bad, not easy to use. So: intuitive software - easy to use without instruction, on the basis that you've never used something with similar intent. Bear that in mind for the rest of this.

Main editing tools	Hover to preview: "Natural" is selected, "noise reduction priority" is previewed

That clear, this brings up one of my objections to Michael Reichmann's comments: that SilkyPix is unintuitive (he's not the only one to state this). For photo software I think it is as intuitive as can be. I've not yet resorted to the Help function or manual. Once you start it up, it is fairly clear how to load photos and then the basic workflow is presented in the main tools along the left edge. Each slider is fairly self evident, and the preset options are pretty descriptive. Hover the mouse over any preset to get the image to show as a preview before selecting, click to make the change. This is probably the easiest RAW development software I've used, SilkyPix's great strength but also its weakness - if you're used to other RAW developers, this is another new interface. It's very much not like Photoshop (very good thing) which means long-term PSers may not like it. I'd say to them: get over it. I will concede that not providing a readily supported RAW format is a big problem for this camera (and many others).

Top menu options: layout options highlighted

Along the top, one finds the various view settings. I use the multi-view or single-view. Interestingly, one has a full range of adjustment available on selected images in multi-view which is unlike any other photo software with this multi-image browser interface. This makes it even easier to use with no need to switch modes to make adjustments.

Sharpening (left) and noise reduction (right) are 2 separate but linked tools

2 quirks: first is that output of the final image is termed "develop" and you need to use the Development menu option. The second is that the Sharpen and Noise Reduction functions work on a split button, which took a little while to deduce. This is actually a good thing once figured out - Panasonic are inherently recognising the trade-of between detail enhancement and noise reduction, evidenced by the presets. Of course you can try going for heavy sharpening and heavy NR at the same time but don't expect great things.

Advanced tools

A couple of less obvious tools are the ones presented bottom left. I've not investigated them all but 2 handy ones are the curves tool for tonal/contrast enhancements and the aberrations tool. The latter is pretty good with a wide range of distortion and fringing adjusters. thing is, I've not yet found any issues with files from the LX3. I've seen talk on the web that SilkyPix is automatically adjusting the RAW files for aberrations, which would make this tool superfluous but I'm not convinced. On using the camera, the screen shows no distortion and more do the out-of-camera JPEGs. If there is software adjustment, it is (also) in the camera, which would be impressive enough. Certainly, switching off the aberration tool in SilkyPix has no effect.

How does SilkyPix perform on LX3 files? Very well. I've found 3 ways to use it. First to batch several RAW files with no adjustments to full 16 bit TIFF files for further work in other software. Second for fine-tuning individual images (not used often) and the third is to create 16 bit TIFFs fom the in-camera JPEGs for further processing. I'll come onto this last in a minute. This leads me to the other objection to Reichmann's comments: noise in the images. Yes, this is not a super low noise, high-end DSLR. However, don't praise the RAW feature for the ability to do further processing and then condemn to straight from camera results. At ISO800 and 1600 the files are very good after noise reduction. I consider RAW files in light of how I process them, and that includes noise reduction. I use Neat Image as a Photoshop plug-in and get excellent results. Noise is sufficiently consistent in LX3 RAW files (after TIFF conversion) that I've been able to create a standard setting in Neat Image, which means just a couple of button clicks to clean up the files. Shadow & pattern detail is still good and smooth tones clean up nicely. I reckon you could enlarge the results quite a bit, although I'd be happier at A4 (8x12") prints. Well beyond my expectations for a small sensor camera, as I've written before.

File open is a jpg (EXIF information first line), "Save as" coverts to 16 bit

Now to the last point for this post: JPEG to TIFF conversion. I'm not quite sure how SilkyPix works underneath but it is able to convert 8 bit JPEGs to 16 bit TIFFs. dcraw has a similar function. I've tried a couple of tests and this works very well. The resulting TIFF file is much more suitable for further adjustment than a JPEG, although not as good as a RAW file. I might even use this feature for converting other JPEGs to allow me to do more work on them.

3 output options (L to R): TIFF from RAW (with Neat Image NR & sharpening); JPEG from camera; JPEG from camera converted to TIFF The 2 TIFF files have had WB correction, the JPEG is straight from camera.

So there it is: Panasonic's SilkyPix Developer software and the Lumix LX3 provide a good combination for getting really good results out of the camera. I'm not claiming the RAW development is best in class, but the workflow is simple

A question for Lightroom users

Is there anyone out there who can help me?

I'm looking at Adobe Lightroom for a number of reasons, one being batch processing. I can't seem to find the info I want, however.

What I want to be able to do, in a script or similar is the following batch process:
1. Standard set of editting tools (brightness/contrast/basic curve)
2. Resize
3. Change colour space
4. Save from 16-bit TIFF to 8-bit JPEG
5. Rename & refile the batch by rules

Would I need plug-ins or prgramming to make it happen?

This is all based around my post to the web workflow. Thing is I can't seem to determine if I can combine these actions together. I see evidence of being able to do each step individually, but that doesn't really help me.

As I say, this is just one component of the reasons that I'm looking at Lightroom but is a big one.

Colour managed browsing is here

Picked from Rob Galbraith, the release of Firefox 3.0 with colour management support. Read the article over there, which has tips on how to get it to work. 've not installed it yet but it'll be high on the list of things to do when I get back from my current travels.

A couple of weeks ago I went to the Firefox/Mozilla pages looking for news on the colour management - not a sniff of a hint of information, yet it's the biggest improvement to my mind.

For those of you still using IE (why, oh why) - get with a proper browser. There's no excuse. (If you're stuck with IE 'cos you're at work - stop slacking off!)

Digital long exposures

Running water, Cumbria, April 2008
After RAW conversion

I have been pretty dissatisfied with long exposures using the digital camera. Anything past about 10-15s leads to horrid noise or detail loss with black frame removal. Plus a stack of filters seems to degrade the image more than film (or maybe that's just me).

Anyway, while processing the above image, I tried something different. I'd taken about 6 exposures of the scene with exposures varying from 1/3 to 1s. I was trying to get a smooth flow to the stream of water on the right. I only had a 2 stop ND filter, which didn't help. Looking through all the exposures I was considering pasting together parts from each to get the effect.

I then turned to Photomatix Pro to blend all 6. Not HDR, though. I just used the average function and some slight tone mapping. What a huge difference, seen here:

Running water, Cumbria, April 2008
After the full Photomatix & post process treatment

Total exposure time for the 6 images was about 5s. The overall result though is of a much longer exposure. There is some great effect from the smaller flows amongst the rocks (100% crop below). Plus noise is gone and detail is great, something Photomatix is really good at.

Crop from final image, click for actual pixels view

So there it is: take a whole lot of relatively short exposures and average the results. Output resembles a longer exposure than the sum of the parts.

I've not tried this yet with some really short exposures, but that would be fun. Say 10 or more at action freezing speeds (1/125 and faster).

Oh, and before you all chip in, I realise there were a bunch of film based techniques similar to this but I've not come across something similar in the digital world.

Advice on pdf photobooks for the web

Viewpoint, Kristiansund, April 2008
Another that didn't fit in the book

Now that SoFoBoMo is over and I've rad all the books and then some, I thought I'd offer up some commentary on the technical aspects of putting together a pdf book for the web. Having done it myself, this is advice coming from both the production and consumption ends of this.

I noticed that a lot of the books posted were huge (30MB and up) and quite a lot were difficult to read and view due to the layout, hence this post. I'm not a professional designer and don't know swathes about book design but these are some simple things that I think would improve the look of the books.

My thoughts:

1. Put cover pages on the book. It looks more professional if there is even a simple pair of covers. Makes it look like a finished product. There were a few on getting to the end I wondered where the rest was.
2. While we're on covers, put your name on it. As an author, stand up and be recognised. There were a couple with no credit on them at all. Use your real name.
3. Image size matters. Unless you are using full-bleed images (right to the edge) please put more border around images. If pictures goes too close to the edge they feel cramped. Same with text & captions. Quite a few would be losing their images in trim if actually printed. As a minimum I reckon 10% of the page width as a border each side.
4. Just because it is aimed for on-screen show, put it together like a print book. The ones that look like they are printed actually read better on-screen. This means things like larger inside margins, no spreading images across the fold, proper leader pages etc.
5. Pick fonts I can read. There are a few where the typography really gets in the way of readability. Think about size, letter spacing etc. Try a few options and run up a trial page of text.
6. Please, not slideshows nor fade effects. This works for a series of images only but we are talking about a book here. Again, it's better if the on-screen book feels like a real print book.
7. Smaller, please, please, please. The key to file size for on-screen display is less resolution, not greater compression. jpeg 25% compression gets you nothing compared to 100dpi output. If you are printing, then create a second version for the web. Output images at 90-100dpi, medium compression. For a SoFoBoMo size book (35-50 images) this should get you down around 5-6MB. Most layout software will allow you to output at different resolutions, use the options.
8. Embed the fonts and colour profile (sRGB for on screen). Ensures it looks nice and as you intended. pdf readers take care of the rest.

Good news from Adobe

So Adobe announced recently their submission of DNG to ISO for acceptance as a RAW format standard. Great news.

I'd been concerned (and am generally concerned) about the fact that a corporation was trying to push as a unified standard their own developement. Thus we become beholden to them and their whims. Thus are the beginnings of evil empires. (OK, maybe a little dramatic.)
The biggest cocern I have in this area is that you become relentlessly tied into software upgrades and therefore huge expense.

I hope ISO get on with this quite quickly and incorporate enough stricture that manufacturers can't tie up the bulk of their information as proprietary stuff. I don't want it to go the way of TIFF which is really a general wrapper for a whole bunch of underlying image formats. This could be the way forward - software, and software version, independent RAW formats. Hopefully with a proper industry standard, more camera manufacturers and software developers are kept in the game.