about the artist
The Problem with RAW: related links
For those who don't have memberships to Adobe's User to User forum, I am quoting the information from those links below. Remember that these are snippets from ongoing conversations:
Jeff Schewe, photographer and digital evangelist, describes EXIF specs, and how camera makers store data in their files (From Adobe's User-to-User forum.):
To understand better, the EXIF spec, as defined by JEITA (see link at bottom) provides guidelines for a variety of digital camera metadata. There is a class considered "public" where the description and the manner of defining the data is standardized.
There are also what are called "private maker notes" where the camera makers can put "private" data. There are only certain definitions for private tags, camera makers can put in pretty much anything, including encrypted metadata such as what Nikon has done with the WB data.
Image width
ImageWidth
Image height
ImageLength
Number of bits per component
BitsPerSample
Pixel composition
PhotometricInterpretation
Orientation of image
Orientation
Number of components
SamplesPerPixel
WhitePoint
Camera Make
Camera Model
ColorSpace
DateTimeOriginal
ExposureTime
FNumber
ISOSpeedRatings
ShutterSpeedValue
ApertureValue
FocalLengthIn35mmFilm
Are examples of public EXIF fields. There are even clearly defined fields for GPS & Audio.
What is not considered "public" can be put in "private" fields. And while there are some standards regarding this, the fact is that the private tags are what has caused a lot of problems relating to all the various undocumented raw file formats.
Added to this problems is that the EXIF spec has been slow to be updated and there are not enough public fields for some of the newer cameras. When there is no public field the camera maker is forced to put the metadata in private fields even when those fields SHOULD be public because of the usefulness of the tags. One example is Camera Serial Number, it's not a public fields so many cameras either don't record the serial number or do but in a private tag.
The private maker notes has and will continue to cause problems in undocumented and proprietary raw files. Since the camera companies don't document how and where the metadata is written in files, nor what the data means, 3rd party developers must "decode" or reverse engineer what the data means. This it time consuming work with no documentation. Even if you HAVE the camera company's SDK, there may be no specific documentation for the private maker notes. Some SDKs expose some data, but not all.
What Dave Coffin, Thomas Knoll and Eric Bibble have had to do to decode the data and figure out where it's written and what it means and how to use it is increadably hard, tedious work. I personally watched Thomas Knoll work to decode white balance metadata on a Canon 1D MII camera that Canon had sent him. The process is very time consuming and to me, boring as heck.
However, the process of adding encryption to make it even harder work is clearly an attempt to try to deny access to the information. Eric and Dave have both broken the Nikon encryption and Coffin has posted the decryption keys on his page. However, while decoding or reverse engineering is an entirely legitimate and legal method of decoding file data structures, the act of encrypting that data raises the stakes for potential legal problems for those who decrypt it. Dave Coffin and Eric Hyman have exposed themselve to considerable risk by decrypting the Nikon WB metadata. I sincerely hope they will face no problems for having done so.
That is why it's _SO_ critical for people to understand what is at stake. Contrary to what people may think, I am not anti-Nikon. I am anti-ANYTHING that I see as a threat to photographers and the photographic industry.
And while some may think that the WB encryption is unimportant, it does cross a real line.
Where Nikon had NOT encrypted any data, it now has.
The fact it's only one relatively unimportant private maker note does not matter. They COULD have chosen to encrypt the whole thing but didn't have to because by encrypting even one small seemingly unimportant WB tag, they have actually had the effect of putting the entire D2X NEF under the protection of that one tag. If you want to support the entire file and all maker notes-something a 3rd party software would prefer to do-you would have to break the WB encryption.
And that is the tip of the spear. Nikon has NOT said they WOULDN'T sue. . .and that is the same as implying they may.
That is a very chilling effect for 3rd parties. And why I appose it so strongly.
Author and Photoshop guru Bruce Fraser describes Adobe's DNG file format in detail, continued here and then here. (From Adobe's User-to-User forum.)
"DNG contains *exactly* the same pixel data as the proprietary raw.
The latest version offers the option to embed the entire original in the DNG, but DNG has always, from day 1, contained all the pixels. DNG doesn't dictate the use of any particular demosaicing algorithms, it simply wraps the pixels and whatever metadata that actually meets published specs in a standard format. That standard format is an extension of TIFF 6.0 and is compatible with the TIFF-EP standard, but it no more resembles the kind of TIFFs you save out of Photoshop than does the compressed bitstream used by fax machines (which is also part of the TIFF spec).
You need a camera-specific algorithm to convert a proprietary raw file to DNG. You don't need a camera-specific algorithm to convert the DNG to a color image. Converting to DNG in no way compromises the full range of the camera (I'd actually argue that ACR is just about the only raw converter that DOES give you access to the full range of the camera), because conversion to DNG does absolutely nothing to the raw pixel data.
Now, there's a second form of DNG—linear DNG—that allows you to store a demosaiced but otherwise unprocessed image. That's the kind of DNG that DxO, for example, writes. It's not a raw file anymore though it's not fully cooked either. I suspect that the existence of this flavor of DNG is causing some confusion.
Saying that DNG is TIFF and some extra tags is about as helpful as saying that the space shuttle is a Buick Riviera with some extra machinery. "
Second post:
-->And as you said and my understanding "DNG contains *exactly* the same pixel data as the proprietary raw." so what convertion?
The conversion from a proprietary raw file that contains large chunks of mystery metadata to a DNG with all tags and offsets documented. It's like the conversion of a .PSD to a TIFF—it's the same pixels, but one is in an open, documented format where the other is in a proprietary partialy-documented one.
-->It might be possible but that's just the lowest common denominator approach.
No. I repeat, you can use any demosaicing algorithms you want in a DNG. DNG is just a documented way of storing the data. It doesn't impose any limits on the algorithms, it just presents the data in a consistent format. If anything, it makes it easier to implement new algorithms because you don't have to spend time figuring out where things are stored in the file or try to decrypt undocumented private tags. (Camera vendors can still build secret sauce that only makes sense to their converters in a DNG, the difference is that in a DNG, the secret sauce area is clearly marked "here be dragons," and you know where it starts and ends, so you can ignore it instead of having to first reverse-engineer it so that you can ignore it.)
Third post:
You need to demosaic to see a color image from a raw file. There are many different ways to do it, but if you want colored pixels, some form of demosaicing has to happen.
Opinions vary as to whether or not there's any value in the secret sauce. Camera Raw ignores anything it can't understand, as do all other raw converters. DNG Converter, unless you ask it to embed the original raw, strips anything it can't understand, and even if you embed the original raw, the mystery metadata isn't readable by DNG-compliant converters. It's simply available for extraction if you want at some point to get the original raw back and run it through one piece of software that does understand the mystery metadata, which is the proprietary converter. DNG Converter doesn't even try to get the meaning of the mystery metadata, and DNG-compliant raw converters don't use it unless the originator of the secret sauce gives them the recipe.
I vastly prefer Camera Raw's renderings of my Canon raws to those of Canon's DPP. So for me, the secret sauce is totally worthless. Canon holds a different opinion, otherwise they wouldn't bother putting it in there in the first place. They're entitled to their opinion, but I think they're delusional.
The things I can do with DNG that I can't do with CR2.
Run it through any raw converter that understands DNG and compare results.
Write metadata directly into the DNG rather than use sidecar files or external databases that can get divorced from the image file.
Have reasonable confidence that my raw files will be readable long afer Canon has ceased to support them.
These are current benefits that I'm getting today. In future, I can look forward to much wider support for raw images from asset managers and browses so that i can see the real image rather than some crappy camera-generated thumbnail, maybe even built right into the OS. That's not going to happen for proprietary raw formats, at least not for all of them, and I can pretty much guarantee that at least one of the vendors you cited as never supporting DNG will go out of business in the next 10 years, so their file formats will be orphaned.
I already have one orphan camera. There is no software from Kodak that will let me open my DCS 460 files that runs on an OS made in this millenium. If it weren't for Thomas Knoll's work, I'd still be trying to get my Daystar clone to boot OS 8.6 every time I wanted to access a DCS 460 raw. The DCS 460 secret sauce came in the form of a .cal file—Kodak's software refused to work unless it was present. Camera Raw doesn't need it, and produces much, much better results from those files than anything that Kodak ever made.
So while it's true that you can do anything with a DNG that you can do with a proprietary raw, the reverse is not true. Hence, they aren't the same.