Question: Why do raw RGB values differ from calibrated RGB values? I thought calibration just set pixels to wavelength (nm)?

troyb is asking a question about spectral-workbench
Follow this topic

by troyb | June 25, 2018 08:50 | #16568


Here are the first lines of spectra 137547.

Calibrated:

| Average| Red| Green| Blue| | 12.7| 7.65| 10.2| 22.95| | 12.7| 7.65| 10.2| 22.95| | 12.7| 7.65| 10.2| 22.95| | 12.7| 7.65| 10.2| 22.95| | 12.7| 7.65| 10.2| 22.95|

| | | | |--|--|--|

Uncalibrated:

| Average| Red| Green| Blue| | 15| 10| 12| 23| | 15| 10| 12| 23| | 15| 10| 12| 25| | 14| 8| 11| 24| | 14| 8| 11| 24|

The small differences could matter for my daughter's project, measuring a greenness index (green minus blue) to estimate whether algae are growing, especially after we had to recalibrate.



17 Comments

Could not find the actual spectra from the info provided. How about an explicit link or post the graphic?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Ah, the table makes the values hard to see. Note for instance that a raw value of 10 scales to 7.65 while a raw values of 23, 24 and 25 scales to 22.95. This happens with all my spectra it seems. That particular spectra is at: https://spectralworkbench.org/spectrums/137547 (and this format can be used to get to any saved spectra).

Reply to this comment...


Also, I didn't save any spectra with saturated channels that I recall, but the recent spectrum https://spectralworkbench.org/spectrums/137643 illustrates that the RGB channels retain the expected integer value only when saturated (value = 255).

Reply to this comment...


Ok, visualizing the spectra helps. 1 - The calibration (wavelength calibration using a CFL source) does not require an amplitude change (though SWB is likely doing this). 2 - Using a Calibration result doesn't alter a measurement spectra's amplitude -- it only corrects the relationship between camera pixels and wavelength (nm). 3 - Once calibrated (wavelength), additional measurements can (should) use that Cal reference. 4 - Remember, the wavelength calibration is only valid if the mechanics of the spectrometer doesn't change (this is a mechanical issue where it is best to leave the spectrometer physical device mounted so it doesn't change for all measurements which are based on that calibration.

Reply to this comment...


Also, 5 - Use 'absolute' measurement values when mathematically comparing amplitude of spectra -- where the only change between plots is the experimental condition being measured.

Reply to this comment...


So, to clarify: what I'm saying is that there seem to mathematical artefacts in the calibrated intensity values. This means that at least for exported csv files, point 2 above is not quite right. The changes to raw intensity are small but I can't figure them out (tried to fish around for the code, but couldn't find that part). So I think this leads to the conclusion there is small bug with unknown consequences (output of code differs from intent), and it will be best remove the calibrations and use raw values when exporting and processing data for quantitative purposes?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


I'll admit I'm not following your comparisons -- but a few thoughts anyway in case they will help. The second plot at least shows a CFL spectra; although 1) some 'tuning' of the hardware might gain more resolution and 2) the RGB channels are clipping. Clipping adds significant error to the RGB-combined 'avg' result which will likely swamp most other errors. I'd suggest the following sequence: 1) mechanically stable setup, 2) CFL capture w/o clipping, 3) wavelength-cal using the CFL plot. Then... take reference spectra using broad-band light (i.e. through water for example) and then, take spectra with test sample. The wavelength cal's output data (which is applied to other plots as the cal reference) only uses the pixel/nm relationship, not any amplitude data from the CFL cal plot. The difference between the 'water' reference spectra (CFL-Cal'd) and the 'test sample' spectra (also CFL-Cal'd with the same CFL nm/pix data) will be your amplitude difference result and will not contain any amplitude factors of the CFL plot.

Reply to this comment...


On using only raw data (eg. using the CSV 'avg' data): Yes, you could implement a simple linear mapping of pixel:nm with the data from a CFL spectra. Then (and assuming the hardware remains unchanged) apply that mapping to all measurement spectra. Finally, assuming no clipping of RGB channel data, you could perform the difference (water reference vs test sample) -- all using the 'raw' data. The SWB 'calibrate' feature is a convenience; it's not mandatory to doing the spectra processing in another space like a spreadsheet.

Reply to this comment...


Our spectrometer is pretty amazing. Your comments are right on, I certainly recommend avoiding clipping (the warning in SpectralWorkbench works really well). The spectrometer's calibration was very sensitive to bumping – and stoft's explorations of harder builds look really useful. It was really useful to do the calibration, and then just take the wavelengths from the saved and exported calibration spectrum.

Reply to this comment...


Thanks very much for the prompt and numerous replies. I think they add up to an answer. There is a small difference in the intensity of the calibrated values, relative to raw uncalibrated values. I made the attached figure to show this. I gather from the replies from stoft that this isn't intended. The difference is small, but I think it may be largest at low values (less than 10 on the scale of 0 to 255).

I also included our experimental results in the attached figure as a second panel.

I'm pleased to say that I've now calculated results and the experiment worked very well. We measured algae with and without an alum treatment that removes phosphorous from the water. The alum makes a flocculent in the water, and our measurements (at 90° to a 20W cool white LED light source) distinguished the 'greenness' of algae (chlorophyll reflects green light, absorbs blue) from the higher reflectance of alum. I suggested we use a normalised difference index (green – blue)/(green + blue) after subtracting baseline (violet).

Rplot.png

Wow, doesn't it look like the calibrated values are losing precision?

One thing I think is happening where you see the .95 decimals is that it's an artifact of the averaging -- I can dig up that code and we can likely come up with a better way to do it. Hang on!

Is this a question? Click here to post it to the Questions page.


Ok, here is the averaging:

https://github.com/publiclab/spectral-workbench.js/blob/7b52a773650951ab3a2b54c6ad60f8f117a6d72d/src/SpectralWorkbench.Spectrum.js#L558

And here is where the image data is being extracted from the image file:

https://github.com/publiclab/spectral-workbench.js/blob/7b52a773650951ab3a2b54c6ad60f8f117a6d72d/src/SpectralWorkbench.Image.js#L95

What do you think? We can try to reproduce this in a JavaScript test to see how it's affecting the output and if we're losing precision past where we should.

Thanks for the careful look at this! Which spectrometer design are you using?

Is this a question? Click here to post it to the Questions page.


There are definitely decimal values that don't seem quite right. Thanks for pointing to the code. I'll try to have a look in the next few days.

Apologies in advance if I'm delayed ... It's well past bedtime here, and I've neglected some other semi-urgent stuff while having 'fun' with this.

Last answer: we got a Desktop Spectrometry Kit 3.0 in the post within the last month. That has the revised webcam, and I ended up putting the dvd grating right on the silver front of the webcam, and adjusting the angle slightly. We also made a bigger slit - about 3x wider - for more light, presumably at a slight loss of resolution.


Hi, the new Lego version does a lot to address rigidity issues, and your recent version of the DSK's webcam will fit in the new enclosure. If you're interested, you can find more at #lego and there's a version of the kit which is "just the legos" to upgrade an older kit!


Hi again. I only had limited time to look at the code. I'm not great with Javascript. Two things stood out. First, it is a little odd that the original RGB pixel values (integer 0-255) were overwritten. I couldn't immediately work out where that happens. Second, I did see that only 4 digits are retained and used for the parameters. I'd have guessed that would be enough, but it would be interesting to see if the problem goes away if you increase this to ~6. Plus, it's probably easy to try?

Yes, legos are a great idea. Fixing the camera to the wood block and it to the velcro has quite a bit of give too. I'm in New Zealand, so I probably skip the international postage and push some wood (or plastic floor tiles) through a table saw for my next try.

Is this a question? Click here to post it to the Questions page.


They're recalculated when you perform a calibration, but also they are averaged together and then limited to 4 to try to maintain the significant digits only. But I think that it's at that point that the rounding can go a bit awry...


Reply to this comment...


Hi, @troyb - we do have a new Lego enclosure which is a step closer to the kind of rigidity @stoft has demonstrated in wood -- we're actually phasing out the older enclosure in favor of this very robust Lego version, and we'd be happy to offer you an upgrade kit at a discount if you're interested?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Log in to comment