Public Lab Research note


PLab Spectrometer Gain-Correction

by stoft | | 2,874 views | 17 comments |

Read more: stable.publiclab.org/n/11638


stoft was awarded the Basic Barnstar by warren for their work in this research note.


Calibrating the PLab spectrometer for absolute intensity will likely remain a difficult and expensive concept. However, I'd like to propose a simpler method of at least improving the relative accuracy of the spectra users can display.

Web-Cam Limits A Compact Florescent Lamp (CFL) provides an inexpensive source for calibrating the PLab spectrometers for wavelength because the CFL has several stable emission lines at known frequencies. However, calibrating the spectrometer for relative intensity over that same range is harder because there are no readily available light sources with uniform output energy over the full visible range of wavelengths. In addition, the sensitivity of digital camera (like the Sanm web-cam) vary substantially with the color of light. Consider this first graphic:

GainCal-1.gif

The measurement data came from an prototype upgraded PLab 3 spectrometer (http://publiclab.org/notes/stoft/03-02-2015/plab-3-spectrometer-upgrade-prototype) and the solar reference data came from the web. The plot show the familiar RGB and averaged-RGB plots that are viewable in SpectralWorkBench (SWB). Notice that the 'Magenta' measured data (from observing the sun) looks only vaguely like the reference spectral curve of 5780K sunlight. Why?

Well, notice the three RGB measured curves. The Blue filtered curve not only cuts off sharply just below 400nm but it is nowhere near "flat" within its 400-500nm span. Similarly, the Green and Red curves 'roll-off' rapidly at either end of their filter ranges and their center-peak gains are different. Combining the curves together does construct a broader range, but the sun still emits significant energy well past the camera's 400nm cut-off and the gain between is non-uniform. Above 650nm the Red filter does pass some near infrared but the signal is weak and rather noisy. So, there are some inherent limits but also notice that the solar spectrum, though not flat, is relatively 'smooth' (as opposed to a CFL for instance) and does extend past either end of the Sanm camera's sensitivity range. So, why not take advantage of the sun's spectrum to improve the accuracy of PLab spectral plots?

Solar Reference Below is a graphic with plots showing the process of gain-correcting the PLab data to match the reference solar spectrum:

GainCal-2.gif

Again, the Black is the sun's spectra, the Magenta is the averaged-RGB PLab device measured spectra from the sun and the Red is the gain-corrected measured solar spectrum. Note that this gain-corrected spectra looks quite similar to the reference plot -- although it does have more noise. The Green plot is the gain-correction curve which represents the multiplier values to be applied to the device data to correct for the fact that the sensitivity (or gain) of each of the RGB channels is not "flat" -- that each channel's data rolls-off rapidly at each end of its pass-band and that simply combining them does not fix this issue.

[Side Note: The gain-correction curve is a product of several processing steps including averaging (because correcting noise just produces noise) and synthetic "repair" of the noisy spectral ends where there is little to no signal. When signal levels become buried in the noise, the potential for the gain value to mathematically 'blow up' is very real. So, below 385nm and above 715nm, I induced a cosine 'roll-off' to zero. There's no useful data at 300nm or 800nm.]

Sanity Check As a sanity check, I took this same correction curve and applied it to the CFL data which was originally used to calibrate the measured sun spectra:

GainCal-3.gif

As you can see, the gain-correction curve is well-behaved in that background noise (including the ends of the spectrum) is not over-amplified and the relative peak sizes look appropriately similar to those published on the web. In addition, the peak shapes and regions with more CFL noise (570-600nm) look well behaved.

Final Thoughts Because the production of digital camera silicon devices and products is such a tightly controlled process, it may be possible to create a "standardized" gain-correction curve for each camera type or model and thereby provide a first-order correction within SWB itself. Individual gain-calibrations could always be performed but evidence suggests that noise levels alone will mean the gain-correction curve must remain smoothed approximations.


17 Comments

This is awesome, Dave! The noise above 700 and below 400nm is substantial, but I think that just defines the usable range of the spectrometer.

@warren, what do you think about adding a solar spectral correction into SWB? is that possible as a secondary calibration?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Thanks. Yes, the camera has virtually no relative sensitivity outside 400-700nm. This does need to be differentiated from observing a bright source outside that range where some signal may be observable -- i.e. measuring attenuation of a fluid using a bright 350nm source might be possible. The point to take from this gain-correction is that without a truely 'flat' light source, 400-700nm is really the limit for corrected spectra.

Reply to this comment...


my experience has been similar-- just because a very bright source in the 800-1000nm range can be detected doesn't mean that we can trust the sensitivity of a fluorescence measurement in that range.

Reply to this comment...


I'm wondering if you took any account of the web camera settings - in particular colour temperature / white balance. This has quite a large effect on the gains of R,G and B. And the sunlight reference curve looks like it is of relatively low resolution compared to the PL spec output. The PL Spec plot may not have noise so much as actual signal.

Reply to this comment...


Thanks; interesting thoughts. Yes, digital cameras can have white balance settings which create a general shift in the RGB output ratios, but the SWB interface does not have that sophistication. SWB therefore has the same interface settings for all Sanm cameras -- whatever that actual color temp might be -- the specs on the Sanm device are sketchy about that. The sunlight reference curve is "smooth" compared with professional-grade plots which show more detail -- but that "detail" results from spectral "absorption holes" which are too narrow for simple digital camera spectrometers to detect. The webcam is, in fact, quite noisy where the sensitivity is low (like the ends of the detectable bandwidth) so that data really is just noise, not real signal data. There is no way to create a gain-correction based on noise so it is better to use reference data with similar detection bandwidth to the hardware being corrected. An incandescent light bulb also has a relatively "smooth" spectrum but it's hard to find light bulbs where you have the actual spectral curve of it's specific temperature. There's lots more very accurate data for the sun.

Reply to this comment...


adding a solar spectral correction into SWB

YES! I'm finishing up some MapKnitter work next week, and debugging today and tomorrow, but would love to use this approach in a re-imagining of the new calibration procedure.

Reply to this comment...


I'm looking for help using a solar spectrometer to gain current incident light values as a base to adjust my reflected image taken in a high quality digital image. I am doing aerial imagery and understand that often the incident light values change. I want to ratio the different intensity values to get a consistent value.

Reply to this comment...


Might be helpful to post the topic in the PLab Spectrometer Google groups where you can get a wide range of feedback. Briefly, I'm not clearly following your intent but it seems you want to measure / reference the solar spectrum so as to remove that variable from the image measurements. I'm guessing you are wanting to color-balance your aerial photos taken in sunlight which can be difficult depending on the degree of accuracy. In theory, if you could measure the general solar spectrum curve (as it relates to the deg-K change during the day as measured on the ground you are photographing, then you might apply an idealized deg-K shift to the image. An alternative might be to white-balance the camera on the ground and take a photo of a color reference card prior to launching the camera. Neither method take into account atmospherics introduced by the altitude of the camera.

Reply to this comment...


Hi, Dave - I'm circling back because I believe the new subtract: feature in Spectral Workbench 2.0 could enable you to do a standard gain correction (I'm hearing 'standard deduction' in my head, which is funny, but kind of similar -- i.e. it wouldn't be super accurate, but would be better than nothing, right?). But for now, the easiest way to try it out would be to upload a curve (as an image) which could be used as the standard. Is there any way you could upload a standard correction (for the sanm camera, for example, or even better for the camera that ships with the 3.0 Desktop Spectrometer) which we could use to test that out? We could try it vs. some lab specs to see how well the correction works.

Also, can you link to your source for the 5780K spectrum? Thanks a lot, Dave!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Jeff - Yes, it may be possible (with some modifications) to use your new feature code for a 'default' gain correction. Also recall my later research note dealing more specifically with using the Solux 4700K lamp for calibration: https://publiclab.org/notes/stoft/03-06-2015/plab-3-gain-correction

However, remember that correcting the camera sensitivity (gain) vs spectral wavelength is a multiplicative process, not a subtractive process as is used when comparing two measurements with a device where the gain remains the same. To correct the camera sensitivity, the raw camera data is multiplied by the Solux correction data and then scaled for plotting. Also remember that although the correction data spans 300-800nm, the actual usable range of corrected data from the camera is only 400-650nm because the camera sensitivity errors dramatically increase at each end where the camera becomes totally insensitive. But, perhaps you could re-use most of your code for processing curve data and just add implementation for correction by adding buried (non-visible) correction curve data and re-scaling (in the case of corrected spectral data exceeding 100%).

The Solux spectral data can be found here: http://solux.net/ies_files/SoLux%20Spectral%20Data.xls.pdf You will note that only the 4700K lamp has an emission curve which is sufficiently smooth and broadband enough for the spectrometer gain connection.

I can email you the Solux and PLab correction data; it is not data directly suitable for display in SWB.

Reply to this comment...


Ah yes, re: multiplicative. Luckily there's an as-yet hidden feature called blend: which is like subtract, but which allows any arithmetic combination of spectra; it runs a math expression on every pixel (kind of like on Infragram.org). So multiplication shouldn't be hard. The new range:min-max operation can limit our work to the necessary range, too.

Is it possible to upload the correction data here? What format is it in? If it's not possible to convert it to an image, I might be able to help do that, either manually in Javascript, or potentially we could reformat it as JSON or CSV data which we could build an importer for in Spectral Workbench. This is exciting!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


The correction data is in a CVS file and the Solux data is in a TXT list file -- both with 2 columns (nm,value). The data should remain in this (nm,value) pair format and not be converted to an image. There is no need to plot the correction data as the 'values' are not % units like the SWB spectral plots -- the data can just be kept in a background data file which is internally read when needed. The Solux data values are all relative linear intensity with no units and so would be similarly used as background data for individual lamp calibrations at some point. I'll e-mail the files.

Reply to this comment...


I guess I'm thinking of how we might use different correction files for different lamps, and how people might upload and share new correction files for new lamps. Using the Spectrum database storage and infrastructure for that seems reasonable, and it lets us use the new Operations system to apply the correction. Does that make sense? Maybe we can flag them as corrections data in metadata so they aren't confused with a real spectrum.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Sure; they are just different file types (content). Initially, for testing, there could be just one "default" selection with buried data as a proof-of-concept and which allows the user to validate that any other cal file is producing at least reasonable results. It's a stepped process: a) unspecified error limits (now), b) ~10% error (Solux), c) lower error (other)? .... with the final error budget yet to be determined.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


1 - Solux4700KGCal_300-800nm_SANM.csv (My cal curve from my MatLab data)

2 - Solux4700KSpectrum_280-800.txt (My extraction of Solux data from web site)

I got the two files:

Solux4700KGCal_300-800nm_SANM.csv

Solux4700kSpectrum_280-800nm.txt

So the .txt file is their reported spectrum from their website -- shall we use another absolute intensity calibrated spectrometer (when we can) to confirm this?

And just to be sure, do I have this right? Each data point of the correction .csv is the Solux reported lamp spectrum divided by the Public Lab v3 Desktop Spectrometer scan of a Solux lamp?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Yes, the Solux4700KSpectrum file contains Solux's 4700K lamp output spectrum, after a 3-5min warm-up period +/-200K on the center color temperature. In my related research note, I show that this +/-200K uncertainty equates to <10% intensity error in the curve data. So, any re-measurement of a Solux lamp would have to be made with a spectrometer that could significantly exceed this accuracy spec.

Yes, but not quite. The correction data is roughly 1x at 500nm so you would just multiply measured webcam spectrum data by the correction data to get the result. (With some spectra it might be necessary to re-scale due to the ~2x correction out around 600nm) However, because the webcam's sensitivity drops off rapidly below 400 and above 650, while the Solux lamp's spectra is broader, the webcam correction error becomes excessive (and also unrealistic because the noise exceeds the signal). To prevent the application of the .CSV correction data from creating horrible (and noisy) plots at each end, the calculated correction data has been arbitrarily "rolled off" using a cosine function. Doing so, results in a nice response plot for CFL spectra which then doesn't "explode" with noise at the ends. It also means it is best to indicate the 400 - 650nm valid region for the resulting plot. (The webcam data's valid range for intensity is really this limited anyway even though you might see some noise and signals beyond those limits.) These effects are all explained in the other note: https://publiclab.org/notes/stoft/03-06-2015/plab-3-gain-correction

Reply to this comment...


Sorry, indeed I should've left my comment on the other post. Moving over there, thanks.

Reply to this comment...


Login to comment.