This does look very promising. How was your spectrometer set up - blue dye removed from DVD-R grating ? (assuming you used a DVD-R), IR filter removed from lens? Any camera settings ? (auto WB turned off etc.)

I suspect that the IR response could be much better if both the blue dye is removed from the DVD-R and the IR filter is removed from the lens. I have seen some plots for the Bayer filters used in web-cams that show the red filter not as band pass, but as low pass. If the red filter really was a good band pass it would seem you would not need an extra IR filter, so I suspect the red filter often is a low pass. I hope to get around to doing some tests on IR response (with and without blue dye and IR filter) using an IR LED as a light source.

The blue filter does seem to be band pass and responsible for cutting off the short wavelengths.

The sample of the Sanmtech camera I got with my PL Spec 2 kit seems to be particularly bad, I don't know if they are all like that. It's noisy and it seems impossible to get the R,G and B to track at any colour temperature, which means there is always a gain imbalance between the channels. The Videw XJD camera supplied with Spec 3 seems to be a better camera, although I haven't had a chance to run any camera tests yet. If there is a lot of variation in camera performance it could perhaps make applying any corrections more difficult.

I've just looked at the Solux lamps. As far as I can see they are halogen lamps with built in blue filters. What a great find.

I've tried daylight incandescent lamps before, but this was a while ago. The lamps I tried were ordinary tungsten bulbs with a blue filter (I have been meaning to try producing some plots for them with the PL spec.) I hadn't realised you could get daylight corrected lamps in halogen. I'll have to get some.

BTW you can also get lighting filters for daylight correction of tungsten filament lamps farily cheaply from Lee filters. Although I guess the Solux with the filtering built in are more convenient. I wonder how much different variation in the supply voltage makes to the lamp spectrum. Lab reference lamps do have regulated power supplies.

This was the Sanm camera of the PLab 3.0 + my own upgrade. No blue-film on the DVD and no IR filter. While removing the IR filter from webcams does add some IR detection, the sensitivity is still poor -- so attempting to gain-correct beyond 650nm is not very effective. The noise is too high so the correction error is too large. I think it is not yet possible to analyze the variation between Bayer filters between Sanm cameras. I suspect that any variation is not with the camera but with the measurement configuration. More test with Sanm camera and more Solex 4700K lamps could build more statistics; then we'd have more data to assign an error value.

Yes, Solux apparently wanted to capture the fine art display market so needed much better halogen lights with broad / smooth spectra. I've not yet found better and none others with published spectral data. Only their 4700K lamp has such a flat spectra which is why it is ideal for this specific use - broadband gain correction. These are 12V lamps and come with a power supply. I've received some stability data, which looks good, but I"m waiting on clarification on the spectral precision -- I''l add it to the note when it's available. Filters don't help. While they can change the overall shape, they do nothing for the variations within a band -- just adds another layer of calibrated values to acquire.

Sigh! well I wondered why I hadn't come across 'daylight' halogen before. It's cos they don't seem to be available in the UK, at least the Solux lamps aren't. The majority of daylight lamps available here are old style tungsten bulbs with a blue coating. The makers don't supply any kind of typical curve and they don't look as though they would be very consistent. There is an Osram 'daylight' halogen, but its a lower colour temperature and the Osram curve for it is an 'artists rendition' and highly idealised. They don't give tabulated measurements, just the curve.

As for the specialist lamp versus filter thing. I was under the impression that the Solux lamps were just ordinary halogen / filament lamps with a built in filter. The output of a tungsten filament lamp, I would have thought, would be fairly smooth. I have seen it said that tungsten lamps are a reasonable approximation to a black body. Never-the-less I take your point that the Solux lamp, which is manufactured to a known performance spec. with published data, is the best choice.

With a tungsten lamp and a Lee filter you'd be trying to build your own daylight lamp and would really need to calibrate it with a commercial spectrometer of known calibration.

Yes, CIR, CCT and other lamp descriptions make knowing what you really get sometimes rather difficult. I have just added a section about calculating there uncertainties for this form of gain-correction and, with the 12V Solux, they look pretty good. As of suppliers, yes, it looks like primarily US: Amazon, Pegasus Lighting, Lamps Direct, etc. Also note that their other color temperature lamps do not have the same smooth spectral curve; the 4700K lamp is unique in this way.

There's been a good parallel discussion of the need for this here: https://groups.google.com/d/msg/plots-spectrometry/ZtLXALzz4Us/F_nXP5sc6TgJ

I proposed that we start thinking (maybe in steps, or even pseudocode) about how to design a macro to accomplish this as a core function of Spectral Workbench.

Dave has very helpfully created a Github issue for it too: https://github.com/publiclab/spectral-workbench/issues/107

What would it look like, step by step?

Whatever the reference source, that spectral response curve needs to be saved in SWB. Also, as in another SWB issue I filed, SWB data should be translated to uniform X-axes increments -- nm increments or 0.5nm increments and aligned to integer nm values. This will be the only way to keep plot comparisons and gain-adjust curves easily aligned. We don't need precision as exists now in the CSV files (like 436.435nm) as the spectrometer is, at very best, only ~2nm FWHM resolution. So, .....

Next, any form of gain-adjust reference should be a "smoothed" curve with curve detail below the resolution of the spectrometer. Any curve 'transients' in the reference (even if they are real and known) will only appear in the adjusted output as noise and so are meaningless to the spectrometer's data.

1 - Normalize the smoothed reference data (peak value = 1.0 relative intensity)

2 - Smooth the PLab's measurement data of that same reference source (eg. 15-point)

3 - Normalize the smoothed PLab measurement data (meas peak = ref peak = 1.0)

4 - Calculated the ratio (multiplicative not subtractive) difference over 400-650nm

5 - Use a cosine function to 'roll-off' the ratio curve to zero below 400 and above 650nm

6 - Smooth (eg 5-point) the result and save as a user's relative intensity gain-adjust cal

7 - User can then use this curve just like their CFL cal data

When SWB displays a spectra which makes use of a gain-adjust curve, it should overlay some graphical indicator of the 400-650nm gain adjust range so that the user can recognize that their data outside of this range should be treated as increasingly unreliable in terms of intensity relative to the data within 400-650nm. Remember, the existence of spectral information outside 400-650nm can be real, it is just increasingly less reliable because the spectrometer's sensitivity falls off so rapidly.

Hmm... I'm wondering what the right graphical indicator for lower dynamic range at the outer edges of the spectrum should look like.

How about just a partial 'fade-out' of the graphics plot range below 400nm and above 650nm (darkened / subdued colors) and then activate a text note in the UL of the plot area like 'Gain Cal Range'. The data, above and below, is still visible but it's obvious that data is, at least, less important (...less accurate, totally uncalibrated).

This project might be of interest for finding a solution for the calibration problem:

https://jethomson.wordpress.com/spectrometer-articles/how-to-build-a-spectroscope/

It describes an amplitude calibration called "Determining the System Function" that corrects the effect of the RGB filter of the camera sensor (similar approach as suggested by Dave Stoft) plus a subsequent radiometric calibration using a separate light sensor. This allows quantitative measurements of spectral irradiance. All code for the calibrations is also available. Perhaps it is possible to add an economic light sensor to the DIY kit to get independent light measurements e.g. this one: http://www.yoctopuce.com/EN/products/usb-environmental-sensors/yocto-light-v3.

While it continues to be an evolving prototype, the new Subtraction tools in Spectral Workbench 2.0 (preview) allow for subtraction of one spectrum from another. I'm curious if you might use this to attempt the correction you outline here. I noted a proposed process for a similar subtraction on the mailing lists a couple days ago:

https://groups.google.com/forum/#!topic/plots-spectrometry/OAXxG40dSyo

• upload an image with two side-by-side cuvettes (hopefully you'll be able to distinguish the two spectra)
• calibrate it (using a calibration from the same setup)
• use the v2.0 interface (see the button along the bottom of the graph to switch into 2.0 preview)
• clone the spectrum so you have two copies
• use "set cross section" on each to choose your two different cross-sections of the spectrum, one for each cuvette, pressing "Apply" to accept the new cross-section
• use the "Subtract" tool to subtract one from the other

I'm VERY willing to help out in this process, as it's very much prototype code, and may still be buggy. Eventually, I'd like to use Sreyanth's Procedures code to help guide ppl through this process and automate some parts. Please, if someone wants to copy this procedure into a research note and try it out, and post results, that would be GREAT.

OK, while we're tackling some of the discussion of gain correction in https://publiclab.org/notes/stoft/02-25-2015/plab-spectrometer-gain-correction#c13258, I wanted to confirm that this is the right product here, Dave:

http://www.amazon.com/18003-Daylight-Halogen-Degree-Kelvin/dp/B0002GS4OG/

How important is the power source for consistency? Something like this? http://www.amazon.com/LET-60-Class-Electronic-Transformer-Lightech/dp/B002RSOULS/

One other question; is there a way to reproduce this method with an edison screw-type bulb, to simplify/cheapen the setup? I guess I'm not finding much... http://www.bulbs.com/Halogen_Bulbs/Medium_(E26)-Base/results.aspx Just thinking of how to make this process easier for folks. On the other hand, if we can test a bunch of the webcams against one another, a standard correction for a webcam type is a good start, anyways.

I used the 35W Solux 4700K. eg: http://www.amazon.com/35003-Daylight-Halogen-Degree-Kelvin/dp/B0002GS4GE/ref=sr_1_2?s=hi&ie=UTF8&qid=1452026828&sr=1-2&keywords=solux+mr16
These halogen lamps get really hot and there is no need for high power.

I could not find data on the spectral variance with source voltage, but since filament bulb intensity is a function of supply voltage, I believe using 12V is preferable. Solux specifies that their specs are based on a 3-5 min "warm-up" period for the bulb to obtain full stability. Since, at best, the calibration accuracy will be no better than about 10%, I see using a stable supply as a necessary part of a calibration.

I bought an Anchorn #AET 60W Class 2 12V supply for Halogen lamps -- but probably most qualified halogen 12V supplies would work.

Yes, because of the ~10% cal-error limit for the process (based on using a Solux lamp) I have suspected that individual calibration of webcams would likely prove unnecessary. I still suspect the webcam sensitivity variation between devices will prove to be much less than 10%. The test could be simple; buy one 12V 35V Solux + power supply and just measure 10 webcams using the exact same idealized setup. Then compare the resulting curve variation (after removing any intensity offset variations due to setup for each). The resulting curve error range should likely be very small. If so, just build-in the correction to SWB and make it transparent. Then, user's results could be specified with a theoretical %error over the 400-650 cal range.

Posting the correction file and Solux published spectrum that Dave sent me here:

Solux4700KGCal_300-800nm_SANM.csv - Dave's correction curve from MatLab data

Solux4700kSpectrum_280-800nm.txt - Dave's extraction of Solux data from website

I'm just checking on this, but I believe we've changed the camera shipping in the desktop kit, which is one reason I'm interested in the process you used to generate the correction file, so it can be reproduced for any given camera. Just to re-state your 7 step process above in pseudocode:

Here I've skipped the "roll-off", although I agree we should do that. I just want to separate out the steps. Once this part is figured out, I have some questions about the roll-off. How does the above look?

Sorry, fixed the formatting.

Oops, one more bugfix to the pseudocode -- now I think it's what I intended.

Hi, Dave - Mathew confirmed that the Desktop Spectrometry Kit v3.0 has always shipped with the JDEPC-OV04 webcam, not the SANM webcam. The SANM was used only in the v2.5 kit. If you have a completely standard kit, your data (and this whole post) should be for the JDEPC-OV04.

It looks like this (click to see full resolution):

Ok, if you look back up above in THIS research note, you will see an Update I have added which includes a single combined plot set using my PLab-3 proto with the new camera and shows the 4700K Solux Gain-Correction curves. The curve data is also included as a set of .TXT files. I've included the 400-650nm correction; although the errors "blow-up" at either end of the webcam's wavelength sensitivity response curve. To use the gain-correction curve data (which is listed for 300-800nm in 1nm increments) just multiply: corrected_val = int16( double(PLab-3_val) * GainCorrection ) for every wavelength of interest. The result can be re-scaled if necessary. I used an arbitrary 530nm as the point to set the gain-correction to 1.0.

That's super, thanks. I think we're pretty close here, but I just wanted to get explicit confirmation that the process I outlined is accurate for how you derived the correction -- mainly in the interest of reproducibility. Did the code I posted make sense, and does it accurately describe your process?

Thanks, Dave -- i'm also VERY close to posting the new Snapshots and Operations systems, which I hope to very soon follow up on with this correction code.

Jeff, pretty close. I'll try again, in case I'm not reading your code accurately....

1 - Solux4700K_spec([nm(300:800),val(300:800)]) = the Mfg's specified spectral data for 300-800nm in 1nm increments

2 - Solux4700K_meas([nm(300:800),val(300:800)]) = the PLab V3 measured spectral data for 300-800nm in 1nm increments where the nm was obtained via a 2-point CFL cal using 435.833nm (blue) and 546.074nm (2nd Green peak)

3 - 530nm is arbitrarilly (selected based on the GRN response curve "flat region" when observing the Solux measurement) set as the "gain-correction=1.0 reference point.

4 - Yes, I'm assuming all raw data has been averaged and processed so as to be working in 300-800nm range with integer 1nm increments

5 - Given the Spec and Meas data sets, we then scale the Spec'd data to make the 530nm data points match such that the gain correction at 530nm will be ~1.0. I believe this is a useful step because it keep the final GCal correction curve data in a sensible value range so that applying the correction to new data will not result in huge mutiplicative shits in spectral intensity values.

6 - Calculate the "raw" gain correction curve where: GCal = SoluxSpec / SoluxMeasured. I believe this is what your code calculates.

7 - However, even at 400nm and 650nm (the two ends of the corrected range which are roughly the limits of using the correction w/o suffering too much error because the webcam's sensitivity is near zip. So, I apply a cosine-based rounding algorithm to both ends of the GCal(300:800) data from step 6. For example, the 'raw' GCal data can easily produce a magnification of ~2.5 or above at just 650nm due to low webcam sensitivity -- which would produce rather high error values so not be very honest.

Make sense?

Hi, Dave and @ygstcu - I have a version of the gain correction code loaded into a local copy of my code, and am testing it out. Dave, can you link me to an un-corrected spectrum on Spectral Workbench which I can run the code on, and then we can compare the output of my correction code to the corrected data you generated? If they match, then we're very close.

Jeff, I was going to provide you with a fresh plot but unfortunately SWB still fails on Firefox -- same "left half only" spectrum with no visible data to match what shows in the image -- so, SWB remains unusable. I'll try an update and see if that helps. Otherwise, I could probably do a download of an uncorrected spectrum you wanted to use and convert that to run locally.

Hi, Dave - I left some requests on your issue to try to reproduce the error you're seeing; did you see that? https://github.com/publiclab/spectral-workbench/issues/285

Ok, just took a look, replied no, still broken. I cleared cache with no effect. I filed another bug report as I cannot even use the switch to a 'new calibration' -- which just brings up the error page (The page you were looking for doesn't exist). Individual bugs like this are a pain if they are not easy to reproduce.

Hi, Dave, thanks. I responded and split out a new issue with the calibration bug and added a couple questions.

Ok, I got a fresh copy of Chrome to work so here's a Solux 4700K spectrum which was callibrated with a 'CFL2700K' new cal data. https://spectralworkbench.org/spectrums/71409 However, the data download is also broken as there is no wavelength data (all zeros) in the CSV format and 'null' in the XML format -- unless some other new feature must be run besides the 'calibrate' function to get that wavelength data to be included.

Thanks, Dave - I believe exports were fixed in https://github.com/publiclab/spectral-workbench/issues/322, which was only published just now with the changes related to your issue. They work now and I confirmed manually.

Hi, Dave - I believe I was able to apply the correction by running the following line in the new Scripting tab:

This used your non-rolled-off correction file. I'm not sure yet how I want to integrate this into the Operations system, but I was able to download a CSV output using this and the JavaScript console:

The output was:

spectralwb-71409.csv

And it looked like this:

The CSV data files have poorly formatted data. The text of the files often looks like this: 206.809,2.55,2.5500000000000003,7.6499999999999995,2.5500000000000003 . Also, since the wavelength precision of the measurements is only 1nm and the accuracy is less, the wavelengths after CFL cal should all be nice integer values with uniform 1nm increments. I'd thought this had been done already -- if not, it should be updated.

Just extracted the Slux4700K original data w/rgb and your gain-corr curve data from above, scaled the curves to eliminate that factor and plotted. Doesn't appear to have had any effect....... the black is the original Solux spectra and the magenta is your processed data.

Sorry, back online after a long weekend. Looks like there was just an error in my code -- i'm going to try and tackle some bugfixes today but after that i'll give it another go. Thanks for checking -- and sorry it's a bit of an onerous round-trip to actually test this out. I can both visually inspect the output vs. your graph now, and also compare it to a fork of itself without the correction.

Understood; though independent checks and tests are still the best method to find hidden anomalies and overlooked issues. When you fix the glitch, I'd expect the ends of the corrected plot to show dramatic error expansion as the camera sensitivity goes to near zero -- i.e. this should be really obvious in the plot.

Also, regarding the unit precision, I've opened an issue at https://github.com/publiclab/spectral-workbench/issues/335 and would love your input on managing precision there.

Hi, Dave - i noticed that https://spectralworkbench.org/spectrums/71409 has a linearCalibration operation - I think by mistake, as it's not a CFL. I removed it (i'm an admin, so i'm able to) but can add it back in if it was intentional?

Also, I was just wondering, your black line in your most recent image does not seem to be an average of the channels, and does not match the data I see before attempting to apply the correction:

I'm working on getting the correction working now, but just wanted to check on this so that I see a clear before/after state that matches yours. Thanks!

Jeff, thanks for pointing this out. I admit I do find the SWB terminology a bit overloaded and the user interface 'unexpected' as I'm not used to it. I'd thought that the Solux plot had used the CFL-2700k cal as a reference but I must have noticed the x-axis pixel values and hit 'calibrate', intending to tell SWB to 'calibrate' the spectra using the last CFL reference I'd specified in the interface when I took the data -- as opposed to 'use this plot to calibrate the spectrometer'. It does seem that SWB can't tell the Solux plot is not a CFL. As for the level, yes, it seems off but I was rushed (a bad thing) and guessed SWB was just scaling the data for display.

OK, a bunch of solved problems!

• figured out the height difference -- needs to be resolved, but I believe due to a bug, we were multiplying each pixel by the final entry in your correction file -- 2.667. In any case it was just scaling the whole graph by 2.667, so my just-published fix resolves that.
• It's hard to get SWB to "know" that something is a CFL, but I'm going to try to add some more notices if, for example, operations are in an order that doesn't make sense -- like a linearCalibration after a calibrate operation. The tools are very new to everyone, so i'm sure it'll be helpful.

Now, on to the success! I believe I got this to work; see screenshot and check out the output data:

Great! Yes, little glitches do creep in but yes, this Solux corrected plot has a much better shape. I'm guessing the slight curve variation between 500 and 650, plus the peaking at about 650nm are similar to what I saw, even though the ideal Solux curve is more uniform. So, I take this as good confirmation of the approach. It also looks like you've handled any error accumulation at the ends to the algorithm is well behaved. Without individual, calibrated-Solux calibrations there probably a little unresolved error, but I'd think this is a good first-order improvement on system gain correction.