Spectralworkbench Blog


Automatic Spectrometer Calibration and Offline SWB - GSoC 2014 Proposal

by Sreyanth | over 10 years ago | 18 | 0

Name: Mora Sreyantha Chary, (fondly called as Sreyanth)

Affiliation (school/degree): National Institute of Technology Karnataka (Computer Engineering, 2014)

Location: Hyderabad, India (during summer). Time zone: UTC + 5:30

Email: sreyanth@gmail.com

Project I want to work on: Spectral Workbench

Proposed Project title: Automatic Spectrometer Calibration & Macros Module, Offline Version of SWB

Project description

Abstract:

The public laboratory’s Spectral Workbench provides its users with tools to share the spectra and work on it. I would like to add few extra yet important functionalities -- A mechanism enabling SWB to automatically calibrate the spectrometer(s) of the users and a Macros module . With these in place, the users will be able to make sure that their spectrometers are accurately calibrated and thus can be helpful while calibrating the newly uploaded spectrums. The Macros module helps the users to automate most of their manual routine work on SWB. This helps the users with high end computation requirements use the Macros module and let the SWB do the work for them. And the second part of this project is a popular demand from various people on the mailing lists -- an offline version of SWB. With the stand-alone Java version deprecated, a browser based offline version, preferably using the existing HTML and JS, and the LocalStorage of HTML5 to store the captures, and a mechanism to upload the captures when connected to the internet, is really necessary. With these two features, the SWB will be much more user friendly and effective too.

The need my project fulfills:

My project aims to automate the error-prone calibration of spectrometers, thus making the process effective. Users may not, sometimes, accurately calibrate the spectrometers. Or, in some cases, may not calibrate at all. To encourage them to calibrate the spectrometers, it will be ideal to auto-calibrate and ask the user just to confirm what (s)he sees on the screen. This will help in easy calibration, easy validation (as the user clicks the button after being satisfied by the calibration) and effective spectral analysis. The Macros module helps the users to write their own macros and plugin to the SWB. Also, an offline SWB is a popular request, which will let the users to use SWB even when there is no internet connection, entirely using their own web-browser without needing to install new software and configuring -- which has proven to be complex at times. My project will bring out a fully functional offline SWB.

How will my project meet this need:

My project will come up with the required mechanism for auto calibration. The steps involved are: 1. Check for CFL patterns. When a CFL pattern is observed (using the spectral matching module -- may be?), ask the user if (s)he would like to auto-calibrate. This can be extended to other spectral patterns like Neon after initial testing of the module’s performance on CFL. 2. Find the major known peaks and calculate the distances between them. Use the ratios of these distances to come up with a candidate calibration. 3. Color detection -- find the middle blue line and bright green line using various image processing techniques. Check for over exposure. If all the 3 channels are over exposed, either accept the candidate calibration, or ask the user to change the setup to make sure the exposure is reduced. Else, use the color information to generate another candidate calibration. Calculate a measure (for time being, let us consider it to be Hamming Distance), and see if the candidate calibration using colors is close enough to the one obtained using the ratios of distances method. If it is close enough, then the system is doing good, accept the candidate calibration from step 2. Else, check for over exposure in any of the channels. If over exposed, accept candidate from step 2. Else, accept it from step 3. 4. Ask the user to name the device -if (s)he wishes to, so that they can use this calibration for other spectrums captured with that particular device 5. Save the calibration, if the user is satisfied with the highlighted lines. Or else, manual calibration -- note that this manual calibration will now be added to our search space. Next time, our algorithms will perform better!! 6. For the time being, let’s consider the linear calibration first. Based on the results achieved, we can extend the system to use a nonlinear calibration model. Macro’s module: 1. Refactor the existing Macros module. Also refactor and add few functions in the SWB API so that easier integration and effective use of SWB is possible. How to go about implementing offline SWB? 1. Use most of the HTML and almost all the JS already written 2. Use the localStorage of HTML5 to store spectrums. 3. Even the calibration is stored. 4. An option to choose and upload all or few of the spectrums captured offline on to online SWB, when there is an internet connection available (achieved by pinging our server) 5. Create a module, to which these offline spectrums can be securely uploaded and processed to make sure this won’t result in a Denial of Service attack (using some sort of CSRF tags). 6. The above said module will be a two way one. One can upload the offline spectrums. At the same time, download the calibrated spectrums for offline use.

Timeline/milestones:

Working on it. I want the ideas to be refined, so that I will come up with the timeline appropriately.

What broader goal is my project working towards?

Effective spectral processing for everyone, everywhere!

What resources I will need:

I am a lone worker who loves to work independently without much guidance. I love learning and implementing. This is my weakness, and sometimes, I fall behind schedule. But I strongly believe learning is necessary for not just implementing, but required to implement it right. I would be needing help from the mentor and the community as well regarding the spectra analysis. I would also be needing a foldable from Public Lab so that I can experiment and test the modules I write instantly. Also any literature, which I can understand is much valuable. I don’t need a popular paper, but I need a right paper so that a CS grad like me can understand it. Also constant feedback and support is always appreciated.

Experience:

I was a GSoCer for Public Lab in 2013. I worked on finding closest match spectrum from the database. More about the project can be found here:

  1. http://publiclab.org/notes/Sreyanth/09-14-2013/finding-closest-match-spectra-from-the-database-gsoc-final-post

  2. http://publiclab.org/notes/Sreyanth/07-29-2013/finding-closest-match-spectra-from-the-database-gsoc-work-done-so-far

  3. http://publiclab.org/notes/Sreyanth/06-24-2013/find-closest-match-spectra-from-database-gsoc-project

Also, I have written my own code to do my course projects. I list them here for you (recent first):

  1. Discovering XML Injection Vulnerabilities in Web Applications -- a framework for vulnerability scanners for XML injections

  2. Twitminer (A classifier which classifies the tweets into ‘sports’ or ‘politics’): Written in Python. (used sci-kit learn for this)

  3. External Merge Sort Implementation for one billion integers (wanted to have some fun!). Written in C.

  4. Top Down Parser for a subset of C language. Written in C using Lex and Yacc.

  5. Multi-threaded chat server and chat application. Written for my socket programming lab in C.

  6. iAd- Integrated Ad Agency Management System. Written in JSP.

  7. Capture the Flag platform. Written in Python. Used Django, PHP and CGI.

  8. eAgromet – Advisory system for Agricultural Scientists. Written in JSP and Python.

  9. A Deterministic Cryptarithmetic solver. Written in Python.

  10. Online Driving Licensing System. Written using PHP.

In addition, my application, CulinarYou! was a finalist in Google Cloud Developer Challenge 2013

Teamwork

Yes, I have worked in a team for my course projects [6th and 10th in the above list]. Also, my last summer internship at the International Institute of Information Technology was also a team project [8th in the above list]. According to me, working in a team needs good communication. Without which the project will be a bit out of the road. Luckily, all my team mates used to discuss the ideas and implementation plan almost daily and we made it a huge success.

Expertise

Well, I consider myself good enough and well suited to handle this project. I am already well versed with the SWB codebase. I am comfortable with RoR and JS. I am good with C and Python too. Web tech, I am good with Django and Google App Engine too. With immense experience in developing web applications, I think I am well suited for this project. And more importantly, I love coding, optimizing, scaling and again looking it all over to make sure it meets MY standards. My standards are: Correct output, less time, neatly labeled variables, sufficient documentation.

Interest

Yes. I always had a thing for open science. In my humble opinion, science is not confined to just scholars. Everyone can learn, use, discover and invent it. Why should only scientists perform experiments? Aren’t the others really fit to even learn some basic phenomena in the nature? And yeah, I realize that public lab is a nice platform working towards this goal. I actually even want to recommend that public lab should start some initiative only for kids to develop the interest for science in them.

Audience

Non – technical users. The user need not know anything how this is implemented.

Context

The main thing that motivates to work on this is, my attachment to this project. I loved working on SWB last year, with great inputs from my mentor as well as the PLOTS community. I thought to switching the project, to Infragram for a while. But, I thought of making SWB much more effective and then jump on to Infragram -- the reason being, I believe I am going to be a developer for Public Lab in the future too. Continuing the same enthusiasm, I want to contribute to the same project. This project posed some interesting coding challenges to me, and am constantly motivated by the way my mentor looks at these issues. So, this project won't only strengthen my coding skills, it also enriches my analytical thinking. With this in mind, I am reapplying for the same project.

Ongoing involvement

By the time GSoC ends, we would have a fully functioning auto calibration system, and an offline SWB. I would like to be a part of PLOTS community ever after the end of GSoC, mainly because of the interesting issues and projects Public Lab looks into.

Commitment

Yes. I have no other commitments for this summer. I want to devote the entire time, approximately, some 40 hrs a week to my GSoC project.

Please note: This is still a draft and is going to significantly change in the upcoming days. Please feel free to give your valuable feedback by commenting on this post, or by emailing me (sreyanth@gmail.com) or Jeff (jywarren@gmail.com)

Read more Follow

spectrometer spectralworkbench matching gsoc

New beta capture interface for Spectral Workbench

by warren | almost 11 years ago | 5 | 3

I've been working in my spare time on a new interface for capturing spectra, pictured above, which is available at:

https://spectralworkbench.org/capture/beta

It uses the more flexible and robust Bootstrap interface widgets which I've been using on the rest of the site, and combines the mobile and desktop interfaces into a single flexible layout (which means that any feature we add to one will be available for both).

Features

Some of this is really just reconstructing the features of the old interface in the new one, but one very nice addition is that you can now click to choose the "sampling row" in the mobile interface:

Screen_Shot_2013-12-12_at_10.59.27_AM.png

You can also do comparisons and run macros from the beta mobile interface. This is all a bit untidy at present, but now that it's implemented, refining it and adding new features should be much easier.

Screen_Shot_2013-12-12_at_11.00.39_AM.png

The still-untidy macros pane on the new beta capture interface.

Questions and next steps

We need to think a bit more about the macro interface and how new macros and additional tools are laid out in this new screen. There's more space than before, but the last interface got cluttered pretty fast, so I'd like to have some overarching layout strategy to avoid that. Also, I'd like to integrate some kind of premade online IDE component in the macros pane, so you can actually prototype in there, as well as search for and run macros.

Anyhow, feedback and bug reports are welcome; what do you think about making this the default interface? Is it ready? Is it at least better than the old one?

Read more Follow

spectrometer spectralworkbench

Flare Spectrometer proposal in Shreveport and Meraux Louisiana

by AndyZ | about 11 years ago | 3 | 1

Shell_Norco_flare.jpg

12-12-12_Calumet_flare.jpg

  1. Proposed sites:
  2. Calumet Lubricants 8 (AI 1214) in Shreveport, LA
  3. Hosted by: Andy Zellinger with Louisiana Bucket brigade and Residents for Air Neutralization thanks to support from Public Lab
  4. Address: TBD
  5. Valero Refining (Ai 1238) in Meraux, LA
  6. Hosted by: Dan Beavers thanks to support from Public Lab

  7. Address: Dan Beavers has confirmed with a property owner - pictures

  8. Overall purpose - what we hope to gain, what the community can gain If emission spectra are detected, attempt to match them with reference data from NIST utilizing the Public Lab Spectral Workbench. We would also compare matched emission spectra to the self-reporting from the refineries themselves utilizing Louisiana Bucket Brigade’s Refinery Accident Database. If, for example, we detect lead, and the company reports lead, we have a great validation of our ability to detect that contaminant while it's happening. We could use this data to empower community based activists groups, and urge regulators such as U.S. EPA to set new emissions standards for frequently emitted elements.

  9. Design specifications of the spectrometer (Detection of pollutant types and limits, size of equipment, how it stores data, etc.) The size of a Public Lab spectrometer is approximately 6 inches long 2 inches wide and 3 inches tall. These spectrometers are available in unassembled kits for $40 US dollars. Developed by Public Lab contributors, this open source instrument is capable of better than 3 nanometer spectral resolution and can record light from ~390 to ~900nm. Though experimental, this tool has already been used to collect spectral data on contaminated water, laundry detergent, wines, and fishtank lights. We are really doing a proof of concept here -- our outcomes may be more about testing the device and its suitability for this purpose, and troubleshooting and improving the experimental setup of our tools, rather than (at this point, anyways) collecting persuasive data which we can take to regulators. That said, we should document what we do very clearly so that others can reproduce what we do. The more people watching these flares, the better!

  10. Placement criteria (inside, next to a window, within xx feet of flare, etc.) PublicLab has a window tripod if necessary. We could super glue a nut to the spectrometer to fit any regular tripod.

  11. Who will be responsible and how much time it will take one person for maintenance, trouble shooting, data download, and data analysis Andy Zellinger will be responsible for downloading Calumet/Shreveport data. Dan Beavers will be responsible for downloading Valero/Meraux data. Public Lab contributors will assist with analysis and troubleshooting.

  12. Materials: • WIFI cell hot spot - contract required for 1 year so funding for the whole year would be requested-$600 • RaspberryPI (2) • PIcam (2) • low resolution thermal imaging sensor and additionally necessary components(2) • 2 spectrometers (2 from Valero site, how many from Chalmette?) • USB powered hub (below items can be lumped together for about $150) • misc. cables • power supplies • misc. mounting hardware • web Storage

Read more Follow

gulf-coast spectrometer spectrum-matching spectralworkbench

Testing spectral workbench webcam recognition with different flavours of linux

by briandegger | about 11 years ago | 0 | 0

What I want to do

find out which flavours work :)

My attempt and results

eeepc 701 "gb memory running puppee NO

optiplex 745 running Kernel : Linux 3.8.0-19-generic (i686) Compiled : #29-Ubuntu SMP Wed Apr 17 18:19:42 UTC 2013 C Library : Unknown Default C Compiler : GNU C Compiler version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) Distribution : Ubuntu 13.04 Desktop Environment : LXDE (Lubuntu-Netbook) Browser: Chromium Version 28.0.1500.71 Ubuntu 13.04 (28.0.1500.71-0ubuntu1.13.04.1) YES

Questions and next steps

Read more Follow

spectralworkbench cross-platform first-time-poster

Firefox/Firefox-Android support for Spectral Workbench!

by warren | about 11 years ago | 0 | 0

As seen in the above screenshot, Spectral Workbench (finally) also works on Firefox and Firefox for Android.

Try it out: https://spectralworkbench.org/capture

What's nice is that for Android users, Firefox has implemented camera selection, which the Google Chrome team has not -- so you won't get stuck using a front-facing camera.

Addendum: Firefox OS does not seem to have fully implemented the HTML5 WebRTC webcam API but I am told it will appear very soon. Not that all that many folks have firefox os, but it'd be nice.

Read more Follow

android spectralworkbench firefox

Spectral Workbench Offline

by warren | about 11 years ago | 2 | 1

I spoke with some Firefox OS folks at Mozfest this weekend about the "offline manifest" system in modern browsers, and was able to put together a basic offline mode for Spectral Workbench.

How it works: If you log into Spectral Workbench, you will from then on be able to go to http://SpectralWorkbench.org whether or not you have an internet connection, on most modern browsers, including on smartphones. It will tell you you're offline but the Capture interface will still work.

However, it obviously cannot save data, and instead will prompt you to download it as images, which you must save and take notes about for later uploading, graphing, etc.

Have fun!

Read more Follow

spectrometer spectralworkbench offline