Public Lab Research note


GSoC proposal: Image-Sequencer v2 : Processing on steroids

by tech4gt | February 19, 2018 13:57 19 Feb 13:57 | #15767 | #15767

About me

Name: Varun Gupta

Email: varun.gupta1798@gmail.com(personal) / varung.it.16@nsit.net.in(educational)

Github: tech4gt

Gitter Nickname: tech4gt

Location: New Delhi, India

Affiliation: BE Information Technology at Netaji Subhash Institute of Technology(NSIT) (Batch of 2020)

Project description

Image Sequencer is a sequential image processing library completely written in javascript which currently supports Node.js and the browser. It has the capability of outputting the processed image at each step as well as a CLI for ease of use right from the terminal.

Abstract/summary:

Goal of the project is enhancing Image-Sequencer for both users and contributors by adding more modules, enhancing existing ones, improvisations on demo, an even better CLI and an a plugins system to make module contribution easier and independent.

Problem

  • The current module contribution system needs for you to clone the code and make changes to it and generate a pull request, this is not a solution viable for ling term because the source code size would keep on increasing, also if a user does not wish to use all the modules he/she would have to download all of them, this also makes debugging difficult.
  • Current demo is not very feature rich and needs improvements in various areas, for instance it doeas not support caching of files for offline use, sharing directly from the demo and the UI can use some improvements as well
  • The current CLI syntax is long and tedious, does not support naming of the output image and hence chainability is not very effective. For instance currently to chain the cli with moving the final image to a new location looks like this
sequencer -i "image path" -o "path to output directory" -s "steps" && mv "output path/sequencer
  • Some modules need very high computational power and the CPU is falling short, Edge Detection for instance takes incredibly long times to process for higher resolution images and hence these calculations should be GPU accelerated
  • The project is not setup with stuff like codecov, codeclimate and others
  • Mobile browsers and lower grade hardware have a hard time processing images due to limited processing power and thermal limitations, also long processing times give an unsatisfactory user experience especially on mobile devices.

According to statistics a considerable amount of users are using devices with low processing power, especially in under privileged communities and developing countries. This hardware can simply not run heavy modules locally.(in many cases mobile devices is the only device people own)

Proposed Solutions

Adding a plugin system (Issue 190)

Making it easier for people to develop modules independently, the modules will be divided into 2 categories, Core Modules and Plugin Modules

  • Core modules would be bundled with the source code and they would provide a basic set of functions
  • Plugin modules would be more complex and would not be bundled with the source code but would be installed via NPM and then can be binded with the main project to customize Image Sequencer according to needs

Currently the modules are hardcoded into the modules.js file but by having a modules.json file which will contain the details for all the modules currently bound to the system and modules.js will load its modules based on the information from this file and being a json file more modules can be dynamically added and removed

Building a CLI Around the new architecture so that non technical users can directly add modules from the CLI, after completion this will be the workflow

npm i image-sequencer
npm i sequencer-blur
sequencer --bind-module sequencer-blur
sequencer -i "

Adding a one line module building option using the dynamic module

RESTful API (Issue 198)

An express server and a RESTful api for clients with low processing power. This expands the reach of the project to a much wider audience particularly people who cannot afford higher tier hardware to run complex modules like Edge Detection.This also provides a new way of using image sequencer and opens the project up for future clients such as native apps. Also the user will have an option in the UI to use local machine or remote server for processing and a time threshold in which if the processing does not complete the code will fallback to using API. Images will be uploaded to imgur api and link will be sent to the api. API structure is listed below:

/api/v1/convert

Request body:
{
    image: String,
    steps : [String],
}

Response body:

{
    images : [String]
}

CLI Improvements

Refactor the cli and add -n --out-name option to name the final output image, by default this will be set to the name of the input image. Extension detection and conversion to seamlessly convert between file formats without even specifying the convert step. The syntax for the above example will look like

sequencer -i "photo.jpg" -s "steps" -o "out dir" && mv "outdir/photo.jpg" "target dir"

A conflict of name will be resolved by prompting the user for a new name and by default adding appropriate number to the name of the image

Also using the -n option for automatic extension detection the following command will invoke the proposed jpg to png module on the final image and export it as a png image

sequencer -i "photo.jpg" -n "photo.png -s "steps""
  • Setup the project with CodeCov, bithound,codefactor, codeclimate and codebeat to increase code maintainability and quality. A deploy to heroku button for quickly deploying and testing our newly created server on heroku Documentaion should look something like this

Current issues on github which will be closed: #177,#178,#176,#172,#159,#158,#157#138,#130,#119,#116,#112

Linting

Since linting is a very important part of development workflow and ensuring code quality, the code written would be properly linted in order to be less bug prone and better readability. Also I would add proper comments so as to make the code easier to understand for new contributors

Timeline/milestones (May 14 - Aug 6)

My semester exams end around 30 May and hence I would not be able to do very significant contributions in the first one-two weeks but I will make up for it in the following weeks(Link to my academic calendar : http://www.nsit.ac.in/static/documents/calender/UG/calender_B.E. even semester 2018.pdf

Weeks 1,2 (May 14 - May 27)

CLI syntax refactoring and addons like adding the final name parameter and implementing extension detection for a new and supercharged CLI which is chainable.

Week 3 (May 28 - June 3)

Implementing the conversion module which converts images into different formats to integrate with the CLI extension detection also implementing one line module genaration with the CLI using dynamic module. Peer and mentor review for the new CLI.

Week 4,5 (June 4 - June 17)

Implementing the new module system and make separate existing modules into core and plugins, publish the plugin modules as npm packages, building the CLI for it.

Week 6 (June 18 - June 24)Fixing bugs or delays encountered, adding unit tests for the features implemented and incorporate the suggestions from the community into the codebase

Week 7,8 (June 25 - July 8)

GPU accelaration for heavy calculations with GPU.js. Adding support for CodeCov, Bithound, Codefactor, Codeclimate and Codebeat to increase code maintainability and quality and updating the documentation with the badges plus a deploy to heroku button so that new contributors can easlily deploy the API. Peer and mentor review for the new module system.

Week 9,10 (July 9 - July 22)

Implementing finalized module ideas. Writing unit tests for the new modules. Peer review for gpu accelaration.

Week 11,12 (July 23 - August 6)

Feedback and bug fixes. Final submissions and completion. Start with the RESTful API work on which will continue after summer

Needs

I would be implementing all the code i have mentioned above but help from other contributors in always appreciated ;-).

We will eventually need to deploy our API to a server with decent processing capabilities.(Can also be done later at anytime)

Contributions

I have been actively contributing to the Image Sequencer project since quite a while now and recently I have been accepted into the Public Lab reviewers team.

This is my fork of the project: https://github.com/tech4gt/image-sequencer

Below is a non-exhaustive list of my most notable contributions to the project

CLI Improvements

  • Added Basic Mode to only output final image (Merged PR #154)
  • Added Progress Bar for syncronus operation (Built PR #180)
  • Added Config parameter to pass the details of the steps (Merged PR #171)
  • Create Output Directory immediately before image processing (Merged PR #167)
  • Inform The user of the async operation(Merged PR #152)

Module Contributions

  • Added Canny Edge Detection Module for detecting edges from a png image (Merged PR #168)
  • Added Brightness Module for manipulating percentage brightness(Merged PR #161)

Documentation and bug fixes

  • Added bookmarklet to replace all images in a page (Merged PR #164)
  • Created documentation for the added features and bug fixes (Merged PRs #174, #162, #160 ...)

Apart from this detected and raised issues (#177, #176, #174, #170, #162, #158, #157, #153) and reviewed PR #155,#179

**Full List of all PRs and Issues submitted **https://github.com/publiclab/image-sequencer/issues?utf8=%E2%9C%93&q=author%3Atech4GT+


Experience

I am a Node.js based web and android developer, a javascript and kotlin lover and contributer to various open source projects as well as author of some.

I interned at open source organization Coding Blocks for the better part of 2 years where I helped with open source development of Node and Android projects and acted as a teaching assistant for the bootcamps. Below is a list of my most notable projects

1. Passport-Oneauth :

A custom passport OAuth2 strategy written in javascript for coding blocks' Oneauth Server which plugs into the passport middleware and provides authentication for the OAuth2 compliant server oneauth, inspired by passport-facebook and passport-google-oauth (Authored)

npm: https://npmjs.com/package/passport-oneauth

github: https://github.com/coding-blocks/passport-oneauth

2. Shortlr :

A custom URL shortening service written in node.js inspired by google's goo.gl adding features like custom shorturls and login with OAuth2 (Contributed)

live demo: https://cb.lk/admin

github: https://github.com/coding-blocks/shortlr

3. Hacker-Blocks Extended :

A Node.js based portal for submission and evaluation of static websites and android apps (Co-Authored)

Hacker-Blocks-Extended/dev-submit is a complete backend built on an express.js and potgreSQL based stack which has a RESTful API for submissions and testing using phantomjs and Appetize.io . Our vision for this project is to build a portal for practicing development like there exist for competitive programming. (Co-Authored) (Ongoing)

github: https://github.com/coding-blocks/dev-submit

4. Chrome Extension for shortlr :

A chrome extension that shorts a url using the shortlr API and copies it to user's clipboard (Authored)

github: https://github.com/coding-blocks/shortlr_chrome_extension

Teamwork

I was a part of the coding blocks organization where we teach students how to code. All the code written here is open source and i worked with the team of interns on many open source projects some of which are mentioned above. Apart from that I also helped students learn as teaching assistant in the bootcamps "Data structures and algorithms in java" and "Android App Development". I cleared doubts and helped people debug their code and hence I have complete understanding of working in a team, respecting other members' opinions and maintaining protocol. I understand the importance of team work in any great project.

_
_

Outreach_I strongly believe in sharing knowledge and collective growth, I have tried to encourage other contributors and newcomers to contribute to the project through opening FTO(First Timers Only) issues(List Here) and I would continue to do so. I will allow for newcomers to contribute to the project by breaking up tasks into smaller pieces and opening FTO issues for easier tasks. Also a part of my proposal is to allow for independent module development so that developers can make modules for image-sequencer without going through the process of getting it approved, also they would be able to publish their modules separately if they want to. I would love to work with other people on the project and i will try to make the project as developer friendly as possible.

_


Passion

I am a javascript lover and an Image processing enthusiast. The way we can change an entire picture by simple mathematical transformations is just fascinating for me and hence i love Image Sequencer. I also admire the way we can apply image processing in real life to get various types of information from plant and soil health to particles in water.

Audience

I believe this project can be helpful to small scale farmers and peasants where the health of their crops and soil can be analyzed and especially in a country like india where majority of the people cannot afford to buy expensive equipment and pay loads of money for analysis.

Commitment

I completely understand the commitment here and I would love to commit myself to this project and grow as a programmer, I currently spend almost 6 hours a day on open source development I am willing to devote any amount of time required for the program and since the program overlaps with my semester break it would be possible to do so.


35 Comments

@warren @liz please review

Reply to this comment...


@warren Updated with the github issues and recent contributions

Reply to this comment...


@warren there is one other idea that i had but i eid not include it here because there was no room so i posted it here https://github.com/publiclab/image-sequencer/issues/190 . If you think that is the way to go maybe we can drop other things and include that

Reply to this comment...


Hi! Lot of great ideas in here. I like the GPU one -- for this and other ideas, can you link to the corresponding GitHub issue next to the item in your proposal? Also just to watch out, if you use #___ syntax on PublicLab.org, it'l link to the PublicLab post of that id, not the GItHub one, and will add some tags... so maybe you should just use the number and not #?

I really like your ideas about a remove server-supported system for some phones, but I worry that it's a huge project. I'd like to emphasize solving some real-world problems with IS before tackling that, so for example trying to port over work by @amirberAgain and @simonPyle on particle detection. Something like that will really do a lot to encourage people that this architecture is promising and useful.

However, i do like the idea of coming up with a syntax for JSON requests to be sent to a remote server for processing. We should document that in an issue.

I really like the GPU idea, as I think most phone GPUs can do a lot to accelerate this process.One idea we'd had early on with @ccpandhare's input (and please @ccpandhare i'd love to hear your thoughts on this proposal) was to see if neighboring modules /both/ use GPU canvas and to pass the canvas instance directly rather than URL encoding and decoding, to save time. We could call it "context passing" or something.

One more thought -- there's a lot of UI work that could be done on the web version to help smooth things out. I think people would love to use this for lots of things Photoshop is used for today, if it were easy to use (since it's freely available too) and you could just open it in a browser tab. So, thinking of things like:

  • background threads so the UI doesn't hang
  • better progress indicators
  • better presented documentation and examples
  • better ability to "save and share" a particular sequence and its settings (we have some really basic URL-based sharing right now) so people can re-use sequences more

More soon!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


@warren First of all thanks a ton for this, I know how busy you are right now. Secondly i kind of noticed the # thing after I posted this so i manually deleted all the tags that were applied to this note hence no worries there. Thirdly I was thinking then maybe we can keep the remote server idea at the bottom of the priority list, maybe we can start working on that in the last week or so and i can continue working on that after gsoc?

And I will totally open issues for all the ideas i have had and link them here. One other thing that i wanted to ask was that maybe we can add this issue about addon modules to the proposal https://github.com/publiclab/image-sequencer/issues/190 we can replace the remote server idea with this on the priority list? As far as particle detection is considered no worries there it will be first thing I start working on after my mid-semester college evaluations and hopefully that will be done before summer.

I will most certainly research the context passing idea and open a well documented issue for that. Lastly thanks for the awesome ui ideas, even i have some ideas for the ui so i will consolidate all of them and make a ui improvements checklist.

Also I had 2 questions Firstly should we open issues for all the ideas that we have right now? Secondly do we include the progressive web app for the demo in the ui improvements checklist?

Once again thanks a ton for this, I really appreciate it.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Thank you @warren, this definitely looks like an awesome task @tech4gt. In the past I experienced with using an AWS EC2 running python for image analysis so I could provide some perspective there if needed. I think that the most important thing would be to search for your bottleneck, if it's the algorithms and not the network transport then doing some cloud-based image analysis is definitely relevant, otherwise not so much. If, on th other hand you need it to run where there is little to no cell reception or to be standalone - you might not want to go there. Have you considered using a single board computer with a GPU? there are even cameras with GPUs for <100$ see https://www.kickstarter.com/projects/1602548140/jevois-open-source-quad-core-smart-machine-vision I have no idea which libraries are available for image processing for Java, but be sure to scope that out. Best of luck!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


@tech4gt The PWA idea and the RESTful API sounds very good! Good proposal overall. Although I think you can work a bit more on the GPU issue and plan for it. I tried covering that in my GSoC project last year, but it turned out to be more time consuming than I thought and we couldn't do it in the stipulated time frame.

The issue with this is some modules work with canvas and some don't. Running those which don't on the GPU means getting into WebGL. Basically, this goes towards WebGL no matter which direction you start in -- speaking solely based on experience!

I'd like to hear more about this! Sounds promising! Though you could work a bit more on the timeline and decide a focus for your proposal, as the ideas you have mentioned are no doubt great, but are vast as well. I personally feel that deciding a scope/domain is important before starting off with anything!

@warren Yes! The proposal is a very fine one indeed! I agree that some work on the UI is definitely needed.

Reply to this comment...


Ah, one more idea -- the web UI for adding new steps is a little bare, no? We could build that out so you can see the description of modules and read more about them before adding them, and also maybe so you can look through the available ones more easily?

https://publiclab.github.io/image-sequencer/examples/#steps=ndvi-red,green-channel

@ccpandhare -- thanks, and also want to say your summer's work was heroic, we always knew there was too much to do in a summer, so I'm so glad we may get a chance to build on it this summer. Thanks for your input!

(btw i'm sure you've linked to the GPU issues already but there is some good info on how we might 1. get GPU processing in Node.js as well as 2. have modules clearly indicate if they require GPU (i.e. web only) vs. ones that work in node AND browser. I think there's some space for this as long as modules are clear about it)

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


@ccpandhare thanks a ton for your input, i really appreciate it and honestly i built this proposal keeping yours from last year as my guide. I will definitely start researching and planning for the GPU issue and allocate proper time for it in the timeline. One other thing i wanted to ask you and @warren was that while i was reading proposals from last year someone mentioned building the web ui with vue.js will that be a good idea, that way we can make a super responsive UI and build it in a way that both web and mobile would be able to take advantage of the offline feature, service workers(which process on alternate thread) and better,more informative ui, what do you guys think?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Also @warren I think you missed my last comment, i asked some questions there, please do have a look, its above @amirberAgain comment

And thanks a lot @amirberAgain for your input I'll surely look through the resources

Reply to this comment...


Yes that's right, one person did talk about using Vue. But at that time Image Sequencer was in its initial stages and functionality had to be preferred over UI. It can be used no doubt it would be a very good choice! But that's what I was saying earlier, you should decide a scope - on one hand you're talking about using Vue (UI) and on the other GPU and RESTful API (functionality, architecture). You can definitely work on multiple things but since GSoC is just 3 months long, it's better to have a focus on a couple of these and not all!

Thanks for the very kind words, @warren !! It wouldn't have been possible without you and the positive culture PublicLab has! I still sometimes think about what I could do better back then :-)

Reply to this comment...


@ccpandhare yeah I'm kinda aware of that actually i wanted to put across all the ideas so we can pick the best ones...but i totally get your point. Please provide some input on which ideas you like the best?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


@tech4gt The best ones are always those which are required in the current scenario.

Image Sequencer needs architectural changes as well as functionality right now. I think it is better to focus on getting users on board – implementing more modules, creating a RESTful API maybe.

I would focus on the modules, features, architecture and then the UI (in order of preference)

But @warren only can say for sure!

Reply to this comment...


@ccpandhare thanks a lot, I'll structure it that way, plus i think we should drop the complete pwa for now, I can add elements like caching for offline use in the web demo itself along with other ui changes @warren suggested. Apart from this I'll add https://github.com/publiclab/image-sequencer/issues/190 at high priority and other improvements like modules and cli that i have already mentioned in the proposal, what do you say? PS dropping the pwa for now will give me a lot more breathing room and then I can focus on other stuff like modules and api very well.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


@warren @ccpandhare i have made changes to the proposal according to the feedback please do have look, thanks a lot

Reply to this comment...


Looks good!!

Reply to this comment...


thanks a ton @ccpandhare, credits to yours and @jywarren 's guidance

Reply to this comment...


:-) hoping to chime in a bit more soon... just got through a LOT of Summer of Code feedback and will try to circle back later today. I hope ryzokuken's input was helpful too!

Reply to this comment...


@jywarren yes everybody has been very supportive and helpful, i am really looking forward to working with this great community, gets me really excited :-D

Reply to this comment...


Just to chime in here on the REST vs. UI -- i think we'd like to see the online demo UI be refined a bit because I think that will help onboarding -- anything that'll help demonstrate IS in a very useful application will help there!

I also really like anything that'll make it easy to submit new modules. This'll help get new people in the door as it's easier than working on the core. A bigger team and more development will also drive interest in core refinements to accommodate real-world limitations we discover as we go!

So, i guess REST ends up lower on my list, even if I LOVE it as a concept. I think it'd be really useful for example to implement a completely separate service-based MapKnitter back-end, which could receive full-size images and distort them remotely for people. https://github.com/publiclab/mapknitter/ I'll leave this idea on your issue at https://github.com/publiclab/image-sequencer/issues/198

Thanks to both of you for thinking this through so well! So in summary, i think summer priorities ought to be:

  1. streamline module submission process for onboarding and growth - https://github.com/publiclab/image-sequencer/issues/190
  2. refine demo UI and provide more examples of usage for exciting applications, for outreach and demonstrations purposes: https://github.com/publiclab/image-sequencer/issues/197

Thank you!!!!!!

Reply to this comment...


@warren thanks a ton, what i have done is i have included the high priority things in the proposal and in the last 2 weeks, i will start working on REST and continue to work on that after summer(Most of the high priority things will be done by then) I hope that's fine and again thanks to @warren @ccpandhare @amirberAgain @ryzokuken for all the guidance and help, looking forward to working with you guys:-D

Reply to this comment...


@warren so is the proposal ready to submit on google's website? if there is anything i can change or elaborate on please do mention, thanks a ton

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


It looks great. We can continue to adapt as we go, too, so no worries. A next step could be to start creating a planning issue with checkboxes on GitHub as you've already started for the demo code. But that's not needed for the GSoC proposal itself. Thank you!

Reply to this comment...


@warren thanks a ton for all your help, so i just copy this on google's website right, i'll do that tom, thanks

Reply to this comment...


I think so -- but the application process changes a bit each year and I've actually never submitted a proposal myself. @ccpandhare -- does that sound right?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Also -- ask other folks on the main call for proposals page that I put up -- there's a number of folks on there who've gone through the process in the past. Thanks!

Reply to this comment...


@warren oh, thanks I'll do that now :-)

Reply to this comment...


Write just an overview on the portal and make this into a pdf, upload it to drive and link it there.

You can edit the project details part of the portal anytime!

Reply to this comment...


@ccpandhare thanks a ton for all your help

Reply to this comment...


@warren Please have a look I just added the Outreach section, should I explain in more detail or this would do?Thanks

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


@warren @ccpandhare I have uploaded the final pdf here https://drive.google.com/file/d/1SeupwVReQ3SX2XJ2fcdRLgoolI_Yt_oz/view?usp=sharing please have one final look if anything needs to be changed, I will then submit thanks a ton

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


This looks very good overall! Though I suggest you incorporate a buffer time to incorporate delays and time spent on bugfixing in your timeline. Also, you should write about unit testing in your timeline and linting in another appropriate section. :D

Reply to this comment...


@warren @ccpandhare i have made the changes you suggested, would this be enough? If there's anything else before submitting this please do tell, thanks a ton :)

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Hi, @tech4gt - this is looking good. I think it's a very thorough proposal and I appreciate all the time you've put into it!

Reply to this comment...


Thanks a ton @warren :)

Reply to this comment...


Login to comment.