Public Lab Research note


[GSOC PROPOSAL DRAFT] MapKnitter Enhancements.

by xvidun | March 07, 2014 20:54 07 Mar 20:54 | #10138 | #10138

Name: Vidhun K.

Affiliation:Degree - B-Tech Computer Science

Location: India

Email: xvidun@gmail.com

Phone: +91 9524281789

Project(s) you're working on or want to: MapKnitter

Project title: Enhancements to MapKnitter.

Project description

Summary:

This project will work on MapKnitter to enhancing the upload interface and working to improve general interface with bootstrap. Project will also focus on adding feature to auto-knit images using the EXIF metadata from images.

Describe the need your project fulfills:

  • Enhance the upload interface to support drag/drop and multiple uploads. This can reduce the waiting time drastically and would be helpful for people working on a mobile device.
  • When knitting an image it would be good if there was some data available on the images from the region an image is knitted to. This way the uploaded image can be compared and reviewed based on the reference.
  • The current process of geo-referencing is quite simple and straightforward. The problem arises when required to upload and knit multiple images, which is troublesome and also time consuming. This project aims to make the MapKnitting process easy on the end users.

How will your project meet this need:

  • First issue that this project will look into is about interface enhancement, some of the key points for interface enhancement are.
    • Using a drag and drop upload interface.
    • Handling multiple image uploads.
    • List all uploaded images off map letting the user select the required ones to knit.
    • Retrieve and display a list of "Other images from this region" sorted with time to display to the user.
    • Make system wide enhancements to the interface using bootstrap. Working on modelling the home page can be a start.
  • Knitting multiple images using the current approach will not be easy as the users will have to scale, rotate and move the image to the right position on the map. This process can be simplified if we could use the EXIF metadata as much as possible to approximately guess the location of the image in the map. This way the user will have to make only slight alterations depending on the error in the GPS recordings. Even if EXIF data was not present we can fall back to the manual approach.

Auto placement and auto-rectification approach: Sample EXIF that would be usefule(Iphone):

| EXIF | Value | --- | --- |
| GPS Latitude Ref | North |GPS Longitude Ref | West |GPS Altitude Ref | Above Sea Level |GPS Img Direction Ref | True North |GPS Img Direction | 346.4727273 |GPS Altitude | 1222 m Above Sea Level |GPS Latitude | 37 deg 44' 10.80" N |GPS Longitude | 119 deg 35' 58.80" W

  • Placing the image in the right position depending on latitude and longitude,
  • Rotating the image depending on the image direction.
  • Investigate a method to find the ratio of scale to the image height with reference to the map. Using this we can either scale the image or zoom in/out the map.
  • If Anish's project on porting Image Distortion to leaflet goes through it will be necessary for this project to work in coordination. This will be acheived with regular discussions and keeping track or the project progress.

Timeline/milestones:

Throughout the development process I will be recording the ongoing development on a blog(once a week), making it easier for the mentor to track my progress.

  • Precoding Period (upto may 19):

    • Get a familiar with MapKnitter and the codebase.
    • Investigate and learn the required tools required for the project.
    • Bond with community while identifying required resources for learning or help.
    • Plan a basic approach to solving the project problems noting and clarifying on the issues involved.
    • Investigate and learn possible libraries that can be used like Jquery-file-upload and extracting EXIF metadata, doing some reading on EXIF and TIFF would do.
  • May 19 - June 5:

    • Work on the implementation for drag and drop functionality probably using the Jquery-File-Upload plugin, can be decided on later.
    • Implement feature to upload multiple images and list them off map and select them when required.
    • Work on bootstrap to improve the general interface in mapknitter.
    • Options for user to view sorted list of images from the particular region.
  • June 5 - June 10:

    • Final touch on the implementation is made and basic documentation, code review, testing and bug fixing is done.
  • June 11 - June 21:

    • Works on the auto Geo referencing implementation is started.
    • Initial works like extracting EXIF data from image, collecting data that can be used is done optionally notify the user on the required EXIF metadata that will be used to Knit the image.
  • June 21 - July 5:

  • Implement the basic functionality for placing the image according to latitude and longitude while testing alongside.

  • July 5 - July 20: (2 day break)

    • Work on rotating the Image depending on the true direction in the EXIF with reference to the map.
  • July 20 - August 11:

    • Implement feature to rectifying the image(scale) depending on the altitude EXIF meta data.
    • Work on correcting image based on the tilt.
  • Upto August 21:

    • Test the overall code, fix bugs, document the areas that are necessary while getting ready for the final evaluations.

What broader goal is your project working towards?

To make MapKnitter more accessible and therefore getting more open geo referenced data.

What resources will you need:

  • People: To point me in the right direction regarding the code base and learning resources.
  • Sample Data: Some EXIF containing sample data (Android / Iphone / skycamera)

Experience

For the past 3 months or so I have learning and hacking around with rails framework while falling in love with it. I’ve learnt rails from various resources incuding Michael Hartl tutorial, Railscasts, Rails zombies for best practices. I’ve come to a better understanding of rails with a couple of good and bad experiences on the learning curve. I use the vim with rails-vim plugin for development. I’ve have decent experience using javascript and also worked with some of the responsive frameworks like Foundation, some bootstrap. I use git for my workflow productivity and code collobaration extensively. This project will mostly depend on javascript, therefore I will learn and become sufficient for the project using the pre-coding period.

Some of the open projects I’ve worked on: I’m relatively new to open source and therefore don’t have any major contributions yet, looking forward to this summer to learn, create and collaborate.

  • Integrated the dogecoin support for the bitcoin ruby gem.
  • Worked on a simple kick-start application using devise for authentication with Openid for google, facebook and twitter.
  • Added a bitcoin payment and donation functionality for the privly-web stack. Used the blockchain bitcoin-receive api for this.
  • Contributed to creating the college tech fest website in 2012.

Teamwork

The idea of working with a team and collaborating on different ideas keeps me self motivated and my overall productivity is at its peak. There were several college programmes like conducting workshops, college projects where I had to work in a team.

  • Working in projects involved working with different people on different level and areas of expertise. This meant that everyone had to give their part to and work on fresh ideas to bring out a successful project.
  • Conducting workshops in college was on a whole different level of teamwork, where we had to contact different professors for ideas and suggestions. The hardest part would have been publicizing the event not losing confidence along the way.
  • Environmental science project to collect household and industrial data to analyze the efficiency of solar cells done by splitting the team into different groups to reach out and get maximum data.

Expertise

I’m particularly interested in Open source software and have been using them extensively for the past three years. I have have experience working with OOP(c, java, python, ruby) and web technologies(Ruby On Rails and Php(CodeIgniter). Apart from this I love algorithms and datastructures, to sometimes work on it in my free time. For expertise relevant to this project I have decent experience working with rails and some javascript. This means that I will have to use the pre-coding period as efficiently as possible to learn everything including the libraries that would be required for this project.

Interest

Although I have interests towards environmental justice I have no major contributions yet. In an effort to work for environmental justice I’ve participated in some of the city clean drives conducted in the college. I have worked on a team based project to collect data to analyze the efficiency of solar cells.

Audience

The prime audience for this technology would be lay users. This project will completely benefit the lay users by making a user friendly and a faster map knitting experience hiding the underlying stack of its working. With the success of this project the user can simply drag drop an image and let the software work out most of the assumptions of the position of the image in the reference map.

Context

MapKnitter although having a good interface for knitting can be made better with this project. This means that there would be more lay users that will be able to use the software, which is a great thing.

Ongoing involvement

This summer will help me collaborate with different people from the community, these connections will introduce me and get me started in making contributions and join volunteering efforts. This would also help me in getting involved in the various other open data efforts and help me in working on the areas that interests me.

  • Post gsoc, I looking into long-term contributions to public lab. Apart from the technical contributions, It would be awesome to volunteer for cartography collective considering that my native is a eco-hub it would be helpful for environmentalists to analyze the real time cartography data. This would not be an easy task considering the lack of awareness but would interesting though to form a community of local members involved in making team driven aerial photographs.
  • After a successful summer, I will make complete efforts to maintain and develop the software written during the summer. This way I will make any enhancements when a new idea for the existing system comes up. I came across a research note posted be user mathew on adding annotations to MapKnitter, looks like an interesting thing to work on for MapKnitter.
  • Apart from this I will involve in collaborating with working with developers from the Public Lab in improving and writing software for Public Lab efforts. This summer will provide me with the necessary resources in terms of knowledge and connections to help me get involved post gsoc.

Commitment

I can give around 35 - 40 hours per week for the summer coding. This summer I look forward to create, learn and make life long connections and contributing to Public Lab for GSoC gives me the exact opportunity. I will set aside all other activities as far as possible during the summer although I have to take some breaks in between(3 days max) and I had reflected them in the timeline as accurately as possible. Even if I have to take emergency breaks I will inform my mentor before hand and make up for the coding activity in the later days. For this GSoC, I am looking to get permission to stay back in the college(hostel) during summer so that I can give my time free from interruptions and will also have the necessary resources and also the working environment ready to work with maximum productivity.


12 Comments

Vidhun,
This looks impressive. The interface for uploading photos is a great place to start improving MapKnitter. The current system requires that photos be uploaded one at a time with a subsequent wait of 20 to 60 seconds during which the user can not interact with the map in progress. The first approach to improving this might be to allow upload of five or ten photos at a time so all the waiting is done in advance. Either "select files" or "drag and drop" would work fine. Drag and drop is ubiquitous so probably easy to implement. The next issue is what to do with the photos after they are uploaded. Should they be displayed on top of one another in the middle of the map? Should they be aligned in a row off to the side? Should they be displayed as icons, any one of which can be opened and displayed as the user needs them?

EXIF location data in the photos could be used to place them on the map, but this has shortcomings:

  1. Very few photos ever uploaded to MapKnitter have GPS data in the EXIF header.
  2. GPS data will come from the camera in the air and not necessarily account for tilt and subsequent displacement of the photo's location on the ground. This location could be calculated, but that requires data about camera altitude, orientation, tilt, and local topography. GPS data on altitude is never very good, not all GPS systems record orientation and tilt, and modeling ground topography is probably beyond the scope of this project.
  3. If the photos are simply placed at the lat long coordinates of the camera, they should be rotated properly, using orientation data (NSEW, which is much rarer than lat long data). If they are not rotated properly, the advantage to the user might be minimal.
  4. Most maps at MapKnitter are just 5 to 15 photos of a relatively small area, taken by a camera that moved horizontally through an even smaller area. So automatically geolocating the photos for most MapKnitter maps will result in all the photos being placed in pretty much the same place anyway. It is not clear that this would be much help to the user.
  5. In some cases of homogeneous ground surface when there are hundreds of photos, automatically placing the photos can be useful. But in most cases it is obvious and easy to rotate a photo into the correct orientation and place it. This is one of the fun parts of MapKnitting.

Assuming a set of photos had good lat, long, tilt, orientation, and altitude data, and the ground underneath was perfectly flat, is it a reasonable summer project to create from scratch a system to place the photos on a map? If such a system was created, would it be easy to also stretch the photo to account for camera tilt?

Some questions for the MapKnitter community at large:

  • Do such systems already exist?
  • Do your aerial photos include lat, long, altitude, tilt, and orientation?
  • Would you invest in the required equipment to record these data if it was used by MapKnitter?
  • Are there other improvements to MapKnitter that would be more useful to typical users?

Chris

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Chris,

Drag and drop is ubiquitous so probably easy to implement. The next issue is what to do with the photos after they are uploaded. Should they be displayed on top of one another in the middle of the map? Should they be aligned in a row off to the side? Should they be displayed as icons, any one of which can be opened and displayed as the user needs them?

This would be a great addition to the image upload interface, my approach would be to upload multiple images and list them off-map. The user can then select photo's to display them on map.
Again my initial proposal was to take only the auto georeferencing project and basic enhancement of upload interface. So adding more features would just be a failure to complete in the summer time. I will think over this and update proposal accordingly or maybe I can work if I complete my project early.

EXIF location data in the photos could be used to place them on the map, but this has shortcomings:

I understand this, but as I mentioned my project idea is to guess the image location on map as much as possible using available EXIF data, so if no data is present the user can fallback to regular approach.

Very few photos ever uploaded to MapKnitter have GPS data in the EXIF header.

Users who use Mapknitter frequently will find the current process painstaking. Due to this, some kind of tutorial or steps on ensuring GPS data can be advised for the users.

Most android devices support GPS(lat, long, alt). Probably an android app for taking images?

I think Iphone photos has GPS in their EXIF(not sure about this, anyone using Iphone here?). Iphone sample data

GPS data will come from the camera in the air and not necessarily account for tilt and subsequent displacement of the photo's location on the ground.

As mentioned earlier, the idea is to position the image approximately using the EXIF data. The position need not be final, therefore the user can correct them as required. This is better than just placing the photo at an unreasonable position on the map.

This is one of the fun parts of MapKnitting.

Oh, I never gave thought to the fun part in knitting, maybe an option to turn off auto georeferencing?

it a reasonable summer project to create from scratch a system to place the photos on a map? That would be great thing, should probably note this and give it some thought.

Thanks for the great points and suggestions, will review your tips, questions and update the proposal accordingly.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


my approach would be to upload multiple images and list them off-map. The user can then select photos to display them on map.

I agree, this is a good way to do it. If the user wanted to upload a few more photos, could it happen in the background?

Does anyone have a small set of photos with GPS data that have been made into a map by hand? It would be really interesting to see the relationship between the photos' final lat long centroid on the map and the GPS coordinates in the jpg files. How strong does this relationship have to be to make this project useful?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Most android devices support GPS (lat, long, alt).

Some day, more people will be using smartphones to take aerial photos, but today, a $40 (ebay) Canon Powershot takes much better photos (bigger sensor, better lens, better control) and is much easier to convert to infrared. Collecting GPS data for these photos generally requires additional (and more expensive) hardware, so it is not done very much, and we don't know much about the quality of the GPS data or how it relates to the ground position of the photos.

Reply to this comment...


Some day, more people will be using smartphones to take aerial photos.

Is there some data on the number of images on mapknitter that contain GPS data. This way we can ensure that the project will prove useful.

Reply to this comment...


Hi! just a few thoughts to add:

  • For this project, you may begin to overlap with this other GSoC proposal a bit -- not the same code or functions, but you'll have to coordinate: http://publiclab.org/notes/anishshah101/03-08-2014/gsoc-2014-proposal-porting-imagedistortion-class-in-the-mapknitter-interface-to-leaflet-js
  • I think the best way to approach this coordination may be to skip making separate "maps" for now, and just focus on people uploading images one by one.
    • So, the map would just show all images by that user, or (with a toggle to change this) all recent images by this user, or all images from all users for this region.
    • Exporting a map would involve dragging to select a group of images which you wish to export, rather than having lots of separate named maps
    • These last few distinctions may not be that important for your project
  • the upload interface, once you're done uploading an image, would drop you back into the map. If it's simple enough, you could put it on both the front page of the website (so people can begin making a map really quickly) and within the map editor interface. But the editor will be changing if the above project by Anish goes forward, so keep your code clean and portable.
  • the upload interface could be modeled on http://publiclab.org/post
  • finally, re: having fun, the auto-placement will basically never be good enough to make some manual adjustment unnecessary.

Great start to your proposal!

Reply to this comment...


For this project, you may begin to overlap with this other GSoC proposal a bit.

I've read the proposal and also did some digging, from what I understand the approach is to move the image distortion class to leaflet.js. Parts of my project relies on using image distortion, probably work with the initial code and make changes later on once Anish has completed his work (portable)?

I think the best way to approach this coordination may be to skip making separate "maps" for now

Do you mean "multiple image" feature? As I mentioned earlier my approach was to list all currently uploaded images off map allowing the user to select what to load on map. Would it affect the coordination if we allowed user to select one image to knit at a time from the list of uploaded ones? Since you mentioned I am confused as to how Anish's project will make changes to editor, apart from porting is there any visual level mods that would be made?

So, the map would just show all images by that user, or (with a toggle to change this) all recent images by this user, or all images from all users for this region.

This would be awesome. Will reflect this in proposal.

the auto-placement will basically never be good enough to make some manual adjustment

True. So any possible reason that user would require for toggling auto-placement? Probably the GPS data doesn't make sense. Thoughts?

I'll be working on modelling the interface. Will have note the list of features that will be required on the interface.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Vidun - i'm going to paste in some questions and thoughts here related to how your project overlaps with Anish's proposal. Some of these are for Anish, some for you, but both projects will be related and it'll be important to figure out with Anish where your work fits together, and how you'll each ensure that your work will function even if the other's does not yet work. And how they'll work together if both succeed.

  • Upload interface is a partial, which can be included in various places
  • Recent images also
  • Separate JS code files
  • Leaflet work contained in Leaflet plugin
  • Could upload function first be integrated with current system, but clearly mark where it would begin to work with Leaflet system once that's done? Via agreed-upon callback and/or constructor method?
  • Upload of each image to server completes with a callback like "imageUploaded()" which the Leaflet system can overwrite when it's ready?
  • Leaflet system, if completed, will need a) transparency, b) outlining, c) locking
  • new Bootstrap interface overhaul for MapKnitter could be part of Upload project.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...



how Anish's project will make changes to editor, apart from porting is there any visual level mods that would be made?

We've discussed his re-implementing the buttons and such in Bootstrap, which would be much simpler. Your work might intersect his if you contained your code to a Rails partial (read up on this... it's like a reusable template fragment) which could be included in the old version or Anish's new version of the editor. Or you could have faith that Anish will finish his work, and just focus on an image uploader/gallery/placement tool which instantiates a new in-editor image from a list of uploaded images (as you said) and hands off the relevant data to Anish's code for the user to begin manipulating.

For the auto-placement, you could pass "suggested" placement along with the constructor to Anish's code to instantiate and display the image.

any possible reason that user would require for toggling auto-placement?

Perhaps at the moment you upload, it could have a checkbox (default checked) that says "attempt to auto-place". Or it could just try, and if it fails, it could briefly show an unobtrusive message that it couldn't find a match. If you right-click the image, or long-click it, maybe it could offer to try again.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


I've been busy these days with exams.

As said I will have to contact Anish, probably look into making the upload feature as a start to begin with the coordination.

About the toggling, I've been weighing the pros and to me I don't think this feature will do any good. I'm not sure of this but can be decided later. Depending on the user corrections, I was also thinking it would be better to make the complete tagging inherently with EXIF(modify GPS corrections by user writing to the image EXIF).

Reply to this comment...


I found a map with GPS coordinates in the EXIF data: https://mapknitter.org/map/view/cuas-balloon-mapping

Reply to this comment...


Login to comment.