Project Details
Project Title: Advanced Searching and Sorting Tool for publiclab.org
Project Proposal: advanced-searching-and-sorting-tool-for-publiclab-org
Link to GSoC Project: https://summerofcode.withgoogle.com/dashboard/project/5438480599482368/overview
Open source Platform Used: Github
Organization mailing list: https://groups.google.com/forum/?hl=en#!forum/plots-dev
Github Organization Page: Publiclab
Repository Contributed to: plots2
Languages/Framework Used: Ruby on Rails, solr search server with sunspot, grape api framework, JavaScript(jQuery), HTML/CSS
Code Links
For this project, we used two repositories.
- Main Repo (upstream) - https://github.com/publiclab/plots2
- Primary Repo (forked from main repo) - https://github.com/david-days/plots2
Since we are implementing completely new technology to the system, one of my mentors (Bryan Bonvallet) advice was to use separate repository for this advanced search tool and avoid technological conflicts from other contributors. Specially from other GSoC projects.
All the commits - https://github.com/david-days/plots2/commits/master?author=ujithaperera
On going pull requests https://github.com/publiclab/plots2/pull/663 https://github.com/publiclab/plots2/pull/652
Project Description
Analysed existed code base and refactored for performance.
when i starting to contribute to publiclab, there was a normal sql query based search in the for the system. But that search was not able to cover up the current searching requirement of the system. Mainly advanced search page was needed to be improved.
Previous advanced search page
As the first step I went through all the search queries in the code based and refactored them to follow up MVC architecture and modified them to make search controller small and thinner, and moved those load to model. Actually here we used separated set of classes to execute these search queries and by putting this queries in this classes (service classes) anyone can touch the search logic without braking the normal request - response cycle.
After this sub task, we managed to generate search results similar to last state of the system. Then we planned this search for resource based structure. with this format we can enable few more features for this advanced searching tool. Such as search logging, search history, search suggestions, etc.
Introducing solr search server to the system
Since Publiclab is a text based (research notes and wiki pages) system and be supportive to Q&A tool, we used advanced text searching facilities provided by sunspot. And by using one text field (like the search box in publiclab.org) we can query easily from sunspot. Sunspot is a middle ware that use to communicate with solr search server through a rails application. Hence by using sunspot libraries we were able to install solr server top of the rails rake. with the previous query setup there were less work to be done for develop solr end points. Mainly text heavy research notes and its attributes were indexed to the solr database and facilitated advanced search page from solr search results.
With the observations of the modifications, step by step I added several additional features to the advanced search page. Few implementations got fail. But most of the tools were added to the page successfully.
one of the key tool that I was Implemented is Grouping. When you do a search for a word by using advanced search page, you can category these search results according to given fields and you can deep look for this results to touch ideal result in from the list.
Results are categorised for updated month
Dynamic search page and typeahead re-design
Real sense of the word, advanced search page is providing lot more details regarding your searching word. It's displaying results in some kind of analytical approach. If you are searching for a simple keywords all around the project, Dynamic search is the ideal one for you.
- simple preview
- full coverage
- auto-complete suggestions
- Quick access
Previous typeahead was a heavy library for the system and it consumes more resources to execute the service. It was hard and time consuming to add modifications or use typeahead with few customisations for a separate feature. Hence as a part of this project we re-design usage of typeahead and avoid heavy libraries to complete the particular requirement.
Previously reload was happened after three characters, Now it's reloading for individual keyup event and more faster.
RESTful API for search and it's configurations
This API developed using Grape api framework and by using this api all any api user can retrieve search results through the API. Dedicated end points are designed to cover up all the API level searches and by using grape we were able to deliver the required informations to the api user. Then an user can follow the details and connect to the search tool through the API.
3 Comments
@Ujitha Just small mistake here. The GSoC project link you gave is the one that links to your dashboard which needs log in. This is the link you should give: https://summerofcode.withgoogle.com/projects/#5438480599482368.
Reply to this comment...
Log in to comment
Hi, Ujitha -- thanks for your update -- your project's been really exciting to see, with vast behind-the-scenes changes for a function (search) which has long needed a thorough revamp. I appreciate your effort in bringing the power of Solr/Sunspot to plots2, while putting extra effort into making Solr installation easy for new coders. Very excited to see so much of your work now live on the site, and eager to get the remaining stuff merged, which I'm very optimistic about!
Thanks; it's been great working with you and I hope to do more together!
Reply to this comment...
Log in to comment
This is one of the largest merges I've been involved in, and in retrospect I think it may be good to be more careful and not merge master branch code into a feature branch before a final rebase -- the intermixing of commits led to some difficulties we could maybe have avoided if we'd just done one clear run-through as a rebase, and avoided any merge commits at all. Or tried to break it into smaller pieces and done a series of smaller rebase/merges instead of just two giant ones. But we're doing pretty well, and I'm sure we'll figure it out!
Reply to this comment...
Log in to comment
Login to comment.