Public Lab Research note


Thinking about a small Linux web app server initiative. Who wants to join?

by rjstatic | July 09, 2014 12:54 | 4,757 views | 12 comments | #10668 | 4,757 views | 12 comments | #10668 09 Jul 12:54

Read more: stable.publiclab.org/n/10668


Definition: Ground Server ~ A Ground Server hosts websites like a Cloud Server hosts websites on the Internet except it's on the ground which means the websites are accessible via your Local Area Network. Ground Servers are usually hosted on little credit card sized computers like the $35 Raspberry Pi or $45 BeagleBone Black from Texas Instruments. It's Ground Computing, not Cloud Computing. Originally coined by Jason Huggins. More about Ground Computing

Intro

Since taking my first web development class in 2004 I have been hosting my creations on the Cloud but for the past few years I've worked with Dogi (Public Lab server admin) hosting web apps in an unusual place, the ground where the applications are actually used. Servers that host web apps used to cost thousands of dollars but with the arrival of small Linux computers like the Raspberry Pi, you can host your own web apps locally for under $50. But it's not just about hosting web apps, the economics of "small Linux" can be used for environmental monitoring and automation which will have interesting implications for areas such as health, agriculture, environmental justice, and many more. In this post I hope to get you excited about the potential of Ground Computing and explain how we can build a foundationational kit for Ground Computing that will solve many of the common pitfalls when starting the development of a Ground Server. We need your help so please read on!

Our journey from the Cloud to the Ground, a summary of recent Ground Server deployments

In Ghana where it is cost prohibitive to have students online all day to access resources like Khan Academy, Open Learning Exchange Ghana and Open Learning Exchange International had been running a large box server in a small community to provide access to such resources. Last year with funding from USAID's All Children Reading program we were able launch the Ghana Reads program in 24 schools where we installed Raspberry Pi computers as the school servers, and with the help of CouchDB, we were able to easily create a sneakernet where we could deliver more content from the Cloud and feedback could sync back up to the Cloud. The school servers published the BeLL Web App which helped teachers plan their course activities and sync resources to $60 tablets.

The first six Raspberry Pi servers with the BeLL Software deployed in Ghana.

The first six Raspberry Pi servers with the BeLL Software deployed in Ghana.

These small Linux computers are good for more than just serving up web apps, they can also do data collection and automation. For example, the Fido project uses a Raspberry Pi with a connected USB temperature sensor to detect when a greenhouse is too hot or too cold. When a temperature is detected that is outside the bounds set by the Farmer in the local web app, the Ground Server text messages the farmer through the local WiFi connection (text messages can be sent for free via emails by the way). See the demo of a Fido in action at Wild Miller Gardens here and the check out the code on Github over here.

Within the first few weeks of having installed Fido, farmer Joel of Wild Miller Garden was alerted via text message that it was getting too cold in his Greenhouse at night for the vegetables he was growing.

Within the first few weeks of having installed Fido, farmer Joel of Wild Miller Garden was alerted via text message that it was getting too cold in his Greenhouse at night for the vegetables he was growing.

Back in 2011 I worked Mathew Lippencott and Molly Danielson of Cloacina on an online app that publishes historical environmental data about compost piles to surrounding neighbors, the same application today could be hosted on a $35 Raspberry Pi computer and the web app could be broadcasted via local WiFi with the help from the a project I worked on called Apitronic's Hive app. See the summary of that platform on the successfully funded Kickstarter page here and the code is on Github here.

Arduino monitoring a compost pile and sending data through text messages to the Cloud. The same thing could now be built for less without the need for the Cloud server.

Arduino monitoring a compost pile and sending data through text messages to the Cloud. The same thing could now be built for less without the need for the Cloud server.

Thinking about an Open Ground SDK

For the past few years Dogi and I have been working fairly quietly on the various projects mentioned above and through that work there are components I've found that are useful across those projects and useful for many other future projects by people like you! I'm looking to create an Open Ground SDK which will be a collection of code modules commonly needed when developing a Ground Server, many of which can be extracted from work already done. I'm also interested in developing an App server which when running will publish a web page for graphically using many of the modules in Open Ground SDK. These apps could serve as examples of how to integrate the Open Ground SDK modules into your own project or could be used as is. We also want to publish an example of Open Ground SDK in action for the Raspberry Pi (and possibly also the BeagleBone Black now that they support Debian!). This project could be called GroundHog which will have the App server preinstalled and could be used as a starting point for your Ground Server project.

Modules

For now, the following modules are wrapped up in the Fido project, Apitronics Hive project, and BeLL project.

  • WiFi Connect - Define what WiFi network the Ground Server should connect to.
  • WiFi Access Point Mode - Turn your Ground Computing device into a WiFi access point so you do not need any additional hardware to access the Apps on your Ground Server.
  • Hostname - Define the hostname of the Ground Server so it can easily be reached at a human readable URL on the network. Example: http://myserver.local versus the IP address which is hard to figure out http://192.168.0.101
  • Heartbeat - Configure where your Ground Server's heartbeat should call back to so the server you're pointing to can keep you up to date on the status of your Ground Server. Great for knowing when your alarms can't alarm you!
  • Updater - A framework for publishing updates for Ground Servers and an API to query for available updates and install them.
  • Disk space - a HTTP service for inquiring about the status of disks attached to a Ground Server.
  • Clone/Backup - A simple button for copying the harddrive of the Ground Server to another attached drive over USB.
  • Data Hive - View data saved from other devices sent to your Ground Server and manage alerts.
  • Sensor Pack - A collection of drivers for commonly used sensors.

How can you help?

Join the conversation

Currently we have a Ground Computing listserv on Google Groups but we are most passionate about having this project incubated at Public Lab. Lets discuss these possibilities on this Research Note on PublicLab.org.

What would you use the Open Ground SDK for?

We are really excited to get feedback from a community of folks working on ideas we've never thought of. Your ideas will shape how we approach the Open Ground SDK so we really encourage your input and brainstorming!

How can we raise funds to support development of this? Your thoughts?

While we think doing the proposed GroundHog distro and the Open Ground SDK is a great idea, we're not sure where we are going to raise the funds to support ourselves to get this work done. Would it be wise to do a Kickstarter? A Reverse Bounty? Do you know of a project we could work on that would result in building GroundHog and ultimately the Open Ground SDK?

What license should we follow? Your thoughts?

All of the projects I've worked on could not have been possible without the Open Source contributions of those who came before me and I intend to contribute back. For the Fido project we picked the AGPL license which I think best represents my intentions of my work remaining a part of the commons. This means that Open Source products that build upon this base will never have to worry about another company piggy backing off their contributions to the commons without also giving back. This is the license I find appealing but what's your opinion? I'm very interested in hearing the opinions of others who are interested in being contributors to this movement.

What do you think about these names?

Open Ground, GroundHog, etc. These are all our first ideas on names. What do you think?


12 Comments

"ground computing" would be a super useful way to host public lab apps locally like Mapknitter, Spectral Workbench, etc. Is there any way we could run the public lab application stack on such a small system? could data be federated with our existing apps?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


This would fit in well with the Commotion Wireless project (https://commotionwireless.net/) which is based on OpenWRT and includes an App Portal. We've been thinking about setting up a Commotion Mesh for the school and local town offices while setting up dedicated application portals for things like environmental monitoring and vehicle tracking. The Commotion team has even set up a few networks to run off of solar power so they can be used for back-up communications in an emergency.

Reply to this comment...


It would be truly fantastic to have such a system for capturing data from environmental monitors -- air quality, water quality -- with the idea that you could choose to store the data locally, or additionally broadcast it and store it remotely. I love that it's designed to be a sort of minimalist setup, with a collection of optional modules, and that it's based on the accessible, hugely popular RPi platform. Really eager to brainstorm about next steps.

ba-gh.png

Reply to this comment...


@mathew That's an interesting proposal. Moving Webs Apps on the Cloud to the Ground. It would be like, "Ya, Mapknitter is a web app but the users host it themselves." There might be varying degrees of "the users host it themselves". Here's a couple of questions a project looking to go from the Cloud to the Ground might ask itself.

  1. app pages are hosted on the ground or cloud?
  2. heavy processing happens on the ground or cloud?
  3. users' own data is stored on the ground or cloud?
  4. all users' data on the ground or cloud?
  5. all users host all users' data on the ground?

For an app like Mappknitter on Raspberry Pi or BeagleBone Black, the low hanging fruit might be (ground, cloud, ground, cloud, cloud) but a long term goal might be (ground, ground, ground, ground, neither because they share data using Tor protocols).

@code4maine I've seen that App Portal you speak of in Commotion Wireless and I've been meaning to look into the code. Dogi and I tried out Commotion Wireless on Ubiquity Nano bridges on a farm in New Hampshire but it's not part of our deployment yet because we defaulted to the standard AirOS (for the moment). I'm very excited about the potential of Commotion Wireless to lower the barrier to entry for folks looking to create wide reaching LANs.

On the App Portal part, I haven't looked into their code yet for that but I've considered an "App Portal" for the Open Ground SDK. One difficult part is if you have more than one installed, segmenting them could be tough given that Web Apps are sometimes designed to work on just one server so there might be a lot of stepping on toes. Maybe that could be solved with something like Docker... Can anyone with more experience with something like Docker weigh in on that?

Lastly, thanks for pushing the Commotion stuff, I'll reach out to Josh King, lead technologist for OTI which oversees Commotion Wireless development, to see if he has any iput on this thread.

@donblair I love that logo! It captures the coolness of this project so well ;). On broadcasting data to the cloud, I see that syncing to the Apitronics Cloud is now a pull request for the Hive code which is exciting! Hive also uses CouchDB so technically if you host your own CouchDB online or use any of the paid CouchDB hosting (Cloudant, IrisCouch, Couchappy, Treehouse) you can sync your data up to the Cloud by copying and pasting your Cloud's URL into your local CouchDB's web forms. Easy peesy (sp?)!

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Exciting stuff. Love to see how the power of these low cost platforms can be used in these contexts. I'm mostly following and trying to apply the tools rather than building them. I've seen several interesting applications from: http://poweringpotential.org/techtent.html to students from Stanford's Engineers for a Sustainable World, using these tools for a variety of projects: http://instagram.com/p/ozgE5kptlH/ http://instagram.com/p/ozXDW3ptnN/ and others I didn't capture.

Reply to this comment...


Re: mapknitter-- right- there would be some sort of trade off between running on the ground and the cloud, perhaps just a local caching of the interface so that Mapknitter could still work with a slow internet connection. But then users would wait for the cloud upload.

@donblair omg that logo is so good why are we not using it right now.

Reply to this comment...


re: licensing, perhaps you could AGPL the software and CERN OHL the hardware (insofar as you develop a custom hardware setup, even if made from commodity components)?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


How about LXC rather than pull requests for updating?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


@crtalermo Linux Containers (LXC) looks interesting. Are you thinking of that as a way to sandbox Ground Apps?

@nemo Have you any experience with LXC?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Found this via the CouchDB blog (http://blog.couchdb.org/2014/07/11/couchdb-weekly-news-july-10-2014/) rather than the homepage — this is something that I've been mulling over and trying to build apps for for many years. (I actually used to think of this idea as "CouchOS", basically a platform that took care of sync/security/scaling for apps the way it once looked like CouchDB would.)

Anyway, this is something that really interests me. Taking the architecture of what Tim O'Reilly calls the internet operating system but putting it in the hands (or rather: apartments, toolsheds, backpacks, outposts…) of users.

It's an uphill battle and I've all-but given up on commercializing it. ("HEY YOU'LL LIKE THIS — it's kind of like some of the things Google/Apple already gives you for free, but doesn't come preinstalled, has to stay plugged in to the correct port on your router — yeah that important box with the blinking lights you aren't supposed to mess with. Since we skimped on the graphic and industrial design, we were able to keep the price down to a mere $350. What? No, sorry, it doesn't run Facebook but there's this federated cryptographic peer-to-peer overlay mesh topology used for messaging; Homeland Security wants to make it illegal for civilians to use but the EFF is — hey where are you going?")

I think a sandboxed approach like you're talking with VMs/LXC is reasonable, gives the most flexibility for running legacy stuff [WordPress/MediaWiki/etc.] and could help protect some core auth/storage/etc. processes from malicious apps.

I've mostly been focused on higher levels though, tinkering with some apps (e.g. ShutterStem, LocLog/Metakaolin, and a handful of small tools while keeping an ear to the ground for technology that could help make this usable (everything from mDNS to Telehash, Tahoe-LAFS to CRDTs, from OpenID to BrowserID to OpenID Connect…).

I'm rambling and am supposed to be catching up on "pay the bills" work, but hope to read through more of what you've already got here and start a real conversation at some point :-)

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Oh, see also https://sandstorm.io another recent push for this I've seen (plus some other projects I have bookmarked, https://delicious.com/natevw/beyond84).

Reply to this comment...


@natevw Thanks for you comments. We should Google Hangout, I'd love to hear more about the projects you know about. It looks like our paths have crossed on the Firefox issue tracker.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Login to comment.