What I want to do
In the first phase of the Open Pipe Kit development we'll build the engine of the car while the second phase will focus on the building the dashboard for end users to interact with. By the end of phase one we should have a working engine that you can ...
- easily write new Sensor Plugins in the language of your choosing
- easily write new Reservoir Plugins for in the language of your choosing
- tell the Pipe Engine what Pipes it should run by describing Pipes in a JSON document
Our first challenge in Phase One is to think about an architecture that will best accomplish the above goals. Here's my first crack at that.
My attempt and results
A Pipe is a process in the Pipe Engine that is configured to use a specific Sensor Plugin and a specific Reservoir plugin along with some additional data on how the Pipe should be managed (ex. frequency of polling the Sensor, what port the Sensor hardware is plugged into, which database the Reservoir Plugin should use, etc.). In other words, a Pipe takes data from a Sensor and gives it to a Reservoir at the frequency you configure.
Below describes how a Sensor Plugin, a Reservoir Plugin, and a Pipe may be described for the Pipe Engine.
Sensor Plugin architecture
A Sensor Plugin is a folder in the ./Sensors folder with the following structure.
./info.json // A file with info all about this sensor
./install.sh // A script that installs the dependencies for the sensor
./uninstall.sh // A script that uninstalls the dependencies for the sensor
./poll.py // A script that knows how to poll the sensor and returns a value
info.json example:
{
"id": "grove_dht"
"name": "Grove Temperature",
"poll": {
"command": "python ./poll.py",
"parameters": [ // A description of what parameters can be fed to the poll command
"port": {
"type": "Integer",
"range": [1, 4]
}
]
},
"measurementUnits": "Celsius",
"miniumumPollFrequency": 2000, // Some sensors can't be polled quickly, this is a safety measure for Pipes that are misconfigured
}
Reservoir Plugin architecture
A Reservoir Plugin is a folder in the ./Reservoirs folder with the following structure.
./info.json // A file with a info all about this Reservoir
./install.sh // A script that installs the dependencies for the Reservoir
./uninstall.sh // A script that uninstalls the dependencies for the Reservoir
./save.py // A script that knows how to save data into a Reservoir
info.json example:
{
"id": "dat",
"save": {
"command": "save.py",
"parameters": {
"location": {
"type": "String",
}
}
}
"miniumumSaveFrequency": 2000
}
Pipe architecture
The Pipe Engine takes directions from a settings.json that describes the Pipes.
{
"pipes": [
{
"id": "738fs9fsejs9h3hdkfs88sfn",
"name": "first pipe",
"frequency": 5000, // in milliseconds
"sensor": {
"id": "temper1",
"parameters": [ "4" ]
}
"reservoir": {
"id": "dat",
"parameters": {
"location": "~/dat-repo-2"
}
}
},
{
"id": "sdfJDFI87udfdf98dfdj90sk",
"name": "second pipe",
"frequency": 60000, // in milliseconds
"sensor": {
"id": "grove_dht",
"parameters": [ "1" ]
}
"reservoir": {
"id": "dat",
"parameters": {
"location": "~/dat-repo-2"
}
}
}
]
}
Questions and next steps
I'm wondering how this looks to everyone. In particular, I'm interested in getting feedback from people who would be interested in writing Sensor Plugins. Does the proposed architecture look easy enough for you? Can we think of a way to make it even easier?
Why I'm interested
I hope that if we make a Pipe Engine that is easy enough for developers to write plugins for, when they are next building a device that polls a sensor for data and saves it somewhere, they'll take advantage of the framework because it saves them time from having to write the equivalent Sensor or Reservoir plugin in their own custom code. When a developer does write a Sensor and/or Reservoir plugin, I hope they'll share that back with the community thus help to create an extensive library of Sensor and Reservoir plugins. This could save all of from having to write a ton of glue code every time we need to get data from a sensor and save it somewhere!
5 Comments
I love the idea of language neutral plugins and process invocation that plays off the convenience and power of a Linux command line environment. As we chatted about, it is important to sandbox potentially buggy code that interacts with unpredictable hardware. One issue for example is a hung process that does not reply - a "clogged pipe" - there needs to be some [optional]? timeout mechanism - an automated plumber to fix the leak (invoke a reset process) or seal it off (issue an exception). Maybe the exceptions need be sent through the associated pipe or a special garbage chute (logging channel/email or text notification)?
Could we use the term "period" instead of "frequency" which has an inverse time association? Also why not provide human friendly types and units of times like floating point seconds, minutes, hours, days? What about annotating periods as in "1.0s", "1.0m", "1.0h", "1.0d"? Check variable names with "miniumum [sic]".
Where do those nice long hashy pipeIDs come from?
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
Some more thoughts about time: managing it in a coherent way will be a major benefit of using the platform.
As you probably know most Arduino boards and even the Raspberry Pi don't have a real time clock (which leads to hilarity when time stamping data entries off the grid). Internet connected systems can usually sync their clocks with a time server, but what about outages and off the grid local networks? Maybe we can include an RTC with an RPi based kit or as an option.
Especially in cases with remote reservoirs, we should be caching at least two timestamps: entering or leaving the pipe. Another feature that I've (pipe) dreamed about is the ability to send timely (and back-dated) annotations into the flow using a mobile device browser interface - some sort of markdown would be sweet. This want was inspired by experiments to manipulate the conditions of an indoor environment and the need to somehow document the crazy things we were trying so it could be correlated with the sensor time series.
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
I created a Wiki page to reflect our most recent ideas, feel free to edit it.
Good point. Perhaps a
timeoutproperty on Sensors and Reservoirs that the Engine can use as a guideline of when to slap around the Sensors/Reservoirs.Fixed.
Sounds good if we can figure out how to make that easy for the user. My fear would be that we tell them they can do something like "5m 31seconds" and then they do "1hour 5seconds" in some combination we didn't expect and it breaks the program. I suppose that's our challenge for this though, make it as easy as possible so supporting any kind of expression of time might be something we should work on.
I was thinking a computer would write that JSON file and create a GUID for each entry, but users may just do it themselves and need to remember to put a locally unique ID in there.
In the past I've used the RasClock module. It occurs to me now the GrovePi board and the RasClock board may mess with each other, something to look into but not a problem for Pipe devices getting data from Arduinos over USB serial.
Sounds cool. In that case the sensor is the human in the Pipe :). But maybe we don't actually program it that way.
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
I haven't much knowledge of the "framework: to understand this thread - other than I develop sensors and then visualize the readings via the cloud.
I would ask of the framework:
how can it be made into a server for sensors?
how do autonomous sensor sample events get processed? - both from autonomous sampling, autonomous data sheet registry - and provisioning feedback to autonomous sensors.
For the modern sensor, M2M/IoT, it is independentky powered entity with power management as a critical design characteristic - it typically powers up, push its sample onto a pervasive internet via a local (wireless) gateway - to be eventually accessed and visualized. In between sampling events its critical to minimize power and shutdown - ie not listen.
For one product to do this I used an XBP-gateway, that created a tcp/ip tunnel through the internet to a server. That is the gateway XBP was always on connected to a cloud server. When the XBP-sensor wakes up, it broadcasts its sample packet, that travels through the tunnel to the cloud server, its entered into a MySql database, and then an ack travels back down the tunnel to the XBP-gateway that broadcasts to the XBP-sensor, which then marks the sample internall as 'delivered'. All usually within 1 second.
So manage its available power the XBP-sensor does the following, if it can't detect the XBP-gateway within a second it turns off, and if there isn't a response from the server withing 4 seconds it turns off. It retries every 15minutes.
Is this a question? Click here to post it to the Questions page.
Reply to this comment...
Log in to comment
Hi everyone -
I posted a big update on the Open Pipe Kit project: http://publiclab.org/notes/rjstatic/07-21-2015/does-the-open-pipe-kit-lower-the-barrier-to-sensor-data-collection-we-need-your-help
@neilh20 This was the part of the Minimum Viable Product goals of the Apitronics Wireless Platform that I helped build the Hive Gateway software for.
Check out the video on the Kickstarter. While the Open Hardware Bee board is no longer for sale, the Open Source Hive Gateway software could be used with any hardware willing to talk Xbee to the Hive.
In contrast, the Open Pipe Kit project has taken an approach to not optimize for power on our first go around, instead optimizing for other factors. See my most recent Research Note I linked to above.
Reply to this comment...
Log in to comment
Login to comment.