[Geojson] Binary GeoJSON

[Reed Underwood]

Thanks for this, Arlo!

At present, I'm using JSON + gzip to store attribute sets (or "documents", in the DB sense - previously, serialized and compressed key-value pairs, scalars only). It seems to be more than adequate. But I'm also serializing other data in different formats (standard binary ones). My thinking was that it might be nice if there was a standard way to bundle all this together.

However, over the last couple of days, conversation here and my own work on the system I'm implementing have converged to push me away from this sort of thing. I'm getting more than adequate performance and gain little from a binary format for nested objects.

Thanks for the input. I suppose I'll stick with what I'm using now. If it worked for you guys, I imagine it'll be fine for me. And it makes my life markedly easier!

Cheers,
R.

On Fri, Feb 03, 2012 at 05:35:40PM +0000, Arlo Belshee wrote:
> A thing to keep in mind: why are you doing this, and will your implementation get the results you're after. I mention this because we just went through a similar process with OData, and decided not to do binary (just gzip the text Json instead, and clean up the text Json representation).
> 
> Your goals may be very different from ours, but make sure that you know what they are and that you keep them clearly in mind. There are downsides to multiple formats (fragmenting the ecosystem), and you don't want to take them on unless you're also gaining something important.
> 
> With OData, our goals were:
> 
> * Not sacrifice readability; improve it where we can
> * Decrease power consumption and latency for consuming OData payloads from mobile devices
> * Still support the wide ecosystem of clients and servers (either endian-ness, all native bit sizes from 8 to 64)
> 
> The third point prevents assuming any of the really fast binary parsing techniques (such as memory overlays). No matter what binary rep we take, a big chunk of the market will not be able to parse it by direct memory mapping.
> 
> Given that, we then measured the CPU usage and wall-clock time to parse a binary format that was not memory mapable, vs the same data in textual Json. We also wrote a new textual Json format, where we eliminated duplication and redundant metadata. Both the binary and the new textual contained the same information.
> 
> In our case, we found that the textual one was about 10% slower than the binary, both in CPU usage and wall-clock time. The old Json was 100% slower than our new "light" format, and the old Atom format was 250% slower than the light Json.
> 
> Thus, going to binary would have gained us a small advantage over just making a better Json format, but not enough to be worth decreasing readability. The light Json format reduced battery usage by 50% from our previous fastest; getting it up to a 55% improvement just isn't important enough.
> 
> For latency, all that matters is the number of bits sent to the client. Really, all that matters is how long it takes a cell phone to do TCP negotiation, but after that it's bit count. The textual Json zipped to within 2% of the size of the zipped binary. So they were identical (within variability of measurement) for download time.
> 
> Thus, our solution to binary: lighter textual Json + gzip.
> 
> Not saying that this will be your best outcome. Just saying that we entered the process assuming that we were going to do an efficient binary format. Fortunately, we clarified the goal to be efficient format, binary or otherwise, as measured by latency and battery usage. That let us find that our best option wasn't actually binary at all.
> 
> Arlo
> 
> -----Original Message-----
> From: geojson-bounces at lists.geojson.org [mailto:geojson-bounces at lists.geojson.org] On Behalf Of Reed Underwood
> Sent: Thursday, February 02, 2012 5:38 PM
> To: geojson at lists.geojson.org
> Subject: [Geojson] Binary GeoJSON
> 
> Hello,
> 
> I'm interested in developing a spec for a binary equivalent to GeoJSON. If there are efforts along these lines already in motion, please let me know. If you're interested in collaborating on something like this, please let me know.
> 
> I plan on building a model implementation in C and would like to move on this sooner rather than later.
> 
> Cheers,
> Reed Underwood
> Boston University
> _______________________________________________
> Geojson mailing list
> Geojson at lists.geojson.org
> http://lists.geojson.org/listinfo.cgi/geojson-geojson.org
> 
> 
> 
> 
> 
> _______________________________________________
> Geojson mailing list
> Geojson at lists.geojson.org
> http://lists.geojson.org/listinfo.cgi/geojson-geojson.org