Twitter JSON buffer

[DynamoBen]

After discovering this I decided I wanted to have the spinneret as the core for my version of this project instead of the Arduino. I've begun porting the Arduino code to Spin and am able to request twitter searches via GET commands. Now I'm working on parsing the JSON responses, but I'm stuck on something. The W5100 driver is set up to copy everything that is currently in the W5100 to a local buffer, and the twitter JSON response is sent in multiple 1024 byte chunks (could be 12K of data).

Does it make sense to copy all of the JSON chunks into a local buffer and then parse it (I would need a huge buffer) or should I add to the W5100 driver so I can read data from the W5100 buffer as I'm parsing it (smaller buffer, but data might came in faster than I can parse)?

[Mike G]

How about trying different buffer sizes to determine which works best for your project.

[DynamoBen]

How about trying different buffer sizes to determine which works best for your project.

That's likely what I will end up doing. The reason for the question was to see whether having a huge local buffer was a crazy idea, and/or to see if SPIN could get through the data fast enough to be able to parse on the fly. If no one has any specific opinions then I will forge ahead with trial and error.

[Mike G]

It's bit a hard to provide guidance without seeing your source code or objects.

I never go over a 2k buffer. I've written well over 100k to the Spinneret. I can even download videos.

[DynamoBen]

It's bit a hard to provide guidance without seeing your source code or objects.

I never go over a 2k buffer. I've written well over 100k to the Spinneret. I can even download videos.

Here you go, all that is happening at the moment is I'm sending a GET request and dumping the reply to the serial terminal. This was done to confirm that I could receive data from the server. I'm interested in your strategy for consuming that much data.

[Mike G]

http://spinneret.servebeer.com:5000/advanced/upload.htm

Get your parsing utility working first. Then you should have a good idea what ya need.

You might need a large buffer or maybe several small buffers. How fast is that printer?

[DynamoBen]

Get your parsing utility working first. Then you should have a good idea what ya need. You might need a large buffer or maybe several small buffers.

I will get it parsing a single response, who knows maybe I'm being pessimistic about how fast I can get through the data.

How fast is that printer?

Unsure, I wanted to get everything working via PST before I invested in the printer hardware.

[DynamoBen]

This is the project I'm trying to port, wish my C skills were stronger. I suspect the "Stream" class is what is allowing them to parse through the data as it is coming in, but for life of me I can't figure out how "Stream" works from a code perspective. https://github.com/adafruit/Adafruit.../Gutenbird.ino

[Mike G]

A stream is an array of bytes. Since you are dealing with JSON, you'll be looking for these characters plus whatever field names in the stream.

 { } : " , []

Ya just need a string parser. Take a look at the OBEX objects that handle strings.

[DynamoBen]

A stream is an array of bytes. Since you are dealing with JSON, you'll be looking for these characters plus whatever field names in the stream.

 { } : " , []

What I meant was that the source project uses the Wiznet chip also, but has a lot less flash availabe in the uC. They use a "Stream" class to get data via ethernet, I was hoping find some tips or tricks in their code.

[Mike G]

A stream is an array of bytes in memory. A stream object provides methods common when dealing with a streams.

A stream looks like -> This is a stream!

[DynamoBen]

A stream is an array of bytes in memory. A stream object provides methods common when dealing with a streams.

A stream looks like -> This is a stream!

I think the low bandwidth of the forum is confusing my point. The Arduino (slower processor, less memory) is able to consume and parse 12K of JSON data via the Wiznet W5100. The code does this without a huge local buffer, which means the Propeller (much faster, a lot more memory) should be able to do the same. I'm reading through all of the Arduino source code, including the "Stream" base class, to better understand how they are doing it.

[Mike G]

DynamoBen, you are missing the point - you can't see the forest through the tress.

I browsed the Arduino code and it's not doing anything different then the Spinneret code that has already been written. However, you must look at the implementation.

You are already capturing the response stream and streaming it to the PST, for crying out loud.

What you don't have is a JSON parser.

Have you taken a look at the tutorial? The source has all kinds of parsing routines.
http://spinneret.servebeer.com:5000/

[DynamoBen]

DynamoBen, you are missing the point - you can't see the forest through the tress.

I browsed the Arduino code and it's not doing anything different then the Spinneret code that has already been written. However, you must look at the implementation.

You are already capturing the response stream and streaming it to the PST, for crying out loud.

What you don't have is a JSON parser.

Have you taken a look at the tutorial? The source has all kinds of parsing routines.
http://spinneret.servebeer.com:5000/

You are right, I'm likely making this harder than it needs to be. Let me double back and reread the resources and suggestions you've given thus far and give it another go.