Twitter JSON buffer
DynamoBen
Posts: 366
After discovering this I decided I wanted to have the spinneret as the core for my version of this project instead of the Arduino. I've begun porting the Arduino code to Spin and am able to request twitter searches via GET commands. Now I'm working on parsing the JSON responses, but I'm stuck on something. The W5100 driver is set up to copy everything that is currently in the W5100 to a local buffer, and the twitter JSON response is sent in multiple 1024 byte chunks (could be 12K of data).
Does it make sense to copy all of the JSON chunks into a local buffer and then parse it (I would need a huge buffer) or should I add to the W5100 driver so I can read data from the W5100 buffer as I'm parsing it (smaller buffer, but data might came in faster than I can parse)?
Does it make sense to copy all of the JSON chunks into a local buffer and then parse it (I would need a huge buffer) or should I add to the W5100 driver so I can read data from the W5100 buffer as I'm parsing it (smaller buffer, but data might came in faster than I can parse)?
Comments
That's likely what I will end up doing. The reason for the question was to see whether having a huge local buffer was a crazy idea, and/or to see if SPIN could get through the data fast enough to be able to parse on the fly. If no one has any specific opinions then I will forge ahead with trial and error.
I never go over a 2k buffer. I've written well over 100k to the Spinneret. I can even download videos.
Here you go, all that is happening at the moment is I'm sending a GET request and dumping the reply to the serial terminal. This was done to confirm that I could receive data from the server. I'm interested in your strategy for consuming that much data.
Get your parsing utility working first. Then you should have a good idea what ya need.
You might need a large buffer or maybe several small buffers. How fast is that printer?
I will get it parsing a single response, who knows maybe I'm being pessimistic about how fast I can get through the data.
Unsure, I wanted to get everything working via PST before I invested in the printer hardware.
{ } : " , []Ya just need a string parser. Take a look at the OBEX objects that handle strings.
What I meant was that the source project uses the Wiznet chip also, but has a lot less flash availabe in the uC. They use a "Stream" class to get data via ethernet, I was hoping find some tips or tricks in their code.
A stream looks like -> This is a stream!
I think the low bandwidth of the forum is confusing my point. The Arduino (slower processor, less memory) is able to consume and parse 12K of JSON data via the Wiznet W5100. The code does this without a huge local buffer, which means the Propeller (much faster, a lot more memory) should be able to do the same. I'm reading through all of the Arduino source code, including the "Stream" base class, to better understand how they are doing it.
I browsed the Arduino code and it's not doing anything different then the Spinneret code that has already been written. However, you must look at the implementation.
You are already capturing the response stream and streaming it to the PST, for crying out loud.
What you don't have is a JSON parser.
Have you taken a look at the tutorial? The source has all kinds of parsing routines.
http://spinneret.servebeer.com:5000/
You are right, I'm likely making this harder than it needs to be. Let me double back and reread the resources and suggestions you've given thus far and give it another go.