+ Reply to Thread
Results 1 to 14 of 14

Thread: Twitter JSON buffer

  1. #1

    Default Twitter JSON buffer

    After discovering this I decided I wanted to have the spinneret as the core for my version of this project instead of the Arduino. I've begun porting the Arduino code to Spin and am able to request twitter searches via GET commands. Now I'm working on parsing the JSON responses, but I'm stuck on something. The W5100 driver is set up to copy everything that is currently in the W5100 to a local buffer, and the twitter JSON response is sent in multiple 1024 byte chunks (could be 12K of data).

    Does it make sense to copy all of the JSON chunks into a local buffer and then parse it (I would need a huge buffer) or should I add to the W5100 driver so I can read data from the W5100 buffer as I'm parsing it (smaller buffer, but data might came in faster than I can parse)?

  2. #2

    Default Re: Twitter JSON buffer

    How about trying different buffer sizes to determine which works best for your project.

  3. #3

    Default Re: Twitter JSON buffer

    Quote Originally Posted by Mike G View Post
    How about trying different buffer sizes to determine which works best for your project.
    That's likely what I will end up doing. The reason for the question was to see whether having a huge local buffer was a crazy idea, and/or to see if SPIN could get through the data fast enough to be able to parse on the fly. If no one has any specific opinions then I will forge ahead with trial and error.

  4. #4

    Default Re: Twitter JSON buffer

    It's bit a hard to provide guidance without seeing your source code or objects.

    I never go over a 2k buffer. I've written well over 100k to the Spinneret. I can even download videos.

  5. #5

    Default Re: Twitter JSON buffer

    Quote Originally Posted by Mike G View Post
    It's bit a hard to provide guidance without seeing your source code or objects.

    I never go over a 2k buffer. I've written well over 100k to the Spinneret. I can even download videos.
    Here you go, all that is happening at the moment is I'm sending a GET request and dumping the reply to the serial terminal. This was done to confirm that I could receive data from the server. I'm interested in your strategy for consuming that much data.
    Attached Files Attached Files

  6. #6

    Default Re: Twitter JSON buffer

    http://spinneret.servebeer.com:5000/advanced/upload.htm

    Get your parsing utility working first. Then you should have a good idea what ya need.

    You might need a large buffer or maybe several small buffers. How fast is that printer?

  7. #7

    Default Re: Twitter JSON buffer

    Quote Originally Posted by Mike G View Post
    Get your parsing utility working first. Then you should have a good idea what ya need. You might need a large buffer or maybe several small buffers.
    I will get it parsing a single response, who knows maybe I'm being pessimistic about how fast I can get through the data.

    How fast is that printer?
    Unsure, I wanted to get everything working via PST before I invested in the printer hardware.
    Last edited by DynamoBen; 03-14-2012 at 10:54 PM.

  8. #8

    Default Re: Twitter JSON buffer

    This is the project I'm trying to port, wish my C skills were stronger. I suspect the "Stream" class is what is allowing them to parse through the data as it is coming in, but for life of me I can't figure out how "Stream" works from a code perspective. https://github.com/adafruit/Adafruit.../Gutenbird.ino
    Last edited by DynamoBen; 03-15-2012 at 11:05 PM.

  9. #9

    Default Re: Twitter JSON buffer

    A stream is an array of bytes. Since you are dealing with JSON, you'll be looking for these characters plus whatever field names in the stream.
    Code:
     { } : " , []
    Ya just need a string parser. Take a look at the OBEX objects that handle strings.

  10. #10

    Default Re: Twitter JSON buffer

    Quote Originally Posted by Mike G View Post
    A stream is an array of bytes. Since you are dealing with JSON, you'll be looking for these characters plus whatever field names in the stream.
    Code:
     { } : " , []
    What I meant was that the source project uses the Wiznet chip also, but has a lot less flash availabe in the uC. They use a "Stream" class to get data via ethernet, I was hoping find some tips or tricks in their code.

  11. #11

    Default Re: Twitter JSON buffer

    A stream is an array of bytes in memory. A stream object provides methods common when dealing with a streams.

    A stream looks like -> This is a stream!

  12. #12

    Default Re: Twitter JSON buffer

    Quote Originally Posted by Mike G View Post
    A stream is an array of bytes in memory. A stream object provides methods common when dealing with a streams.

    A stream looks like -> This is a stream!
    I think the low bandwidth of the forum is confusing my point. The Arduino (slower processor, less memory) is able to consume and parse 12K of JSON data via the Wiznet W5100. The code does this without a huge local buffer, which means the Propeller (much faster, a lot more memory) should be able to do the same. I'm reading through all of the Arduino source code, including the "Stream" base class, to better understand how they are doing it.
    Last edited by DynamoBen; 03-16-2012 at 06:34 PM.

  13. #13

    Default Re: Twitter JSON buffer

    DynamoBen, you are missing the point - you can't see the forest through the tress.

    I browsed the Arduino code and it's not doing anything different then the Spinneret code that has already been written. However, you must look at the implementation.

    You are already capturing the response stream and streaming it to the PST, for crying out loud.

    What you don't have is a JSON parser.

    Have you taken a look at the tutorial? The source has all kinds of parsing routines.
    http://spinneret.servebeer.com:5000/

  14. #14

    Default Re: Twitter JSON buffer

    Quote Originally Posted by Mike G View Post
    DynamoBen, you are missing the point - you can't see the forest through the tress.

    I browsed the Arduino code and it's not doing anything different then the Spinneret code that has already been written. However, you must look at the implementation.

    You are already capturing the response stream and streaming it to the PST, for crying out loud.

    What you don't have is a JSON parser.

    Have you taken a look at the tutorial? The source has all kinds of parsing routines.
    http://spinneret.servebeer.com:5000/
    You are right, I'm likely making this harder than it needs to be. Let me double back and reread the resources and suggestions you've given thus far and give it another go.

+ Reply to Thread

Similar Threads

  1. ybox2 twitter client
    By Fjornir in forum Propeller 1 Multicore Microcontroller
    Replies: 0
    Last Post: 02-26-2011, 06:19 AM
  2. Follow Parallax Inc. on Twitter!
    By Jen J. in forum Announcements
    Replies: 0
    Last Post: 02-03-2009, 02:21 AM
  3. sd card buffer
    By Owen in forum Propeller 1 Multicore Microcontroller
    Replies: 3
    Last Post: 01-13-2008, 12:00 PM
  4. serial buffer?
    By Archiver in forum BASIC Stamps Archive
    Replies: 2
    Last Post: 06-20-2004, 09:09 AM
  5. Oh my buffer... where are you ????
    By Archiver in forum BASIC Stamps Archive
    Replies: 3
    Last Post: 01-26-2002, 04:35 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts