While I was on the train coming home from work an interesting idea hit me while I was trying to solve a very common problem. In general, the problem is that the web sucks. But, I think I found an interesting solution.
The Problem
Your standard web protocols are pretty limiting in what kind of things you can do with your clients without ‘cheating’. The core issue is that the internet is great for GETTING and PULLING data but it becomes a real pain when you’re trying to PUSH or PUT data to something else. For example, without some AJAX magic it’s impossible to push data to the user’s browser without them asking for it first. This same problem exists between two separately owned and managed webservices. Needless to say, this limitation frustrated me.
So let me explain the problem I am facing. So let’s say we have a server that is updated quite frequently. Let’s say every 2 minutes there is some new piece of data that is relevant to me and I want to know about it when it is available. The stock market is a good example of this. There are a handful of webservices that let you ask for the current price of a stock. That’s great if you’re not making any time sensitive decisions based on this information changing. But, we know the stock market changes fast. By the time you ask for some information your data could already be out of date because something else has happened to it. To know about it you would have to make another request. You can see how to be updated you would need to put yourself in an endless loop of requests to stay up to date.
Luckily, a very awesome design pattern called the Observer Pattern prevents us from having to do this. Instead of asking for data, you get pushed the data when a certain event has occurred. Brilliant right? This is exactly what I needed… but wait a minute. This pattern only works great when certain means of communication have been established and when everything resides within the same application. What I really want is a web service that I have no control over to call me and push me data when it has something. Sounds a bit selfish, no? Yeah…
So, this is when I arrived at the conclusion that there has to be some sort of way to have a distributed observer pattern. A way to subscribe to events in a loosely coupled way. This is what I’ve been playing with for the past couple of days.
My Solution
What if I created a web service where anyone could register as a listener by providing a RESTful URL? Let me explain using the stock market example again.
Let’s say we have a stock exchange server that has the most recent stock information and it exposes a web service method for people to register as listeners. The signature of this method might look something like this:
RegisterListener(string tickerSymbol, string responseURL, enum format);
tickerSymbol : The four letter name of the stock you are interested in.
responseURL : The URL that the service would actually POST data to when there is new info.
format: Specifies if you want the data in the xml or json format.
Lets say I pass in this data to the method:
tickerSymbol: ‘GOOG’
responseURL: ‘http://www.vyrotek.com/service/stockinfo‘
format: XML
Now whenever the stock server has new information about the Google stock it can pull all the URLs that are registered to that stock and POST some XML data to the URL I provided. This might sound like a bit of work for the server to do but it actually cuts down on the amount of work both the server and client if the client was constantly requesting the server for new info instead.
Random Thoughts
I think this is a pretty different way to look at data aggregation as well. There are lots of sites that aim to bring together all your content from various websites you visit all into a single place. But many of the services just mash up various RSS feeds and depend on requests being made over and over to get the latest. Perhaps this could address the ownership of data as well. A service could be created that acts as a central storage point for all your ’subscriptions’ and it would provide the data in any form you need it in. Allowing you to create applications on top of that. Now you’re only querying your own service that is full of new data instead of a handful of services that may not have anything new at all.
Has anyone ever seen something like this implemented or used? I’m sure there are other solutions to the problem, but this one is definitely interesting to think about. What do you guys think?
(Updated to rephrase some things)
If you like this blog please take a second and subscribe to my rss feed
Tags: ajax, design patterns, observer pattern, POST, web service
Comments: 15 comments
All the fields that are marked with REQ must be filled
ryan
July 15th, 2008 at 10:02 pm
This has some merit but it doesn’t really solve your initial problem about updating browsers.
Some ideas I have played with are long lived web services and streaming HTTP. In the first, I call the web service asynchronously, and when I get a response I immediately call it again. The next time the service has to be smart enough to know who I am and what it gave me the last time I called it - so it holds back on responding until it has data to send. I had some issues with how to block and had to move on before I could do too much with it. In the other idea I would write to the HTTP stream over time and constantly flush so the data would get pushed out. It was successful in pushing data to a web page without using AJAX but I never really played with it too much either.
Maybe someone just needs to develop a browser that works more as a two way communication device.
Jason
July 15th, 2008 at 10:30 pm
Ryan, I definitely considered that sort of solution but keeping requests alive really hurts webserver performance.
Pushing data to a webbrowser isnt really a huge concern. I’m mostly looking for a way to actively respond to new data. For example, maybe I want to send an SMS to a user when a certain stock has gone too low. It’s almost like email. You’re having data sent to your servers for you to pick it up later rather than trying to request it from a server that isnt yours. You’re basically signing up for a newsletter of new content. But the content is in a format that can be worked with.
Pushing data could really change the way we ’subscribe’ to things. You would basically have a new kind of inbox that would just hold new website content. It could make downloading content for offline viewing much easier as well.
Maybe it should be called PushRSS (pronouced Pushers?, heh)
Jason
July 16th, 2008 at 7:22 pm
Check out this article. It’s really the only other resource I could find on the subject. There are some interesting comments about the business value of such a pattern.
They describe this concept as “A resource subscribes to a peer resource via a GET that supplies its own URI, and is notified of subsequent state changes in that resource through a POST back”
http://duncan-cragg.org/blog/post/distributed-observer-pattern-rest-dialogues/
josh
July 18th, 2008 at 12:21 am
Quite possible the inverse of how we currently think of services. My concerns are primarily involved in the implementation of such a method.
How often does the service manage these references?
How efficient are the executions - are they asynchronous?
Jason
July 18th, 2008 at 7:23 am
Some new methods on the service doing the pushing could help with managing the references. Of course you would have a RemoveListener method to remove a listener. Each url reference would be stored in a database and would be pulled up only when an event has occured that is relative to whatever was registered. Like with the google example, once the service detects a change in stock it queries its own DB and pulls up all the urls that want to know about GOOG and now fires off POSTs to all those urls.
As for how efficient it would be. I would imagine it would be pretty dang quick. Since you dont care about the response that comes back from all the URLs when you POST some new data to it. Its up to the receiver to make sure they are available for the POST. I would also imagine that there would be some preset timeouts if certain destinations seem to be unreachable.
This model isnt really like your typical Async call to a webmethod. You register once with the webservice and then you’re registered forever to anything that happens to GOOG. You just teach the webservice where to send you the very delayed response of data. But now the service can respond/push multiple times for the next million years and you only ever had to call them once!
josh
July 18th, 2008 at 8:47 am
So, from the core service’s perspective, you simply retrieve the collection of ‘listeners’ fire off the POSTs? This is where I was wondering about the async processes to make sure your service doesn’t get bogged down if a URL no longer exists. I like the idea of having preset timeouts for these situations.
Hurry up and post a demo!
ryan
July 18th, 2008 at 9:40 am
My only issue is that this is useful primarily between services. It would REALLY be useful to discover a way to do this without requiring the end users to have a web server running. I wrote a lot of distributed apps where I have to poll for data and a push model would be really nice but there isn’t anything out there that does it in a security friendly manner.
Jonathan Ross
July 18th, 2008 at 9:55 am
I distinctly remember doing push behavior using WCF. I am not sure how it would work with AJAX or a browser, but it is something that should be looked into if you want that type of functionality. It was called a callback contract. Take a look at that.
ryan
July 18th, 2008 at 10:10 am
I think that model uses WSDualHTTPBinding which is actually very similar to what Jason proposed, although I think it would only work between a server and client that were both using WCF. Maybe I’m wrong about that though.
Too bad the server side of things is Tomcat in my situation :(
Jason
July 18th, 2008 at 10:45 am
Ok here is a REALLY quick example. I setup a subscriber on http://www.sideclicker.com There you can see all the data that has been pushed to the url http://www.sideclicker.com/listener.ashx.
If you want to play with it, open the example solution and run the Pusher web application. You’ll a site that lets you add subscribers and the ability to push data to those urls. If you enter http://www.sideclicker.com/listener.ashx as a subscriber and push data to it you will see it at http://www.sideclicker.com
http://www.sideclicker.com/example.zip
Note: The example is a bit backwards since I cant have sideclicker be the pusher since it cant push data to your individual PC’s. I’ll start on an winapp version that will have timed pushes. Hopefully that will illustrate it better.
josh
July 18th, 2008 at 11:57 am
Sounds like Jason is proposing more of a client/server solution in which the client is a service. So Distributed Observer Pattern is like an SOA Observer, no?
Whereas Ryan, it sounds like you’re talking about functionality like appending ‘callback’ parameters to JSON-based services to update the browser - only you want it to happen dynamically.
Given the stateless nature of requests/responses, I’m not so sure that this is truly possible? Cached services would be an efficient way to mimic it I suppose.
Jason
July 18th, 2008 at 12:12 pm
Thats exactly right Josh. Live UI on browsers is such an icky subject. My focus was more SOA oriented from service to service. There are many advantages to using this pattern that dont involve the browser.
Check out my latest blog entry on my site to see how this pattern could be used: http://www.vyrotek.com/2008/pushing-some-new-ideas/
Bek
July 18th, 2008 at 3:09 pm
While I was working at Extend, I had done some research on pushing option for one of our services. It sounds like what you want to do is to implement observer pattern using independent services(one of them will act as Subject, the rest are observers).
Duplex messaging topology(encapsulated in wsDualHttpBinding) is a perfect match for this kind of situation. It is also interoperable with our technologies. You will have a Subject Service and multiple Services subscribe to it using Callback methods(you cache OperationContext.Current in WCF). Then whenever your subject has an update available it will iterate through the observers list and call an appropriate method. Since this technology uses two one way methods you don’t need to keep the request alive.
One problem, however, is that your callback contract is determined by Subject…and other services might have to change if the service versions. Also, you will need to use whatever binding Subject service chooses. If this is not good enough you could make it so Subject and Observers use only one way methods without duplex binding. That way they are truly independent of each other. However, then subject needs to take care of finding the callback channel on its own which will require some addition code.
This Juval Lowy’s article a cool way of implementing Subsribe/Unscubscribe mechanism: http://msdn.microsoft.com/en-us/magazine/cc163537.aspx
Bek
July 18th, 2008 at 3:15 pm
damn it, wordpress ate my line breaks…
Jason
July 29th, 2008 at 8:37 am
So I learned what this model/pattern is called. ‘WebHooks’ Check out http://webhooks.pbwiki.com/ for more info.
Leave a reply