I'd love to poll a web page that has constantly updating dynamic data, and grab that data. I don't want to reload the entire page every second. Is there way to grab that data without reloading the page?
I'm coding as a console app for demo purposes.
Not really. For starters, HTTP is stateless. When you perform an HTTP request, you'll always get static data back, even if dynamic routines were used to generate it.
Doesn't sound like you can hit the data used to construct the page in any other way ( i.e. direct link into DB, Web service, etc ) - if you could you wouldn't be scraping web pages.
Even if you could do that, wouldn't make much of a difference. You're still going to have to compare what you have now with the data the remote service has. Absent some sort of push notification, you're polling, and if you're polling, yes, you have to check all the time.
You could use the header data returned to check the last modified date:
...however, it's completely up to the server to set this date -- so it may or may not be accurate.
Another option to prevent manual scraping may be to simply get a checksum on the data returned. You could store it off and compare the markup's sum to what you have stored.
You might consider writing the application an an HTML Application (HTA).
This gives you full browser capability, plus the ability to hook into events, and display the data in another window, or write to a disk file.