Query a Webpage

[jjme88]

Hi,

I know and have used Excle's Web Query tool to set up and use a webpage as a data source. But is it possible to do something similar with asp.net?

if any of you have any ideas I would love to know your thoughts?

jj

 

[jmeckley]

while this might be possible with excel it's a gross misuse of the platform. A webpage is html that is meant for human readability (presentation). it is not a data source.

a webpage can serve data to a client with web services via xml but these are not webpages. its data delivered via a web request. you could have a webpage render the xml in html, but this is working harder not smarter.

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[JoeAtWork]

jmeckley - can't agree with you on that. If something has the data you need, then it can be your data source. Just because it was meant for human eyes doesn't mean it can't be gathered programmatically - sometimes referred to as "screen scraping".

After all, where do you think Google gets it's data?

Joe Schwarz
Custom Software Developer

http://www.plandev.net

 

[jjme88]

my issue is that for a research project i am working on there are a number of websites that publish data via there websites in a variety of tabular forms. I need to gather this data from those websites. Now while Excels web query tool works a treat for this I need to replicate this functionality in an asp.net application.

I am aware that there a a raft of software apps you can buy to do this but I need somethnig more bespoke and integrated into my application so was hoping someone could give me some help on this.

JoeAtWork I agree with your comments!
jj

 

[jmeckley]

than you need to screen scrape. to screen scrape you need to make web request to the page and then use regex to parse the data from the response. You might be able to use Waitn to parse the html too.



Jason Meckley
Programmer
Specialty Bakers, Inc.