need advice for web page scanning database application 1

[jesse77]

hello, I'm new. I need advice from someone patient. I just need to know where to get started if I want to code an application that scans webpages and compiles a database with the information. specifically, I want to scan stock trading forums for ticker symbols.

I'm a newbie, sure, but I'm intelligent and have a lot of time on my hands to learn how to do this. what I need is a kind person to point me in the right direction, maybe what code I should focus on, a key book to read before beginning, etc. that is all I ask... no technical questions. I will RTFM when I know which one to open.

thanks!

 

[ramani]

HI jesse

Baltman shows in the following thread, how to read this forum's FAQ section. You can get a start with that..

How can I navigate through all these Tek-Tips FAQs?
thread184-687849



ramani
(Subramanian.G)

http://winnersoft.coolfreepages.com/

 

[jimoo]

It sounds like baltman's FIVE STAR FAQ is what you need.

Check it out.

baltman (TechnicalUser) Oct 23, 2003
How can I navigate through all these FAQs?
FAQ184-4339


Jim Osieczonek
Delta Business Group, LLC

http://www.deltabg.com

 

[jimoo]

Sorry for the duplicate reply ramani, we must be logged in at the same time and you have faster fingers

Jim Osieczonek
Delta Business Group, LLC

http://www.deltabg.com

 

[jesse77]

all I found was a search page, and folks, I have no idea what to search for! I don't know what language or even what words to try. I tried "crawl" and "crawler" and "webpage crawler programming" to no avail.

I'll go back and keep trying, but I still need some pointers.

 

[jimoo]

Here is a snippet of baltman's code. I searched google to find a random stock page. Try this, and it should at least bring data back to you. You can refer to balman's original code to do more.

PUBLIC gcVarFAQArea, gcFilterField, gcFilterValue, gcThisTitle


WAIT WINDOW AT SROWS()/2, SCOLS()/2 "Getting Executive Orders..." nowait
lcURL="

http://finance.yahoo.com/q?s=msft"

objHTTP = CreateObject("MSXML2.XMLHTTP&quot
objHTTP.Open("GET", lcURL, .f.)
objHTTP.Send
FAQs=(objHTTP.ResponseText)

* MESSAGEBOX(FAQS)


WAIT WINDOW AT SROWS()/2, SCOLS()/2 "Organizing Data" nowait

VarFAQRating=""

CREATE CURSOR MyCursor (myText C(250))
Y = ALINES(FAQ_Array,FAQs)

lncount = 0
lnData = 0

FOR x = 1 TO ALINES(FAQ_Array,FAQs)

lcString=FAQ_Array(x)

* remove tabs
lcString = StrTran(lcString,CHR(9),"&quot

IF NOT EMPTY(lcString)
lnData = lnData + 1
APPEND BLANK
REPLACE myText WITH lcString
lnCount = lncount + 1

* comment this out later, just so you can see it in action.
IF lnCount < 15
MESSAGEBOX(lcString)
ENDIF
ENDIF
ENDFOR

GO TOP
BROWSE

Jim Osieczonek
Delta Business Group, LLC

http://www.deltabg.com

 

[baltman]

One of my less heralded posts from last year may also be of some help. thread184-266072 concerned automatically reading webmail and fowarding new messages, but the concept of reading dynamic links and then 'doing something' remains the same.

Brian

 

[jimoo]

Probably less heralded because most of us did not have a need for it, but with the internet becoming an extension of our hard drive (my quote for some time now) I think it will be appreciated more.

Certainly worth a star anyway.


Jim Osieczonek
Delta Business Group, LLC

http://www.deltabg.com