Html Extraction from WebBrowser Ctrl

[Vec]

This code:

WebBrowser1.Document.body.innerHTML

Returns the body of the html doc. How do I get the whole doc, meta tags and all? -------------------------------------------------------------------------
-------------------------------------------------------------------------

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Univer

 

[vb5prgrmr]

Use advanced search. I posted this a couple of times as so has others. Use all words webbrowser source any date in your forums and you will find the answer about 4 or 5 down.

Good Luck

 

[strongm]

And here's one way of doing it without a Web Browser control (you'll need a reference to the Microsoft HTML OBject library:
[tt]
Dim DocumentFactory As HTMLDocument
Dim myHTMLDoc As HTMLDocument

Set DocumentFactory = New HTMLDocument

Set myHTMLDoc = DocumentFactory.createDocumentFromUrl("

http://www.google.com",

"&quot
Do Until myHTMLDoc.readyState = "complete"
DoEvents
Loop
Debug.Print myHTMLDoc.documentElement.outerHTML

 

[vb5prgrmr]

Well strongm, I was pointing Vec to your answer that comes up with the search, but that works also.

 

[strongm]

I was bored, I had a few spare minutes, I had VB open...

 

[Vec]

It works almost perfect, exept it does not pull the content form the description and keywords meta tags. ANyone see the bug?

TextBox named: txtOutput set to multiline
TextBox named: txtHtmlCode set to multiline
Timer named: tmrWebBrowser set to enabled = true. 500 interval
Command Button named: cmdStart
WebBrowser Component named: WebBrowser


'The Code

Private Sub Form_Load()

WebBrowser.Navigate "[URL unfurl="true"]http://www.audio-warehouse.com"[/URL]

End Sub

Private Sub cmdStart_Click()

Dim WebPageCode

WebPageCode = WebBrowser.Document.firstChild.OuterHTML

txtHtmlCode.Text = WebPageCode

Dim strHTML As String, lPointer As Long, MetaEnd As Long

Dim Start As Long, Finish As Long, strName As String

Dim PageTitle As String, PageDescription As String, PageKeyWords As String

strHTML = UCase$(txtHtmlCode.Text)

lPointer = InStr(strHTML, &quot;<TITLE>&quot;)

If lPointer > 0 Then

Start = lPointer + 7

Finish = InStr(strHTML, &quot;</TITLE>&quot;)

If Finish > Start Then

PageTitle = Mid$(txtHtmlCode.Text, Start, Finish - Start)

End If

End If

lPointer = 1

Do

lPointer = InStr(lPointer, strHTML, &quot;<META&quot;)

If lPointer = 0 Then Exit Do

MetaEnd = InStr(lPointer, strHTML, ">")

lPointer = InStr(lPointer, strHTML, "NAME")

lPointer = InStr(lPointer, strHTML, "=")

Start = InStr(lPointer, strHTML, """") + 1

Finish = InStr(Start, strHTML, """")

If Finish > MetaEnd Or Finish < Start Then Exit Do

strName = Mid$(strHTML, Start, Finish - Start)

strName = Trim$(strName)

lPointer = Finish

lPointer = InStr(lPointer, strHTML, "CONTENT")

lPointer = InStr(lPointer, strHTML, "=")

Start = InStr(lPointer, strHTML, """") + 1

Finish = InStr(Start, strHTML, """")

If Finish > MetaEnd Or Finish < Start Then Exit Do

Select Case strName

Case "DESCRIPTION"

PageDescription = Mid$(txtHtmlCode.Text, Start, Finish - Start)

Case "KEYWORDS"

PageKeyWords = Mid$(txtHtmlCode.Text, Start, Finish - Start)

End Select

If Len(PageDescription) > 0 And _

Len(PageKeyWords) > 0 Then Exit Do

lPointer = Finish

Loop

PageTitle = Trim$(PageTitle)

PageDescription = Trim$(PageDescription)

PageKeyWords = Trim$(PageKeyWords)

txtOutput.Text = &quot;<Title>&quot; + PageTitle + &quot;</Title>&quot; & vbCrLf & _
[code]&quot;<META NAME=&quot; + Chr(34) + &quot;Description&quot; + Chr(34) + &quot; CONTENT=&quot; + Chr(34) + PageDescription + Chr(34) + &quot;>&quot; & vbCrLf & _

&quot;<META NAME=&quot; + Chr(34) + &quot;Keywords&quot; + Chr(34) + &quot; CONTENT=&quot; + Chr(34) + PageKeyWords + Chr(34) + &quot;>&quot;

End Sub

Private Sub tmrWebBrowser_Timer()

If WebBrowser.Busy = False Then

cmdStart.Enabled = True

tmrWebBrowser.Enabled = False

End If

End Sub

-------------------------------------------------------------------------
-------------------------------------------------------------------------

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Univer

 

[strongm]

Boy, we know from thread222-509531 that you aren't really interested in help files or MSDN, but it's beginning to look like you don't even read the replies you are given here!

Threads thread222-514080, thread222-513378 and thread222-512711, all responses to your queries on this subject, contain the info you need. The Instr solution is a red herring, because it was a solution for parsing text. But your source is actually an HTML document, so you don't need to go to those lengths. The following requires a reference to the Microsoft HTML Object library
[tt]
Option Explicit

Private Sub Command1_Click()
txtHTMLcode.Text = vecFunction("

http://www.audio-warehouse.com",

True) ' tek-tips will put semi-colons after the URL, so you need to remove them
txtOutput.Text = vecFunction("

http://www.audio-warehouse.com",

False)
End Sub

Private Function vecFunction(strURL As String, Optional Parse As Boolean = True) As String
Dim DocumentFactory As HTMLDocument
Dim myHTMLDoc As HTMLDocument

Dim MetaTag As IHTMLElement
Dim MetaTagCollection As IHTMLElementCollection

' This from THIS thread
Set DocumentFactory = New HTMLDocument

Set myHTMLDoc = DocumentFactory.createDocumentFromUrl(strURL, "&quot
Do Until myHTMLDoc.readyState = "complete"
DoEvents
Loop

' From thread222-513378 and Thread222-512711
Set MetaTagCollection = myHTMLDoc.documentElement.getElementsByTagName("title&quot
If Parse Then
vecFunction = vecFunction + MetaTagCollection(0).outerText
Else
vecFunction = vecFunction + MetaTagCollection(0).outerHTML
End If

Set MetaTagCollection = myHTMLDoc.documentElement.getElementsByTagName("meta&quot

For Each MetaTag In MetaTagCollection
If InStr(UCase(&quot;Description KeyWords&quot, UCase(MetaTag.Name)) And MetaTag.Name <> &quot;&quot; Then
If Parse Then
vecFunction = vecFunction + vbCrLf + vbCrLf + MetaTag.content
Else
vecFunction = vecFunction + vbCrLf + MetaTag.outerHTML
End If
End If
Next

End Function
[/tt]
of course, if you do want to do it with the webbrowser and Instr, you need to change

WebPageCode = WebBrowser.Document.firstChild.OuterHTML

to

WebPageCode = WebBrowser.Document.documentelement.outerHTML

 

[AntonV]

I think we should go to this guy and ask a couple of questions.
First has any of the input so far been of any help?
Next is the whole document going to be saved locally for you to work on? Meaning are you wanting a way of capturing a web page and then walking through the various elements in order to develop your web pages?
Last if the only thing you want is to save a web page try this ....
Dim hElm As IHTMLElement
Dim htmltext As String
fso As New FileSystemObject
Dim file As Object

Set hElm = brwWebBrowser.Document.All.tags(&quot;html&quot.Item(0)
htmltext = hElm.outerHTML

Set fso = CreateObject(&quot;Scripting.FileSystemObject&quot
Set file = fspenTextFile(File1.Path & &quot;\&quot; & File_Name, 2, True)
file.Write htmltext
file.Close

Set file = Nothing
Set fso = Nothing

With the appropriate referencing you will be able to set a copy of the current wbebrowser page as a file locally on your hard drive.

If this is helpful then let us all know.
As it gets frustrating trying to follow some of the questioning with out a feedback.

NO FLAMING ALLOWED.....
NO HELP WILL FOLLOW.....

We all are trying our best to be the best we can be. Corny but true.

HTH --- Have a better day today than any other in your life.

 

[Vec]

Thanks guys,
In response to AntonV, I am not saving the whole doc, bsically I am constructing a meta tag extraction app to pull title, keywords and description. This is not an app for anyone else, or for sale, a client or any type of distribution. It is purly for myself and the learning curve. I have already learned many different things from all of you and I thank you for that. The reason I need to use a webrowser ctrl and not just Inet is becasue Inet gets forbidden pages a lot using Google. Example, place an inet on a form and draw the html from the inet to a text box, in the form load event, navigate to

www.google.com

, no problem right? Then change the nav to statement to

[URL unfurl="true"]http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=car+audio&btnG=Google+Search[/URL]

this example searches for car audio. Pulls the page right? Wrong! The html you are looking at is Googles Forbbiden Access Page. But you can push google searchs like this into a wb ctrl.

SO the big picture of what I am eventually trying to create is an app that will allow you to enter search words into a txt box, enter a number of pages to open in another text box (to open one after another contoled by the wb busy function) then read the meta info. (Just the title, keywords and desc)
If I can get it to that point, (which I will!) I can do many things with it, example, create logs of the top sites key words etc.
So that sums up my &quot;vision&quot;. I know you can easily buy commercial apps to do that and in fact I own one. but I obviously want to learn.

StrongM
Sorry if I ticked you off, I have been reading all of the posts, but a lot of what happens is that I follow the post or ref to another thread and either can't get it to work at all or can't get it to work right. I hope that my explanation above clears up what I am trying to do.
Be patient with me, I am not a pro programmer like you guys! In fact a year ago, I was impressed when I could use a command button and create a mxgbox by pressing it. Thought I was ready for the big times -------------------------------------------------------------------------
-------------------------------------------------------------------------

&quot;Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Univer

 

[strongm]

No, no, that's fine. As long as you're picking stuff up.

I just think that it may have been of benefit to all concerned to keep all those seperate threads on the same subject in one single thread. That way there'd have been a better picture about what you were really tring to do and, hopefully, as a result, a set of more and more appropriate responses to help you solve the issue.

 

[Vec]

I will keep them together in the future, sometimes I tend to wander. -------------------------------------------------------------------------
-------------------------------------------------------------------------

&quot;Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Univer