Read data in a WEB page

This forum can be used to talk about general design strategies, new ideas and questions in general related to wxWidgets. If you feel your questions doesn't fit anywhere, put it here.
Cristiano
Earned a small fee
Earned a small fee
Posts: 11
Joined: Tue Jun 17, 2014 11:53 pm

Read data in a WEB page

Postby Cristiano » Wed Nov 22, 2017 3:26 pm

A web page shows some data updated every 10 seconds:
https://isslive.com/displays/topoDisplay1.html
I need to read those numbers (I want to save’em in a file).

I tried with wxHtmlWindow::LoadPage(), but it doesn’t work.
Please, could someone help me?

ONEEYEMAN
Part Of The Furniture
Part Of The Furniture
Posts: 2058
Joined: Sat Apr 16, 2005 7:22 am
Location: USA, Ukraine

Re: Read data in a WEB page

Postby ONEEYEMAN » Wed Nov 22, 2017 3:47 pm

Hi,
You need to catch page loaded event (see documentation for a proper name) and do the logic there.
Basically the idea is - you need to wait until the page is loaded and the data becomes available to you.

Thank you.

Cristiano
Earned a small fee
Earned a small fee
Posts: 11
Joined: Tue Jun 17, 2014 11:53 pm

Re: Read data in a WEB page

Postby Cristiano » Wed Nov 22, 2017 5:09 pm

Hi,
I don't see any page loaded event. I just do:

Code: Select all

wxFileSystem::AddHandler(new wxInternetFSHandler);
wxFrame *frame= new wxFrame(this, wxID_ANY, "");
wxHtmlWindow *html = new wxHtmlWindow(frame); html->SetRelatedFrame(frame, "HTML : %%s");

if(html->LoadPage("https://isslive.com/displays/topoDisplay1.html")) {
   wxMessageBox(html->ToText());
}
delete html; delete frame;

but I get: "Unable to open requested HTML document".

EDIT
I'm also trying the webview sample. The page is correctly loaded (even if the numbers are not showed), but it's not updated... :(
Last edited by Cristiano on Wed Nov 22, 2017 5:33 pm, edited 1 time in total.

User avatar
doublemax
Moderator
Moderator
Posts: 11536
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: Read data in a WEB page

Postby doublemax » Wed Nov 22, 2017 5:32 pm

wxHtmlWindow can't load https streams. Only http.

If you want to use a browser control for that purpose, you'll have to use wxWebView and get the page source in the wxEVT_WEBVIEW_LOADED event handler.

The same question was asked just a few days ago:
viewtopic.php?f=1&t=44035
Use the source, Luke!

ONEEYEMAN
Part Of The Furniture
Part Of The Furniture
Posts: 2058
Joined: Sat Apr 16, 2005 7:22 am
Location: USA, Ukraine

Re: Read data in a WEB page

Postby ONEEYEMAN » Wed Nov 22, 2017 5:36 pm

Hi,
Can you try and run any of the samples/html sample program?

Thank you.

Cristiano
Earned a small fee
Earned a small fee
Posts: 11
Joined: Tue Jun 17, 2014 11:53 pm

Re: Read data in a WEB page

Postby Cristiano » Wed Nov 22, 2017 5:46 pm

doublemax wrote:If you want to use a browser control for that purpose, you'll have to use wxWebView and get the page source in the wxEVT_WEBVIEW_LOADED event handler.


I just see the field name (like "USLAB000032"), but no number is showed.

Cristiano
Earned a small fee
Earned a small fee
Posts: 11
Joined: Tue Jun 17, 2014 11:53 pm

Re: Read data in a WEB page

Postby Cristiano » Wed Nov 22, 2017 5:47 pm

ONEEYEMAN wrote:Hi,
Can you try and run any of the samples/html sample program?


Sure! Why do you ask?

ONEEYEMAN
Part Of The Furniture
Part Of The Furniture
Posts: 2058
Joined: Sat Apr 16, 2005 7:22 am
Location: USA, Ukraine

Re: Read data in a WEB page

Postby ONEEYEMAN » Wed Nov 22, 2017 5:54 pm

Hi,
Probably the page didn't load yet, therefore you don't have the full source yet.

Can you post some code using wxWebView? Like doublemax said - you should catch the page loaded event.

EVT_WEBVIEW_LOADED(id, func):
Process a wxEVT_WEBVIEW_LOADED event generated when the document is fully loaded and displayed. Note that if the displayed HTML document has several frames, one such event will be generated per frame.
.

Thank you.

Cristiano
Earned a small fee
Earned a small fee
Posts: 11
Joined: Tue Jun 17, 2014 11:53 pm

Re: Read data in a WEB page

Postby Cristiano » Thu Nov 23, 2017 9:52 am

ONEEYEMAN wrote:Can you post some code using wxWebView? Like doublemax said - you should catch the page loaded event.


It's what I'm doing, I simply took the webview sample and I changed the line
WebApp() : m_url("https://isslive.com/displays/topoDisplay1.html")
Then, I use the WebFrame::OnDocumentLoaded() function (which is connected to the wxEVT_WEBVIEW_LOADED event) to show the page:
wxLogMessage("%s", m_browser->GetPageSource());
but I just see the field name (like "USLAB000032") and no number is loaded. If I refresh the page, I get exactly the same page source.

EDIT
I also used libcurl (as suggested by doublemax in another thread), but I get the same page source.
Is that a strange page?

User avatar
doublemax
Moderator
Moderator
Posts: 11536
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: Read data in a WEB page

Postby doublemax » Thu Nov 23, 2017 10:27 am

It seems the actual update is done with JavaScript. In that case you won't see it in the source.

You should look for other sources of information, e.g. http://api.open-notify.org/iss-now.json
If you only need the position of the ISS, that should be sufficient. If not, keep looking for other URLs. I'm almost certain there are sources for this information that are easy to process.

The above URL doesn't use HTTPS, so you could even process it using wxURL or wxHTTP without using a browser control

Edit:
Found another URL:
https://api.wheretheiss.at/v1/satellites/25544
from:
https://wheretheiss.at/w/developer
Use the source, Luke!

Cristiano
Earned a small fee
Earned a small fee
Posts: 11
Joined: Tue Jun 17, 2014 11:53 pm

Re: Read data in a WEB page

Postby Cristiano » Thu Nov 23, 2017 10:44 am

Thank you very much for the time you took to help me, but I need to read those numbers (the link you posted uses the TLEs, which I'm already using).

If I save the page from Firefox, I see those numbers. Probably the best way is to automate the saving in .txt format (now I'm trying AutoIt).

ONEEYEMAN
Part Of The Furniture
Part Of The Furniture
Posts: 2058
Joined: Sat Apr 16, 2005 7:22 am
Location: USA, Ukraine

Re: Read data in a WEB page

Postby ONEEYEMAN » Fri Nov 24, 2017 3:04 am

Hi,
Since it uses JavaScript why not use the new GSoC functionality - Integrate JavaScript into wxWebView?

Thank you.


Return to “General Development”

Who is online

Users browsing this forum: No registered users and 2 guests