matsche
Joined 8 years ago
Karma
191
Submissions
16
Member since
Mar 9
Recent submissions
Comment · 5 years ago
Comment · 5 years ago
Yes, you can either limit how many request can be send concurrently or enforce a fixed or random delay in between consecutive requests.
Basically this a just a futures_timer::Delay [0] that is reset after each request, which is non blocking.
Comment · 5 years ago
Thanks, you're right. I fixed it just now by also requiring the tokio feature for all the tests in the Cargo.toml.
Comment · 5 years ago
The guys from corridor crew did a great break down of that amazing scene [0]
Comment · 5 years ago
I haven't used scraper too much. I personally find the predicate approach of select.rs [0] easier to use. However in this case the selector approach just made more sense.
Standalone html5ever can be a bit cumbersome to work with directly, scraper is basically an implementation of the html5ever's `TreeSink` trait, where as `select.rs` uses the hmtl5ever `RcDom` to parse the document but stores it in a more convenient way. If you look for a minimal approach you should at select.rs which basicially only depends on html5ever
Comment · 5 years ago
tokio support just landed https://github.com/mattsse/chromiumoxide
Comment · 5 years ago
Unfortunately this crate has been abandoned by its author for quite some time now. The main differences are that are that chromiumoxide uses async rust by design and that all the devtools protocol types are generated.
rust-headless-chrome was very helpful for some parts and bootstrapped some parts of chromiumoxide, such as the key and mouse clicking. However I looked at chromedp and puppeteer itself for the most part.
Comment · 5 years ago
> The pdl files are the canonical protocol definitions and are maintained manually by the devtools team [0].
The JSON protocol definition files are then in fact being generated from those pdl files. You find them mirrored in the devtools-protocol-repo [1]
[0] https://chromedevtools.github.io/devtools-protocol/ [1] https://github.com/ChromeDevTools/devtools-protocol/
Comment · 5 years ago
I started that way, but that was more work than writing a generator. Luckily I found https://github.com/chromedp/pdlgen
Comment · 5 years ago
It uses async-tungstenite [0] for the websocket part. At the moment configured only with async-std-runtime but since async-tungstenite also supports tokio, the option to use tokio instead of async-std will be added as well.
Comment · 5 years ago
It only supports the chrome devtools protocol[0]. chromiumoxide is therefor more like a (limited) puppeteer rust port. Support for javascript page injection is currently being worked on
I once implemented the crossref api in rust in order to search for publications from the command line https://github.com/mattsse/crossref-rs