+1 to make nicer APIs! It is always good to have more high-quality API designs to look at.
That said, it looks more like an API experiment, not a practical solution for a day job, at least in its current state:
* response body encoding detection is wrong, as it doesn't take meta tags or BOM into account;
* base url detection is wrong, as it doesn't take <base> tag in account;
* URL parsing (joining, etc.) is implemented using string operations instead of stdlib, so a careful inspection is required to make sure it works in edge cases. For example, I can see right away that .absolute_links is wrong for protocol-related urls (e.g. "//ajax.microsoft.com/ajax/jquery/jquery-1.3.2.min.js")
* html2text used for .markdown is GPL - I know people have different opinion on this, but in my book if you import from
a GPL package, your package becomes GPL as well;
* each .xpath call parses a chnk of HTML again, even if a tree is already present
* shortcuts are opinionated and with no clear behavior, e.g. .links deduplicates URLs by default, it deduplicates them using string matches (so e.g. different order of GET arguments => URLs are considering unique); it checks for '.startswith('#")' which looks arbitrary (what if these links are used in a headless browser? what if someone wants to fetch them using _escaped_fragment which many sites still support? why filter out such URLs if they are relative, but not if they are absolute?)
That said, it looks more like an API experiment, not a practical solution for a day job, at least in its current state:
* response body encoding detection is wrong, as it doesn't take meta tags or BOM into account;
* base url detection is wrong, as it doesn't take <base> tag in account;
* URL parsing (joining, etc.) is implemented using string operations instead of stdlib, so a careful inspection is required to make sure it works in edge cases. For example, I can see right away that .absolute_links is wrong for protocol-related urls (e.g. "//ajax.microsoft.com/ajax/jquery/jquery-1.3.2.min.js")
* html2text used for .markdown is GPL - I know people have different opinion on this, but in my book if you import from a GPL package, your package becomes GPL as well;
* each .xpath call parses a chnk of HTML again, even if a tree is already present
* shortcuts are opinionated and with no clear behavior, e.g. .links deduplicates URLs by default, it deduplicates them using string matches (so e.g. different order of GET arguments => URLs are considering unique); it checks for '.startswith('#")' which looks arbitrary (what if these links are used in a headless browser? what if someone wants to fetch them using _escaped_fragment which many sites still support? why filter out such URLs if they are relative, but not if they are absolute?)