mfarragher
Joined 7 years ago
Karma
33
Submissions
7
Member since
Nov 19
Data scientist
View on Hacker NewsRecent submissions
Comment · 5 years ago
I was surprised while I was developing this package how little code was needed in the end to write the functions, once the markdown data is extracted from files. md -> HTML -> ASCII was the hairy part but the results from html2text look great. The html2text config options aren't well-documented - there was a lot of trial-and-error.
File system properties do seem like an intuitive next step to extend the metadata 'getter', as the pathlib.Path objects are easily accessible. I was aiming to bring timestamps in but realised that the timestamps aren't available consistently in Linux, so I'll need to think about a cross-platform design.
FrontMatter is a good shout, I haven't used any in my notes, hopefully that content can be parsed neatly.
Comment · 5 years ago
Hi all,
I've made a new Python package - obsidiantools - for getting structured metadata about your Obsidian.md notes and analysing your vault. Complement your Obsidian workflows by getting metrics and detail about all your notes in one place through the widely-used Python data stack.
Check out the Github page for more detail about the API and its usage. There's a link to the 'obsidiantools in 10 minutes' demo, which is set up in a virtual machine through Binder, so you can interact with the code for the demo there!
Very little code is needed to analyse a vault. What I find really great is the ability to do sophisticated analytics through the integration with NetworkX graphs. In my demo I've applied the PageRank algorithm to the vault - analysis of backlink quality is just one example of how graph analytics can complement your knowledge management workflows, especially on larger vaults.
I go into more detail on future development, screenshots and more in this Obsidian community topic: https://forum.obsidian.md/t/obsidiantools-python-package-for...
In practice this should work for Zettelkasten vault formats - the API has worked OK for one of my vaults where I had dates/years with hyphens and I think it would handle the default 12-digit note names.
Comment · 7 years ago
Yes, that's a good point on the plots. I'll try to make them more prominent in future docs.
These are some things I've included in the library which aren't implemented in Statsmodels:
- Breusch-Pagan studentized test of heteroskedasticity (available in R)
- Standardized / beta coefficients (still an open feature request in https://github.com/statsmodels/statsmodels/issues/3857 )
- Leverage vs residuals squared plot (there's an influence plot but not something similar to Stata's lvr2plot)
Even the most common metric I use for assessing models – root MSE – isn't stored in the Statsmodels object summary. To assess an OLS model in Statsmodels I'd find I do so much repetitive code, yet in Stata the commands are fairly succinct.
Other things I also added to make encoding of variables easier:
- InteractionEncoder
- DummyEncoder (to cover different ways of treating missing values)
The more I thought about these missing features, the more I thought they can be wrapped up in a more coherent way. :-)
Comment · 7 years ago
Hi there, I'm a data scientist and economist who uses the main Python stats libraries regularly. I was frustrated by how long it takes to fit basic regression models and diagnose models, so I began working on a package called Appelpy (Applied Econometrics Library for Python).
The aim: Make regression modelling as easy as pie.
Now that I've tightened up my code coverage and fleshed out some documentation – ReadTheDocs and notebook tutorials that can be viewed with Binder & Nbviewer – I'm sharing the library more widely!
The library is built upon Statsmodels but I've tried to make a more cohesive interface for regression modelling, with model diagnostics in mind especially. Model diagnostics are the time-consuming and repetitive part of regression modelling in Python, but through Appelpy diagnostics can be done with minimal code.
This is the first project I've released on PyPI and I was working on it to hone my software engineering skills, so I'm interested in tips and feedback.
- Mark