stevepike

One-off libraries that don't have a runtime dependency on Rails are typically very low-maintenance. You can mostly leave them alone (even a security vulnerability is unlikely to be exploitable for how you're using one of these, as often user input isn't even getting through to them). For instance a gem you install to communicate with the stripe API is not typically going to break when you upgrade Rails. Or adding httparty to make some API requests to other services.

Then there are libraries that are really framework extensions, like devise for authentication or rspec for testing. These are tightly coupled to Rails, sometimes to its private internals, and you get all sorts of nasty compatibility issues when the framework changes. You have to upgrade Rails itself because you really do need to care about security support at that level, even for a relatively small company, so you can end up in a situation where leaving these other dependencies to fester makes upgrading Rails very hard.

(I run a startup that's a software-enabled service to upgrade old Rails apps).

Comment · 7 months ago

It looks like this is a community continuation of the axlsx gem which was maintained back in the day by Randy Morgan (randym) over at https://github.com/randym/axlsx. One of my earliest open source contributions was adding support so that you could output spreadsheets with "conditional" formatting (color something red if it is below some value, for instance). I remember Randy being extremely supportive of new contributors and it made me want to be a part of the ruby community.

Thanks for continuing the work!

Comment · 8 months ago

Yeah, earlier in the GPT days I felt like this was a good example of LLMs being "a blurry jpeg of the web", since you could give them something that was very close to an existing puzzle that exists commonly on the web, and they'd regurgitate an answer from that training set. It was neat to me to see the question get solved consistently by the reasoning models (though often by churning a bunch of tokens trying and verifying to count 888 + 88 + 8 + 8 + 8 as nine digits).

I wonder if it's a temperature thing or if things are being throttled up/down on time of day. I was signed in to a paid claude account when I ran the test.

Comment · 8 months ago

I'm a bit surprised it gets this question wrong (ChatGPT gets it right, even on instant). All the pre-reasoning models failed this question, but it's seemed solved since o1, and Sonnet 4.5 got it right.

https://claude.ai/share/876e160a-7483-4788-8112-0bb4490192af

This was sonnet 4.6 with extended thinking.

Comment · 8 months ago

Reminds me of https://xkcd.com/1200/

Comment · 11 months ago

PHP would definitely be in scope, either that or Java are likely to be next for us. If you are familiar with PHPs ecosystem I'd be interested in your take on what's most important / problematic there.

Comment · 11 months ago

Thanks! We support Python, JS, and Ruby right now (started with dynamic languages).

I'm not sure what you mean by prioritization on the issues, but generally we are trying to help you figure out what to upgrade next, and to actually do it too.

Comment · 11 months ago

This is hard for lots of companies. Some ignore the problem entirely until there's a fire drill (which can be a huge risk if you end up on an old major version that won't get patched). Some keep everything up to date, and then taking a new security patch is trivial. It's always risk/reward tradeoff between the risk of breaking production with an upgrade and the value an org sees from staying up to date. We work on this problem at Infield (https://www.infield.ai/post/introducing-infield) where we tackle both sides of the project management: "Which dependencies should I prioritize upgrading" and "How difficult and likely to break production is this upgrade".

To your specific points

> 1. How do you decide what's actually urgent? CVSS? EPSS? Manual assessment?

The risk factors we track are open CVEs, abandonment (is this package supported by the maintainer?), and staleness (how deep in the hole am I?).

We also look at the libyear metric as an overall indication of dependency health.

> 2. Do you treat "outdated but not vulnerable" dependencies differently from "has CVEs"?

We group upgrades into three general swimlanes:

  - "trivial" upgrades (minor/patch versions of packages that respect semantic versioning, dev/test only packages). We batch these together for our customers regardless of priority.

  - "could break". These deserve standalone PRs and an engineer triaging when these become worth tackling, if ever.

  - "major frameworks". Think something like Rails. These are critical to keep on supported versions of because the rest of the ecosystem moves with them, and vulnerabilities in them tend to have a large blast radius. Upgrading these can be hard. You'll definitely need to upgrade someday to stay supported, and getting there has follow-on benefits on all your other dependencies, so these are high priority.

> 3. For those using Dependabot/Renovate/Snyk - what's your workflow? Do you review every alert or have you found a good filtering system?

We offer a Github app that integrates with alerts from Dependabot. While security teams are happy with just a scanner, the engineering teams that actually do this upgrade work need to mash that up with all the other data we're talking about here.

Comment · 1 year ago

This is cool, it looks to me like you're integrating static analysis on the user's codebase and the underlying dependency. Very curious to see where it goes.

We've found dependency upgrades to be deceptively complex to evaluate safety for. Often you need context that's difficult or impossible to determine statically in a dynamically typed language. An example I use for Ruby is the kwarg migration from ruby 2.7->3 (https://www.ruby-lang.org/en/news/2019/12/12/separation-of-p...). It's trivial to profile for impacted sites at runtime but basically impossible to do it statically without adopting something like sorbet. Do you have any benchmarks on how reliable your evaluations are on plain JS vs. typescript codebases?

We ended up embracing runtime profiling for deprecation warnings / breaking changes as part of upgrading dependencies for our customers and have found that context to unlock more reliable code transformations. But you're stuck building an SDK for every language you want to support, and it's more friction than installing a github app.

Comment · 1 year ago

This is very true. It can be a real fire drill if it turns out you need to go up a major version in some other dependency in order to get a security fix. It can get even worse in JS if you're on some abandoned package that's pinned to an old version of some transient dependency which turns out to be vulnerable. Then you're scrambling to migrate to some alternate package with no clear upgrade path.

On the flipside sometimes you get lucky and being on an old version of a package means you don't have the vulnerability in the first place.

libyear is a helpful metric for tracking how much of this debt you might have.

Comment · 1 year ago

This seems to show the power of the reasoning models over interacting with a prompted chat-tuned LLM directly. If I navigate backwards on your link Sonnet 4 gets it right.

I've used a similar prompt - "How can you make 1000 with exactly nine 8s using only addition?"

Here's GPT 4.5 getting it wrong: https://chatgpt.com/share/683f3aca-8fbc-8000-91e4-717f5d81bc...

It tricks it because it's a slight variation of an existing puzzle (making 1000 with 8 8s and addition only).

The reasoning models seem to reliably figure it out, though. Some of them even come up with a proof of why it's impossible to do with 9 8s. Here's o4 getting it right: https://chatgpt.com/share/683f3bc2-70b8-8000-9675-4d96e72b58...

Show HN: Track Deprecation Warnings in Prod

4 stevepike 1 year ago 0 comments