antichronology
Joined 9 years ago
Karma
50
Submissions
28
Member since
Aug 27
Recent submissions
Comment · 1 year ago
Comment · 1 year ago
> finding sequences on NIH
Almost! Yes most of the data is on NIH sub-institutes. For us we take most of the data from NCBI and intelligently pair it together. The training objective of our model takes pairs of sequences (thus the Joint Embedding Architecture) and trains the model to recognize that they are semantically similar but differ in appearance. This is conceptually similar to a lot of the contrastive learning literature from computer vision.
Sounds like a fun side project :)
There are some great tools out there for putting together plasmids for gene therapies where you can plug in different "elements". Promoters UTRs payloads - check out SnapGene I believe they have a free version.
I personally am hopeful that the political headwinds will blow over. When it comes to cancer vaccines it's one of the most exciting new modalities for treating cancer.
1 in 2 Americans are going to get cancer in their lifetime so no matter political affiliation, the need for health will ultimately drive people to invest in the modality.
Comment · 1 year ago
That would be really cool. Navigating SRA and mining out reasonable $ relevant tasks is a huge bottleneck.
I find it takes a large amount of effort to parse what the authors are doing, whether the data is high quality, and how to pre-process it in a way that makes sense for the task at hand.
Would love to chat more about how you're thinking of evaluating quality of these agents.
Comment · 1 year ago
Here is the link for benchmarks and their utility: https://youtu.be/JdT78t1Offo?t=1444
"We have internal benchmarks. Yeah. But we don't we don't publish them."
"we have internal benchmarks that the team focuses on and improving and then we also have a bunch of tasks like I think that accelerating our own engineers is like a top top priority for us"
The equivalent for us would be to ultimate looking to improve experimental results. Benchmarks are a good intermediate point but not the ultimate goal
Comment · 1 year ago
Thanks for engaging. Do you mind elaborating on your stance?
From where we sit - there are people with diseases and mRNA is an effective way to revert them to a healthy state.
I'd be interested to hear more where you're coming from
Comment · 1 year ago
I watched an interview with one of the co-founders of Anthropic where his point is that although benchmarks saturate they're still an important signal for model development.
We think the situation is similar here - one the challenges is aligning the benchmark with the function of the models. Genomic benchmarks for gLMs and RNA foundation models have been very resistant to staturation.
I think in NLP the problem is that they are victims of their own success where the models can be overfit to particular benchmarks really fast.
In genomics we're a bit behind. A good paper on this is DartEval where they provide levels of complexity https://arxiv.org/abs/2412.05430
in RNA the models work much better than DNA prediction but it's key to have benchmarks to measure progress.
Comment · 1 year ago
Huge fan of the work! I'm a big fan of papers from Seelig lab :)
Comment · 1 year ago
We haven't looked into this deeply yet sounds interesting. Do you have any resources where to start looking at this? Feel free to reach out to us
[email protected]
Comment · 1 year ago
We think that Orthrus can be applied in a bunch of ways to non-coding and coding RNA sequences but it's definitely fair we're a bit more focused on RNA sequences currently instead of non-coding parts of the genome like promoters and intergenic sequences.
For the data - Orthrus is trained on non experimentally collected data so our pre-training dataset is large by biological standards. It adds up to about 45 million unique sequences and assuming 1k tokens per sequence it's about 50b tokens.
We're thinking about this as large pre-training run on a bunch of annotation data from Refseq and Gencode in conjunction with more specialized Orthology datasets that are pooling data across 100s of species.
Then for specific applications we are fine tuning or doing linear probing for experimental prediction. For example we can predict half life using publicly available data collected by the awesome paper from: https://genomebiology.biomedcentral.com/articles/10.1186/s13...
Or translation efficency: https://pubmed.ncbi.nlm.nih.gov/39149337/
Eventually as we ramp up out wet lab data generation we're thinking about what does post-training look like? There is an RL analog here that we can use on these generalizable embeddings to demonstrate "high quality samples".
There are some early attempts at post-training in bio and I think it's a really exciting direction
Comment · 1 year ago
This is a really interesting direction. There is this big field of Cell Free (cfRNA) cancer detection. We talked to a few people in the field and think that embedding sequences for this direction could be really valuable. One challenge here is that it's hard to set up evaluation tasks since the public data is scarce
Comment · 1 year ago
> ground truth
Hey yes, the ground truth for our evaluations is measured experimental data. Our models are benchmarked using mRNABench, which aggregates results from high-throughput wet lab experiments.
Our goal, however, is to move beyond predicting existing experimental outcomes. We intend to design novel sequences and validate their function in our own lab. At that stage, the functional success of the RNA we design will become the ground truth.
> peer reviewed?
Both mRNA bench and Orthrus are in submission (at a big ML conference and a big name journal) - unfortunately the academic systems move slow but we're working on getting them out there.
> synthetic mRNA sequences
I think you're asking on generalizing out of distribution to unnatural sequences. There are two ways that we do this: (1) There are these screens called Massively Parallel Reporter Assays (MPRAs) and we eval for example on https://pubmed.ncbi.nlm.nih.gov/31267113/
Here all the sequences are synthetic and randomly designed and we do observe generalization. Ultimately it depends on the problem that we're tackling: some tasks like gene therapy design require endogenous sequences.
(2) The other angle is variant effect prediction (VEP). It can be thought of as a counterfactual prediction problem where you ask the model whether a small change in the input predicts a large change in the output. This is a good example of the study (https://www.biorxiv.org/content/10.1101/2025.02.11.637758v2)
> experimental verification of their predicted properties
all our model evaluations are predictions of experimental results! The datasets we use are collections of wet lab measurements, so the model is constantly benchmarked against ground-truth biology.
The evaluation method involves fitting a linear probe on the model's learned embeddings to predict the experimental signal. This directly tests whether the model's learned representation of an RNA sequence contains a linear combination of features that can predict its measured biological properties.
Thanks for the feedback I understand the caution around pre-prints. We believe a self-supervised learning approach is well-suited for this problem because it allows the model to first learn patterns from millions of unlabeled sequences before being fine-tuned on specific, and often smaller, experimental datasets.
Comment · 1 year ago
There is a number of different technologies. Some of the big ones are:
- mRNA therapies: These therapies deliver a synthetically created messenger RNA (mRNA) molecule, typically protected within a lipid nanoparticle (LNP), to a patient's cells. The cell's own machinery then uses this mRNA as a temporary blueprint to produce a specific protein.
The big example here is CAR-T therapy from Capstan which just got acquired for 2.1B. Their asset,CPTX2309 , is currently in Phase 1. Previously to do Car-T therapy you had to extract a patient's T-cells and genetically engineer them in a special facility. Now the mRNA gets delivered directly to the patient's t cells which significantly lowers the cost and technical hurdles.
- RNA interferences (RNAi): Used for gene expression knockdown through natural cellular mechanisms for viral detection. The big example here is Alnylam with 5 approved therapies and a number in clinical trials.
- Antisense Oligonucleotides (ASOs): Short single stranded RNA molecules that get delivered directly to the cell and target an existing mRNA. The big win here is Spinraza which is the first approved treatment for Spinal Muscular Atrophy (SMA) which previously didn't have a treatment. The Spinraza clinical trial (ENDEAR) was so effective that they deemed it unethical to continue it because the control arm wasn't receiving the treatment. Prior to Spinraza most patients would pass away prior to two years of age.
Comment · 3 years ago
I think the apt comparison here is landing on the moon. Although overtly it carries little value, the technologies developed along the way are very valuable.
As the article mentions improved fertilization, more accurate genome editing, and advances in in-vitro gastrulation are the real prize.
To me there is also something profoundly sad about the loss of molecular information with each extinct species. Having a simple hope, perhaps naive, to reverse this process is comforting.
Comment · 3 years ago
There is a great novel on a related topic: Permutation city by Greg Egan.
The concept is similar where the protagonist loads his consciousness to the digital world. There are a lot of interesting directions explored there with time asynchronicity, the conflict between real world and the digital identities, and the basis of fundamental reality. Highly recommend!
The immune system recognizes a sugar as a PAMP, or Pathogen-Associated Molecular Pattern, which is a signature of a potential microbial threat.
This initiates pyroptosis an inflammatory form of programmed cell death causing the cell to burst. This rupture releases tumor antigens and DAMPs (Damage-Associated Molecular Patterns), which are "danger signals" from the dying cell
The release of DAMPs shifts the Tumor Microenvironment (TME) from an immunologically "cold" to a "hot" state, promoting a potent Type I Interferon (IFN-I) response.
The release of DAMPs shifts the Tumor Microenvironment (TME) from an immunologically "cold" to a "hot" state, promoting a potent Type I Interferon (IFN-I) response.
This response recruits Antigen Presenting Cells (APCs), which engulf the newly released tumor antigens.
---
mRNA vaccines are somewhat of a parallel approach where the antigen selection and delivery happens manually. An mRNA vaccine delivers the encoding sequence for specific tumor antigens to drive production and presentation, training the immune system. One of the big challenges of this space is optimal antigen selection from the patient's tumor.
One thing I'm not fully clear on is why only tumor cell react to PAMP instead of healthy cells. Could be a promising approach but molecular biology is pretty tricky and the devil is always in the details.