akisej

Let me ask you something I always ask my reports: if you were your manager, what would you do?

Pretty interface, although I remain unconvinced of how I'd actually use it. If I'm just prototyping for myself, LLM providers offer a decent history, and I rarely need to share notebook-style explorations of LLMs with my team. For production use cases at logicloop.com/ai we just add our prompts into code.

What's the use case you're envisioning people using AI notebooks for?

Appreciate the effort you put into this. In terms of user flow, I typically need one or two of these use cases at a time. In that case, I just type my conversion into a search engine like Google, and often use their default box.

Can you share what types of use cases you've seen people use KodyTools for?

Yup, that's part of it but I mean it bidirectionally - users can accidentally leak data to models too, which is concerning to SecOps teams without a way to monitor / auto-redact.

These common issues tend to prevent LLMs from being used in the wild: * Data Leakage * Hallucination * Prompt Injection * Toxicity

So yes it does include prompt injection, but is a bit broader. Data Leakage is one that several customers have called out, aka accidentally leakage PII to underlying models when asking them questions about your data.

I'm evaluating tools like Private AI, Arthur AI etc. but they're all fairly nascent.

Great starting point! These diagrams notably miss a LLM firewall layer, which is critical in practice to safe LLM adoption. Source: We work with thousands of users for logicloop.com/ai

How about Amplitude or Heap? For an open source alternative you could consider PostHog.

Very cool, but curious if you see people actually directly interacting with LLMs vs in a script as part of a larger application? I see myself needing debugging, visualizing output etc. so much that an IDE makes more sense to me as an interface, so want to learn about cases where that doesn't.

Aw man, sorry to hear this about your friend. Inanimate objects are directly subject to the laws of physics, but living beings that have intention and will are able to circumvent those. For example, I can jump despite gravity existing. Yes, "in the long run, we're all dead", but applying laws of entropy as a reason to not live seems indicative of a lack of will, rather than a natural law every being must follow.

As long it's clear that this is fiction, how would something like this be more damaging than a series like The Man in the High Castle, or other sci-fi that imagines an alternate universe? I think it's a nifty technique that allows us to viscerally imagine and live out our parallel universes.

Yeah, in general the more data you're able to use (assuming the context window supports it), the better results tend to be. We arrived at the data schema being a good enough compromise at which the benefits outweigh the risks for several use cases. Besides, some data stores that are generated by third-parties actually have common schemas (think Sendgrid / Hubspot activity data), so you're not risking much but potentially gaining a lot of sales ops productivity.

This seems overall well-written and well-explained, but curious for that piece on fine-tuning. This article only recommends it as a last resort. That makes sense for a casual user, but if you're a company seriously using LLMs to provide services for your customers, wouldn't the cost of training data be offset by the potential gains you have and the edge cases you might automatically cover by fine-tuning instead of trying to whack-a-mole predict every single way the prompt can fail?