Panos

I think this "we ignore you" behavior is specific to Norse, behaving like a low-cost airline in financial trouble. Whatever they can save from such incidents by ignoring customer complaints benefits their bottom line.

Given that it took me some time to locate someone willing to act as my representative, I consider that a way to send him more business. I can point to his LinkedIn page, but not sure if he checks messages there.

OP here. I wrote this up because the discourse around AI in legal contexts usually swings between "it replaces everyone" and "it hallucinates case law and gets you sanctioned." This was the case in the middle, where it actually shone: navigating an international dispute in a foreign conciliation court (the Norwegian Forliksråd), where even finding a human lawyer with all the necessary knowledge would be challenging, and paying one would be completely inefficient.

As I note at the end, the AI eliminated the knowledge bottleneck, but it still took 11 months of waiting for companies, agencies, and courts to reply.

Happy to answer questions about the routing or the EU261 technicalities. And to preempt the obvious one: a credit card chargeback only covers the original ticket cost. This was about getting the statutory €600/passenger penalty plus our overnight care expenses (and it was mainly the latter that prompted the whole story; Norse would probably have saved money if it had just arranged hotel rooms instead of handing us a $25 voucher).

Point well taken. :-)

I actually touch on this exact risk at the very end of the post. AI can automate drafting and knowledge retrieval, but that is only a fraction of the overall process.

The legal system is fundamentally slow, by design. It still took 11 months of waiting for agencies, airlines, and courts to move. That inherent slowness is creating friction to filter out low-effort "slop"; that latency partially accomplishes that.

Not an issue of cost, at all.

Absolutely the easiest solution would have been to have a written exam on the cases and concepts that we discussed in class. It would take a few hours to create and grade the exam.

But at a university you should experiment and learn. What better class to experiment and learn than the “AI Product Management”. Students were actually intrigued by the idea themselves.

The key goal: we wanted to ensure that the projects that students submitted was actually their own work, not “outsourced” (in a general sense) to teammates or to an LLM.

Gemini 3 and NotebookLM with slide generation were released in the middle of the class, and we realized that it is feasible for a student to have a flaweless presentation in front of the class, without understanding deeply what they are presenting.

We could schedule oral exams during the finals week, which would be a major disruption for the students, or schedule exams during the break, violating university rules and ruining students vacation.

But as I said, we learned that AI-driven interviews are more structured and better than human-driven ones, because humans do get tired, and they do have biases based on who is the person they are interviewing. That’s why we decided to experiment with voice AI for running the oral exam.

I agree that I am not yet confident to use this approach for my technical classes. I am still very unhappy with any option for assessment for technical classes, but I would not trust an LLM to come up with good questions. NotebooksLM does come up with decent quizzes, but nothing super hard.

For the use of LLM in classes: I understand the reasoning, but I found LLMs to be extremely educational for parsing through dense material (eg parsing an NTSB report for an Uber self-driving crash). Prohibiting students from using LLMs would be counterproductive.

But I still want students to use LLMs responsibly, hence the oral exam.

By the way the voice agent flagged the system as “the student is obviously fooling around”. I was expecting this to be caught during the grading phase but ElevenLabs has done such a good work with their product.

Guys, thank you for such fooling around. All these adversarial discussions will be great for stress testing the system. Very likely we will use these conversations as part of the course in the Spring to get students to see what it means to let AI systems “in the wild”.

Not the case for the class in the blog post, but we also have many online classes. Many professionals prefer these online classes because they can attend without having to commute, and can do it from a place of their own convenience.

Such classes do not have the luxury of pen-and-paper exams, and asking people to go to testing centers is a huge overkill.

Take home exams for such settings (or any other form of written exam) are becoming very prone to cheating, just because the bar to cheating is very low. Oral exams like that make it a bit harder to cheat. Not impossible, but harder.

Just in case, I am the author of the blog post. For our "AI" class, it felt like a good class to experiment with something novel.

No, we do not want to eliminate the pen and paper exam. It works well. We use it.

The oral exam is yet another tool. Not a solution for everything.

In our case, we wanted to ensure that the students who worked on the team project: (a) contributed enough to understand the project, (b) actually understood their own project and did not rely solely on an LLM. (We do allow them to use LLMs, it would be stupid not to.)

The students who did badly in the oral exam were exactly the students who we expected to do badly in the exam, even though they aced their (team) project presentations.

Could we do it in person? Sure, we could schedule personalized interviews for all the 36 students. With two instructors, it would have taken us a couple of days to go through. Not a huge deal. At 100 students and one instructor, we would have a problem doing that.

But the key reason was the following: research has shown that human interviewers are actually worse when they get tired, and that AI is actually better for conducting more standardized and more fair interviews. That result was a major reason for us to trust a final exam on a voice agent.