Chobilet

"We propose Zero-Error Horizon (ZEH) for trustworthy LLMs, which represents the maximum range that a model can solve without any errors."

I'm again taking your responses in good faith, but the abstract answers your question about what they are trying to achieve. For any statistical significance, you'd want to point to a baseline comparison(e.g. what I'm guessing is what you mean by "no reasoning" here). You'll also note within the paper, the author argues and cites that failing at the baseline step(e.g. multiplication) has shown "that error often adversely affects subsequent reasoning [38, 44]".

Which indicates to me, we don't need to use further "reasoning" given previous results/studies show a decline once our base has an error. To me, this seems like a fair assumption. Given though this is an active field of research, and we are largely testing a black box application, we can't say for certain. Further studies(like this one) will give researchers a better understand at what is and isn't possible.

I'm sure this comment was made in good faith, but most researchers would rightfully understand these intricacies, and this is likely intentional(as noted in the paper). At a quick glance, I cannot say whether or not the paper has been peer reviewed(though unlikely/in process given how recent it was published). In general, you'd find published papers also listed in a specific journal/conference(i.e. not just the archives which anyone can submit to).

Additionally, many of us in the field of researching LLM's are curious to understanding the boundaries and limitations of what is capable. This paper isn't really meant as any sort of "gotcha", rather serve as a possible basis point for future work. Though with a caveat I'm still digesting the paper myself.

No worries! HN does not have the best notifications for responses. Similarly, I'm responding fairly late(funny how that happens).

I see you guys have closed off candidates in your job posting. You all do seem like a lovely crowd. I myself am a bit too junior(though not inexperienced) to be a founding/senior member(I read the job title incorrectly the first time), but if you guys end up opening another position, I would definitely love to apply.

Are you guys open to python experience or is it strictly ruby?

lol you know how it goes :). The mind does what it does.

Sorry all, I misused the word finite state. I meant it more from a combinatorics viewpoint(e.g. we only have X amount of operations per Y interval). You could consider my solution to be brute force code.

Abstractly I do this:

  read_file()
  lines = read_lines()
  sum = 0
  while lines:
    left = get_first_num_forwards(line)
    right = get_first_num_backwards(line)
    sum += integer(left+right)
  return sum

I define get_first_num() something like this:

  get_first_num(line):
    lowest_index_pair = None
    for key,val in dict.values():
       get_index_of_key_if_exists()
       if_exists: update_lowest_index_pair()
    index,num find_first_instance_num() //just gets the first num that appears
    update_lowest_index_pair()
    return lowest_index_pair[1]//just returns the number

Basically the idea is very similar to yours. We parse each line 11 times in both direction(10 per the word_vals dict and once more to find the index of the first numerical) which is only 22 parses. Then we grab the minimum index from this list and concat with the opposite side.

I just don't do any replacements at the cost of a longer run time. But I figure the cost of 11 parses was low enough that it wouldnt impact the run time significantly for this exercise.

The key point is that overlaps are not an issue because we check for string comparisons in the methods

Without going into spoilers, its interesting that people jumped to regex to solve this. For me that was a fairly non intuitive when I first saw the problem(both parts).

What jumped to me is the problem statement indicated a finite number of states and I crafted a solution based on that information. But its really cool to see how we all jump to different implementations.