nickwatson

Thanks for the feedback, hoc. I am already re-coding the whole thing to use more of an Agents/crew style approach where certain roles can be assigned to critique the research, fact-check, etc.

I re-built with CrewAI but was limited so now doing the same again but using AutoGen.

The criticism is important to this, as is any research project. That's why good research works, if it is challenged. I'm implementing that "challenging" directly into the LLM as separate agents.

Balancing token efficiency with regular check-ins and critique is important, and something I want to ensure the user has control over.

Nearly to version 0.1.3, and much to learn still.

I loved that use case! I will keep testing, refining and hoping others will do until we get there. Persistence is what matters, to try our best and strive towards collaboration, openness and accuracy.

Obscurity is dispelled by augmenting the light of discernment, not by attacking the darkness.” - Quote by Socrates.

Thanks for all the thoughtful feedback today, everyone.

I’m logging the ideas (grounding, source ranking, etc.) and will open issues tonight.

Heading offline now but I’ll circle back tomorrow. Feel free to keep the questions coming!

I updated the Readme now, to describe as "CleverBee: AI-Powered Online Data Information Synthesis Assistant" and put emphasis on the synthesis.

Also, put a new section in place:

What cleverb.ee is not Cleverb.ee is not a replacement for deep domain expertise. Despite explicit instructions and low temperature settings, AI can still hallucinate. Always check the sources.

It does it several times, so maybe re-re-re-search works?

Very true, and it got me thinking a lot.

As humans, we align to our experiences and values, all of which are very diverse and nuanced. Reminds me of a friend who loves any medical conspiracy theory, whose dad was a bit of an ass to him, and of course, a scientist!

Without our cognitive biases, are we truly human? Our values; our desired outcomes inherently are part of what shapes us. and of course the sources we choose to trust reinforce this.

It's this that makes me think AGI can never be achieved, or human-like ability for AI to think, because we are all biased, like it or not. Collectively and through challenging each other, this is what makes society thrive.

I feel there is no true path towards a single source of truth, but collaboratively we can at least work towards getting there as closely as possible

I'd like to humor you a bit on what you say (going off on a little tangent here).

- Were all the existing sources (e.g. news, podcasts, etc) ever reliable? - Do people lobby for certain outcomes on some research/articles?

And finally...

- Now we know LLMs hallucinate, and news can easily be faked, are people finally starting to question everything, including what they were told before?

Of course, mostly rhetorical but I think about this a lot - if it is a good or bad thing. Now we know we are surrounded by fakeness that can be generated in seconds, maybe people will finally gain critical thinking skills, and the ability to discern truth from falseness better. Time will tell!

For now the way I see it is people are becoming reliant on these tools, and only will a community of people collaborating to better the outcomes can ensure alternate agendas do not lead the results.

I'm very curious. As it's main purpose is built to really search and decide balanced/unbias sources and not really have an opinion, what the result would be of such a question. I'm curious if it will give an answer on this. I just gave it the request:

"I am struggling what to call this other than "Deep Research tool" as really it is looking online and scanning/synthesizing sources (that's you, by the way!). With that in mind, someone suggested "literature review" but it makes me think of books. I wonder if you can see what this kind of "research" is and suggest a name to describ it based on all the information you uncover on what good research looks like."

Let's see how it gets on...

Also, something I think about a lot (you sound like a deep thinker!) - when we discover something that is untrue, can it make it true? (purely hypothetical thought)... if 1000 people were told coffee was bad for them, does the mind-body connection take over and amplify this into reality. We are certainly in interesting times!

I agree, LLMs have biases. It was my primary desire to build this tool and put the weight on the LLMs to synthesize rather than think about and interpret the subjects. It's actually the main goal of this tool - maybe I don't articulate that as well as I could be - I'm open to suggestions here!

I agree to first figuring out which research is most important and reliable. There is a planning stage, to consider the sources and which ones hold credibility.

In addition, the user has full control over the sources the tool uses, and even add their own (MCP tools).

In addition, being open source, you have full control over the flow/prompts/source methods/etc and as a result can optimize this yourself and even contribute improvements to ensure this benefits research as a whole.

I welcome your feedback, and any code amendments you propose to improve the tool. You clearly understand what makes good research and your contributions will be highly valued by all of us.

At the moment the measures taken are:

- Full content analysis by Primary LLM (Default is Gemini 2.5 Pro) with link hard-coded alongside each piece of content with structured output for better parsing. - Temperature right down (0.2), strict instructions to synthesize, precise prompts to attribute links exactly and without modification.

What I hope to introduce:

- Hard-coded parsing of links mentioned in final report to verify with the link map created throughout the research journey - Optional, "double-checking" LLM review of synthesized content to ensure no drift. - RAG enhancements for token-efficient verification and subsequent user questions (post-research)

Do you have any further suggestions?

Right now I hope to strike the delicate balance between token efficiency, with enhanced grounding as optional settings in the future. I have a big task list of things, and this is one of them. I will ensure to re-prioritize alongside user requests for the different features.

Of course, being open source, contributions are highly welcome. I would love to see large community involvement. Collaboration benefits everyone.

P.s. I have spent hundreds of dollars in tests. I'd say for every 1 hour of building, about 3 hours of testing have gone into this, debugging, optimizing quality, ensuring guard-rails are in place.

If you go to the repo, also check out the config/prompts.py file - it will give you a little more insight into what is going on (there are code checks as well, but generally it gives you an idea).

I agree. In all honesty I was just following on the trend that has been popularized by OpenAI/Google so it is more relatable but will mention the "literature review" as you suggest, it's a good idea.

I didn't give the wording too much thought in all honesty - was just excited to share.

Where would you suggest to put the literature review text? Readme.md?

What about something like "synthesized findings from sources across the internet" or something like that.

When I see the word literature, I immediately think of books.

Fixed, and pushed to the repo.

If it hits a timeout but the content is still there, it will now return the content (without the error message) to ensure something small like this doesn't get synthesized.

That's the thing about sytnthesizing and I feel one of the strengths of this system, is the LLM doesn't put too much of it's own "creativity" (which usually comes with hallucination) on to the top of the findings!

TL;DR: Fixed.

I never said "heavy reliance" on Reddit/YouTube. It actually is requested to use discernment to recognize poor, or biased sources and opinions, and label them as such (see the example report on Coffee which I shared previously in another comment).

Most the time it has only sought out one or two post/youtube videos as it can recognize the low credibility value.

It comes loaded with a PubMed MCP tool and the beauty of it being open source is you can exclude or limit the sources as much as you want, or add in new sources - that's why I wanted to open it up, to allow for critique over methodologies and allow for improved, balanced research from experts.

It is also requested to evaluate the source and whether or not they have "some benefit to gain" from the article, to ensure it balances this into the research, also.

I felt they would just "cast the net wide" with a quick search-collect at scale, then load in the LLMs own training on to the top, and reports I generated were giving me hallucinated content.

I wanted something more - collect-evaluate-decide loop to iterate through discoveries and actively seek out diverse sources.

Good spot! I already am on the case with that one.

The Rush University website takes a long time to load (check it out), and the script recognized the article had loaded but something on the website was causing it to hang (waiting for networkidle status) so it terminated the parsing early and worked with the content it had.

So in a nutshell, it still parsed but with a warning/error that this happened.

I'm optimizing this now to exclude the wording from the report, or make a note.