This is unfortunately the status quo of nutrition research; A long-term RCT is the "gold standard", but it is exceedingly difficult to recruit subjects and ensure their compliance over meaningful periods of time. Which is part of why constant flip-flopping about whether something is healthy or not is almost a trope in journalism. Nonetheless, a few principles have been well established: vegetables are good, fruit is (mostly) good, refined grains and free sugars are bad.
The issue with personal experiments is often that they are just as biased and cannot be conducted over meaningful time-scales. As an anecdote to illustrate this, I am significantly more productive and energetic when consuming a single sugary, chocolately coffee, but it would be foolish to conclude over such a short period of time that my personally ideal diet should include sugary coffee. I'm not deluded that this is a healthy practice, however; free sugars, fructose in particular, are demonstrably a major factor in the pathogenesis of lifestyle-related diseases.
The issue with personal experiments is often that they are just as biased and cannot be conducted over meaningful time-scales. As an anecdote to illustrate this, I am significantly more productive and energetic when consuming a single sugary, chocolately coffee, but it would be foolish to conclude over such a short period of time that my personally ideal diet should include sugary coffee. I'm not deluded that this is a healthy practice, however; free sugars, fructose in particular, are demonstrably a major factor in the pathogenesis of lifestyle-related diseases.