IsaiahShiner
Joined 8 years ago
Karma
47
Submissions
19
Member since
Nov 16
Recent submissions
Comment · 6 years ago
Comment · 6 years ago
Hmm, I can see how I'm being confusing. This is partially trying not to give information overload with poorly written explanations, especially for features which I haven't added yet. It's also because dit is really trying to solve everything with Perfect Data all at once, and I need to go back and be clear about when I'm talking about which concepts. To solve Perfect Data, we need a single system that can do:
- Validation: This is for when you ask for a name and get "4jZw3ef\n". This already exists in database constraints, APIs, and, as you said, in things like JSON-Schema and Avro. The key difference is that the 'schema languages' are not languages. They don't have full flow control, libraries, classes, not even simple string functions. Dit does, by proxy of allowing for embedded languages, which let you do whatever you want.
- Schemas: Obviously other schema languages already do this. This is covered by dit classes. To solve the "FirstName", "LastName" issue, you just make sure you're using a Name schema like the one I demonstrated, in which Name contains a FamilyName/GivenName. But then everyone is using different classes for the same data, which you can solve by...
- Global Online Schemas: The only one really doing this is Schema.org. Everyone can agree that https://schema.org/Person is a reasonable official schema for a person. However, they have no validation, and changes are decided by a consortium, not free change. DitaBase.io would have a sort of copy of GitHub, in which each class is it's own page, and doesn't explicitly belong to anyone. Using meritocracy, democracy, and compromise, the community would agree on which exact schema of Person is best. If someone disagrees, they can make their own style and try to explain why it's better, while offering easy conversion between their style and the old style.
- Format Support/Universality: This is what I was getting at when I said "you can't write data in dit". Dit cannot afford be locked into one technology platform, the way JSON-Schema is locked to JSON, and the Semantic Web is locked to HTML. How do I write down data when you ask for it outside of a dit? In .json, .csv, .xml? Why not .yaml, .toml, .xlsx or .gsheet? There's no way to serialize data raw data as dit, only take data written in a .dit and print it some other way. The print functions and query system should allow you to request data in any format you want, including more than one. And ultimately, dit could never provide all of the features people could want. https://ipld.io/ is basically doing a JSON-Schema style of thing, specifically for the block chain. Dit isn't going to replicate their custom block chain features, so instead dit should support IPLD, by allowing data to be read and printed in their format, and converted every which other way you can think of. I mentioned DitaBase having it's own object model, but the truth is this would almost certainly reference Schema.org, at least by extension.
Perhaps the thing I was least clear on overall is that dit is still an evolving concept. I know where dit is now, and I know where dit wants to go, but I'm not really sure how to get there. Maybe assigners are a terrible way to solve that specific problem. That's fine, we'll throw them out and make something better. All I know is that Perfect Data needs to be solved, is solvable, and dit has a much better chance of solving it than any other existing solution.
Comment · 6 years ago
Ah, this? https://news.ycombinator.com/item?id=11945722
Super interesting! It's like, data, without an interpreter, has no meaning. We rely super heavily on humans as general purpose interpreters, and when we design computers, we make sure as much of the meaning as possible is implicit.
Rich was saying that the interpreter is secondary, which is only true if data serves a single purpose. Alan is getting at the idea that data could serve every purpose if only the interpreter wasn't implicit. If the concept of "data" referred to both the value and the meaning, the language and how to interpret it.
But, from a reddit comment: An interpreter doesn't magically give meaning to data, since the interpreter has to translate to something, and that something has to be written by a program that understands both the from and to sides of the conversation.
The point of dit is to literally be that universal language. Yes, you do need both sides to explicitly lay out what they main, both to convey and understand the meaning. The difference is that the number of connections is unlimited. You need a human for that critical first step, but then the machine can speak "dit" language to any other machine that speaks dit. Rich couldn't conceive of the idea that dit could be real. I think it can. I think we can make Licklider's vision, data and programs that can move around without respect to the original developers intentions because they all speak dit.
Thank you for sharing, super interesting!
Comment · 6 years ago
Dit is ready!
About 4 months ago, we posted just a website to HN: https://news.ycombinator.com/item?id=21263839
Now we're launching the MVP and hope you'll join us in trying to make data much better!
For a primer on what this is all really about, I wrote quite a bit on this exact topic: https://github.com/isaiahshiner/dit-cli/blob/master/docs/wha...
Comment · 6 years ago
Hello everyone!
Its been 2 and a half months since I posted https://news.ycombinator.com/item?id=21263839
I've made a lot of progress. The most significant thing is certainly the demo! (https://www.ditabase.io/demo.html) I only knew vaguely how the actual product would look, and the demo forced me to put it to code.
This resulted in two important changes: 1. No more schemas, only objects 2. Dit is the real product
Objects take the place of schemas in every way. I can no longer picture what exactly you would want out of a schema that you can't get with an object. They cover schema relationships, and because of the way scripts work, every object can use a different language and it doesn't matter. Schemas seemed like a headache, but object orientation is very simple.
Then, the only logical delivery mechanism is a single file type like dit. Whatever you want that's related to DitaBase can be neatly packaged in a dit. That includes all the things I know about now and all the features I might want to add in the future.
The very next thing is certainly the MVP. This will consist of a CLI Linux app that validates dit files in much the same way the demo does. I am optimistic that I can release that by February 1st.
I am looking for feedback almost exclusively on the demo, and whether it makes sense to you as a first product. What projects can you picture using it for? What features seem make or break to you? Do you still think it's a terrible idea? I want to hear it all.
And to be clear, I realize that many of you are probably not my first users, but you are developers. In the future, I hope all of you will be users, so your feedback is still important.
Please let me know what you think!
Comment · 6 years ago
Hey everybody! Looking for feedback on the essential components of the product. I would love to hear all your ideas on how dit could work differently, and how I can make it better!
Comment · 7 years ago
For block length, if you can't use context or control character/symbols, then the label would be required to specific the length of the payload in bytes, or similar. When you reach the end of the given length, then the next token should be the closing payload tag/brace.
The important thing here is that everything gets described. The structures within a field, such as label, meta, validators, etc. always follow the specification set by the previous label. That label then sets the specifications for the next payload. And since the validators and such are Turing complete, the specifications can be anything, rebuilt within dit schemas.
Comment · 7 years ago
Let me see if I can boil down your comments to specifc questions that I can answer.
"Is DitaBase compliant with JSON-Schema?"
The answer to this question is critical to understand about how DitaBase is different: if you want it to, then yes! The validators and converters are fully Turing complete, and not limited to Javascript either. If you can make a language interpreter in C/C++ (think Javascript V8), then you can use it with DitaBase. So whatever format you want, just rebuild it in (or port it to) dit schemas. Therefore, it isn't possible for DitaBase to not be compliant with something (at least, as far as I can think) because if someone thinks its important, they'll rebuild it in dit schemas.
"Schema.org is just documentation. Are there programatic, queryable schemas?"
Yes, but only optionally. In theory, all that is required to be a dit is a validator. You could inherit from "thing" and just directly have some text and a validator. But if you want to use Controllors (CRUD for a specific schema), Containers (abstract the shape of the data from the content) and other things like that, then yes, you need programatically defined schemas, just like in J-S. If someone has them, then you can use them. If not, in theory, if their schema is public, then you could just make a pull request with the relavent additions. DitaBase is very open compared to something like Swagger.
"Uh, anonymous functions? Duh?"
Oh, yes that makes much more sense that the ridiculous names I was giving those in the example. That actually just never occured to me. Of course these things will have to get slowly fleshed out over time, and I suppose there's no reason you couldn't have both.
"GraphQL, BSON, custom tools, cool things"
This is the great thing about DitaBase. Since you know the exact structure of any dit, and everything about dits is generic, you can just build all of this on top, without any overhead. DitaBase will almost certainly build some things like this in house, especially something like GraphQL, and some data analysis and statistical plotting utilites. Or perhaps, similar to how VScode absorbs open source projects into itself when they get really popular, DitaBase could probably do the same.
Comment · 7 years ago
You are correct, DitaBase is in some ways extremely similar to Linked Data/Semantic Web/RDF etc. In fact, I would say that DitaBase is, in some ways, just another attempt at making the Semantic Web deliver on its original promise. DitaBase obviously couldn't exist without HTTP.
"Does dit use RDF?"
The answer to this question is critical to understand about how DitaBase is different: if you want it to, then yes! The validators and converters are fully Turing complete, and not limited to Javascript either. If you can make a language interpreter in C/C++ (think Javascript V8), then you can use it with DitaBase. So whatever format you want, just rebuild it in (or port it to) dit schemas. Therefore, it isn't possible for DitaBase to not be compliant with something (at least, as far as I can think) because if someone thinks its important, they'll rebuild it in dit schemas.
The reason DitaBase will succeed where Linked Data failed is by making DitaBase something that people in their day jobs want to use too, because the tools are just so good. Right now, a company with a data problem is extremely incentivized to seek a solution that solves only their specific problem. Obviously, because it's cheaper. But end-to-end solutions usually are not compatible with things like Linked Data, which needs solutions to be more general.
If a company sees that the best solution to that problem is becoming compliant with DitaBase, then obviously they're going to do it. But surprise surprise, now they're compliant with Linked Data as well.
Comment · 7 years ago
Hey! Well, I can tell you that as of right now, it's hovering around 0.8% conversion, which isn't terrible. We'll see if I can improve that before activity on the page dies down.
You're right, what I have now is relatively messy. I was balancing brevity and specificity. If I was properly technical in my descriptions, it may have read like documentation. And then the other issue is that I have both business people and programmers on the page, and each gets kinda confused because I'm not sure who to cater too. Same problem with the case studies. I had them more specific before, but I cut out what I thought was unnecesary and now they appear rather simple.
I think the solution is to split the whole thing up into several dedicated pages, which I resisted because it would take longer, and this is all probably less important than building product right?
Regardless, I do now intend to add an FAQ, and write some sort of a technical paper/RFC, (which I should probably do anyway before I run off and start writing serious code). I'm not totally sure what to do about the case study, maybe just replace the entire thing with a video?
I would love to hear more about the project you tried! Naturally, I haven't been able to Google examples of people who tried this idea before and failed. It does sound different in that, I really do intend dits to be fully, totally and completely generic. Nothing other than the idea of labels with scripts which describe payloads is sacred. Any language, any style, anything. This means that it can't possibly miss a certain use case (you can just build it) but it does mean that the whole thing is rather complicated, and as others have said, maybe too complicated, like a pipe dream. I am certainly willing to accept that, but no way to find out the result of that without building it.
I really question whether a more restricted system like JSON-Schema could ever become dominant because it's much more prescriptive, especially when you have a limited number of possible data types. In my mind, data can truly look like anything, so coming up with a rigourous standard doesn't seem realistic: there can always be a more broad standard, ala https://xkcd.com/927/ as others have mentioned. I intend to beat 927 by taking the high ground. Its like Lisp: you can't make a more generic Lisp, because anything that satifisfies the components of Lisp is just a dialect. You can't have incompatible data or an incompatible competitor to DitaBase, its totally generic. I also don't see why you couldn't do UI generation with DitaBase using the controllers (I concept I have not explained yet, they allow CRUD for a specific schema).
I am not very knowledgable on Twitter, but now seems like a great time to start! I appreciate the invitiation!
Comment · 7 years ago
An example dit is now live at https://www.ditabase.io/example.html
> How do you make sure that both sides of communication run the same logic? This will probably require some way to send language configurations along with a dit. If you send someone a dit which requires Java code, but the client doesn't even have Java, the dit should be able to recover from this somehow, by automatically installing Java, creating the configuration, etc. I'm not exactly sure how this should work, but its one of those clearly solvable problems that I'm trying to leave be until I have a good example of the real problem.
> Is the code transmitted together with data? What is benefit of running this code on receiving side instead of sending side? Yes, the code is always tied directly to the data. Currently, source code must be sent, but in the future, it could pull from a link, or even run the validation offsite, via an API (important in case you want closed source classes).
In theory, there is no reason for the receiver to run the validation, (assuming the receiver trusts the sender to validate). But one purpose of dit is to prevent a message from ever being separated from it's meaning. If a client receives some data in a dit format they have never seen, they can learn how it works by looking at the dit. The validation code can even be used as ad-hoc unit tests when trying to process the data automatically. And also remember that querying (and converting, mutating, anything else that might be added) also requires the dit code.
> Do you serialize and store validation and query code on disk? Is this good idea? Currently, the modified code is regenerated every single time. You can see the code in /tmp/dit/. This is slower, but it does mean that the code and serialized data are always as new as possible. Caching could be an option, if done carefully.
> .old data that cannot be queried as well as new data. ...old dates cannot be converted to various date formats while newer can. I'm actually not sure what you mean here, I can't picture how this could happen. Could you explain?