gabriel

Joined 19 years ago

Karma

412

Submissions

Member since

Feb 23

Decades later and still trying to figure out how Linux works.

email: munozga `at' gmail.com

twitter (read-only, but I do see DMs): https://x.com/gabriel__munoz

github (sigh, the new sourceforge of the Internet--I told you so, years ago, that this would happen): https://github.com/gmunoz

[ my public key: https://keybase.io/gmunoz; my proof: https://keybase.io/gmunoz/sigs/hKq4BVo6xTRRixRxH4pP2GfWvggRinGOAXDDK91YfYE ]

View on Hacker News

Recent submissions

Comment · 1 year ago

I would back it up a little bit and say that any EDR thing would be capable of observing the source of the functions that a program will run and detect outliers. It's a great program to write, everyone should give it a try! It can also be unexpectedly complicated to get all of the corner cases right and you'll drive yourself mad once you try to think of the ways your detection method can be circumvented.

Comment · 2 years ago

Go deeper. You're right on the cusp of it since all of your questions are fantastic. Even digging into one of your questions will bring up highly relevant material for understanding how a Linux system functions.

    > The sed command is not explained at all. Even the description is non-sense if you don't know already what they are talking about. What is this "default directory", why do you need to set it, why there and only there, why that command works? Even the command itself is something, which I need to check the manual what the heck it does, because it's not a simple one.

By "default directory" they just mean that the upstream GCC source code has a file with default variables and they are modifying those variables with the sed command to use $prefix/lib and $prefix/usr/lib instead of $prefix/lib64 and $prefix/usr/lib64, e.g. lines that contain "m64=" and replacing "lib64" to be "lib". This is what sed is used for: To make string substitutions based on pattern matching. Think through ways of writing the sed command and testing on your own file to see how it behaves. This will lead you to more tools like diff, grep and awk.

    > > -enable-default-pie and --enable-default-ssp
    > The description is almost unusable. So we don't need it, but it's "cleaner", which in this context means exactly nothing. So what happens if I left out? Nothing? Then why should I care?

Go back and re-read the sections up to this point. Make note that you're in Chapter 5, which is the bootstrapping phase for the toolchain cross-compilers. Then look into the features that are mentioned. You can see descriptions by running the ./configure --help most times or looking up the GCC documentation. Those features are for security purposes and if you put that in perspective of the bootstrap phase they aren't needed if the only purpose of the temporary GCC binaries is to compile the final GCC in a later phase. To your point, perform an experiment and enable those features to see if there really is a difference other than time spent to compile. GCC incrementally adds security features like this and they are a big thing in hardened distributions.

    > > --disable-multilib
    > Okay, it doesn't support "something". I have no idea what is multilib, or why I should care. Basically the next arguments' description tells me that because it wouldn't work the compilation otherwise. And then..

A great feature to look up! Check out https://gcc.gnu.org/install/configure.html and search for the option and you'll find that it has to do with supporting a variety of target calling conventions. I can see how that'd be pretty confusing. It has to do with the underlying hardware support for application binary interfaces that GCC can utilize and it turns out you probably only need to support your native hardware (e.g. x86_64). That is, you're compiling your system from scratch and it'll only run on your native hardware (x86_64) but if you were a C/C++ programmer maybe you'd want to have support for other hardware (64-bit ARM systems are pretty common today as an example). So you can save time/space by disabling the defaults and honestly the defaults it includes are just not all that relevant on most systems today.

    > > --disable-threads, --disable-libatomic, --disable-libgomp, --disable-libquadmath, --disable-libssp, --disable-libvtv, --disable-libstdcxx
    > But why would they fail? I want to understand what's happening here, and I need to blindly trust the manual because they just tell me, that "they won't work, believe us".

Try it and find out. I would expect that they would fail due to reliance on other dependencies that may not have been installed or included in this bootstrapped build. Or maybe be/c those components don't behaved well with the LFS-based bootstrap methodology and ultimately aren't needed to bootstrap. Sure trust the LFSers but also think through a way to test your own assertions of the build process and try it out!

    > > --enable-languages=c,c++
    > Why are these the only languages which we need? What are the other languages?

GCC supports many language front-ends. See https://gcc.gnu.org/frontends.html. Only C/C++ is needed be/c you're bootstrapping only C and C++ based package sources. You can validate this as you build the remaining sections. It's conceivable that if you needed other languages in the bootstrap you could include them.

    > So at the end, descriptions are not really helping to understand what's happening, if you don't know already. The last time when I started LFS (about 10 years ago), that was my main problem. That you already need to know almost everything to understand what's really happening, and why, or reading manuals, or trying to find basically unsearchable information (like why libatomic compiling would fail at this step). So after a while, I started the blind copy-pasting, because I didn't have the patience of literary months, and when I realized that this was pointless, I gave up.

It's a steep learning curve, especially of a bootstrap which by its nature is circular! tbh, that's sort of the utility of LFS, it can take you up to a certain point but there are so many options and pitfalls in building a Linux system (or really any complex piece of software) and the real usefulness is pushing through, picking something and learning about it. Then using what you learned to apply to the unknown. GCC is one of the most important packages too, so there's a lot to unpack in understanding anything about it, but the impact is large.

Comment · 7 years ago

There's also this: https://0xax.gitbooks.io/linux-insides/

Comment · 8 years ago

I believe the premise of the book is to expose the reader to an element of style that bridges different programming environments. Thus, "style" here is meant more in the terms of a restrictive technique to allow an exposure of these techniques, not writing in idiomatic, or latest-and-greatest Python.

I'd suggest this book even to experienced engineers to focus on different types or a style of programming that he/she may not be familiar. And when you recognize some of the styles it'll be a re-enforcement technique that you were doing something that is seen over and over again (much like a pattern language). I recall going over styles that are seen more in systems programming (like C). The book will contribute to an understanding of code architecture styles across different languages.

For the new engineer, I'd also highly suggest this book as you may begin to see a variety of ways to solve a problem. And I would not concern myself with idiomatic Pythonic styles as that will come with time.

Comment · 8 years ago

Lisp in Small Pieces: https://pages.lip6.fr/Christian.Queinnec/WWW/LiSP.html

An excellent and very fun read about compilation, interpretation, and programming semantics.

I don’t think you already have to know LISP, although there isn’t an introduction, but if you know any programming it would be enough to pick up the programs that are used as examples. Source code from the book is available and you will end up programming as you move along and it is a delight to see yourself make a program that can interpret another program.

Comment · 9 years ago

I think you mean Inline::Perl5: https://github.com/niner/Inline-Perl5. It doesn't parse Perl 5 and run it inside of Perl 6 though, it just runs the Perl 5 interpreter for you from a Perl 6 program and lets you marshall data between the interpreters. It's a legitimate technique.

Comment · 10 years ago

Thanks! I certainly will. It looks pretty cool. I'm certainly learning it and finding the technical details in the documentation to be quite interesting.

Comment · 10 years ago

I kind of agree with the assessment.

I've been prototyping with VoltDB be/c it has a pretty interesting model that should be able to achieve a near-linear scale of write operations for tables that are partitioned. After reading the docs on VoltDB [1] is became clear to me that they are putting the design constraints up front and if you can work through those constraints [2] you can achieve some wicked scale. But it's a bit more complex than your typical single host database.

The work that VoltDB makes you deal with up front are a lot like the work that would have to be done for a multi-master setup in PostgreSQL to function correctly. I like how VoltDB puts those problems up front, but I'm having problems seeing VoltDB as a general purpose solution. The old PG database I work on right now I can't see in the VoltDB, but maybe parts of it would fit OK.

I look forward to the tooling in PG to get better and better. It's a great community and I do like the work that 2ndquandrant is doing. I like how they approach the community with their work. I do the BDR work is important to understand so that when you're in a situation that calls for it you can take advantage of it (same with VoltDB).

[1] https://docs.voltdb.com/UsingVoltDB/IntroHowVoltDBWorks.php#... [2] https://docs.voltdb.com/UsingVoltDB/DesignPartition.php#Sche...

Comment · 10 years ago

A good read for API design is "The Little Manual of API Design": http://www4.in.tum.de/~blanchet/api-design.pdf

It falls into the category of non-web API. It's a good read and I think it carries over to web APIs quite well.

Comment · 10 years ago

Yep, I think that's a pretty big part of the learning process. Realizing that you had a successful build but some things weren't stable. Naturally this would pose the question: What went wrong? To solve that problem you have to figure out a way to debug the system, make a decision as to how to rectify it, apply a fix, then figure out how to verify.

Now imagine the effort that has to go into making Debian work or what if the issue was related to sporadic kernel crashes. It's a lot of fun if you're into that sort of thing.

Comment · 10 years ago

My first LFS was around 2001, which I think would be LFS v3.0. I can't begin to express how amazing the project was to my learning and development as a programmer.

When I started I was completely new to Linux and the command line. Everything I do today is rooted in the foundation that I learned by doing a close read of the book. The book requires no coding, just command-line, but soon after my first and second reads of the book I began to automate the build process that took me down the path of understanding how distros are put together.

I've only read the book end-to-end once or twice (I think that's all that is needed), but I've done hundreds of builds. Make no mistake: The book is difficult to get through. It requires trial and error as well as a close read of the book to understand why the build is done in the way that it is, which should be explained thoroughly, but if it isn't I'm sure the editors want to hear about the difficulty you are having. There are also bound to be problems bootstrapping from your <insert_favorite_distro>. That's always been a challenge and it's part of the build methodology that LFS uses to create a stable foundation to build a proper system (which relates to the efforts today regarding bit identical reproducible builds).

Comment · 10 years ago

I work with about a 600 table 2 TB PostgreSQL database and I started to use DataGrip, when it was in beta, for constructing complex queries. It's quite good for writing larger queries, but also taking auto-generated queries from an ORM and re-formatting them to help with debugging application behavior.

For db-specific operations I still use PostgreSQL's command-line client psql. It's an invaluable tool, but it's really nice to have options.

I also like the ability to pin query results so that I can quickly compare result sets in slightly-modified queries that I'm trying to fix.

Not so great at exploring the schema. Could be that I just didn't learn the tool enough to make that easier.

Comment · 10 years ago

I'd go for an invite if you hae some still. Email in profile.

Comment · 10 years ago

I'd take an invite as well if anyone sees this, thx :) edit: Email in profile

Comment · 11 years ago

I'm reading "Lisp in Small Pieces" by Christian Queinnec, which is a book that incrementally builds a scheme interpreter with increasing functionality. I could get away with using Guile to run the earlier examples, but by Chapter 3 I had to use one of the lisp interpreters that the author suggested in his code repo (https://pages.lip6.fr/Christian.Queinnec/Books/LiSP-2ndEditi...). I ended up settling on SCHEME->C and this presentation reminded me of this project and associated work by Joel Bartlett. Checkout the debian code repo (sudo apt-get install scheme2c scheme2c-doc) and the research report for the project: * https://github.com/barak/scheme2c * http://www.hpl.hp.com/techreports/Compaq-DEC/WRL-89-1.pdf

If this presentation appeals to you at all I highly recommend reading Lisp in Small Pieces. It provides a great balance between implementation and theory that you get something running in every chapter and lots of recommendations as to where to go if you want more information.

I'm about half way through and I look forward to my 2-3 hours every night that I spend on it.