OSDev.org

Posted: **Sat Aug 10, 2019 1:00 pm**

Schol-R-LEA wrote:Damn, I was in the middle of editing my post again, so I'm not sure if you saw the part about how grep works vs. how the IDE's search would presumably work. Sorry for that. The point about how regex compilers operate may be relevant.

Also, does anyone have any comment on my post about Oberon and its solution to the question?

I did a quick re-read, I'm assuming the edit was just the addition of the second part, and no fundamental changes in the first?

Given how often the edit problems happen to you, have you considered not doing this type of "changes" not by editing, but by posting? =)

Schol-R-LEA wrote: I think that much of this argument comes down to matters of perspective. LtG is thinking in terms of the high-level problem ("How do I find all instances of this function being called") while XenOS is focused in the lower-level ("what substrings in this file match the function name"). The former isn't really any 'better' than the latter, or vice versa, but they are not the same question, and do not result in the same answer.

As a user of a hammer, I don't care how it's made. As the manufacturer I do.

As programmers we're often playing both parts. I have a high level task I need to complete (the what), so I have to figure out how to do that (the how).

The further away I can stay from minutia, the happier/faster/more productive I am.

Schol-R-LEA wrote: I would add that calling grep 'brute force' is a bit misleading, at least in terms of how it performs the search. LtG seems to be saying that it's 'brute force' with regards to how it accomplishes the objective, though, rather than how the algorithms perform.

I'm not sure what the distinction between "how it accomplishes ..." vs "how the algo perform" is. Is the latter in respect to performance (ie. cycle count) and not how it performs it as in how to it does it?

By brute force I refer to two separate but close problems:
- it has to go thru the entire data set, when only part of it is ever relevant for any specific question
- going thru the entire data set is wasteful

Magic text doesn't allow for narrowing, it's impossible to just skip stuff. You don't even know where the next line starts (until you've read this line), let alone what it means.

To get results you need narrowing, and to do that with grep you give it (in practice flawed) understanding what that magic text means, to the extent that it gets you the results you want. Even though, at some point, the full semantic meaning was available (eg. the timestamps in logs I mentioned earlier).

Schol-R-LEA wrote: Still, it might be worth considering how a regex compiler such as grep(1) actually operates, as it isn't actually doing string searches in the usual sense at all (and I am using the word 'compiler' advisedly, as it actually generates code in a compile-and-go fashion).

I think the usual sense is that a string is just an array of bytes, and let's go thru them all would apply.

Schol-R-LEA wrote: Also, the point about IDEs is that they aren't doing a string search, either, but some sort of table or tree lookup instead - the IDE has already analyzed the code and is working with the data structures it already has.)

Not just the IDE, the compiler will have them also. My point is that all the tools related will need them, and thus create them, I'd rather share them and take full advantage of them. I often have to use grep in IDE's because they can't do the job fully, for instance with multi-repo, and it's us, the programmers that create these problems. Just because we might want to split a repo for organizational reasons doesn't mean the IDE shouldn't be able to "hyperlink" to the definition of a function.

Schol-R-LEA wrote: It is a good deal more sophisticated - and more resource-intensive - than either of you seem to realize. I know that this may not sound relevant, but my point is that the 'simpler' tool in question is anything but, and both of you are basing part of your argument on the same fallacy; even if you are both using it for opposing points, the fact remains that it is a fallacy.

I think I've been pretty vocal on it being resource-intensive, and more to the point, wasteful.

I also don't consider grep simple, at least from the user perspective. Solar said he has grep-fu, but I think making a tool (search) complex is not the right path, and I think Google proved that. (Is that similar to Godwin's law?)

Posted: **Sat Aug 10, 2019 1:04 pm**

LtG wrote:
Schol-R-LEA wrote:Damn, I was in the middle of editing my post again, so I'm not sure if you saw the part about how grep works vs. how the IDE's search would presumably work. Sorry for that. The point about how regex compilers operate may be relevant.

Also, does anyone have any comment on my post about Oberon and its solution to the question?
I did a quick re-read, I'm assuming the edit was just the addition of the second part, and no fundamental changes in the first?

Given how often the edit problems happen to you, have you considered not doing this type of "changes" not by editing, but by posting? =)

Considering that it just happened again (when I added the part where I asked if you had anything to say about my post concerning Oberon), you may have a point.

Posted: **Sat Aug 10, 2019 1:15 pm**

Schol-R-LEA wrote: This is in some ways in line with my own views, though I am coming at it from an even more radical angle (that the concept of files itself, both functionally and conceptually, tends to constrict how data is used, imposing mental burden and forcing a lot of unnecessary operations when retrieving said data).

I'm guessing my angle is similar, the thoughts I've expressed in this thread are slightly narrowed to try to remain within the scope of the topic, even though I think it's drifted a bit now.

I try to experiment with different ideas and see what could possible work, though in practice I haven't done any coding for my OS for months now. Basically I got fed-up with C++, and am now considering whether I should create my own language too, to make writing the OS simpler. Redesigning the world isn't easy =)

Schol-R-LEA wrote:
LtG wrote:Though all of that is orthogonal to GUI vs CLI/TUI, but I think relevant because trying to do all of that in a CLI/TUI isn't very feasible.
I think that XenOS's point is that this is exactly how they are doing it - that they don't use a GUI in the sense you mean, period, and only use graphics at all when they need to run something that is inherently graphical such as an image editor (and that even then, much of it is done using shell-launched tools such as filters, with only the end result being displayed after completion). This was a normal mode of operation for decades, it just seems unfamiliar today because it isn't how most people learn it now. Make of this what you will.

I think you're right. I also think we should do better.

I think CLI is a subset of a GUI, and therefore by definition less. I'm also greedy, if I can have more, I'd like more.

I also think that the CLI/TUI way is more batch oriented, like with grep. The way _I_ usually work with grep is to define the bare minimum, run it, if it's too much, I cancel it (ctrl-c), then narrow it. It's borderline interactive. Google on the other hand is very much interactive, and it's not running locally. I have a data set (eg. all data on my system, preferably (though I don't have a plan yet) all the systems I have access to), then I narrow down the search, be it emails and specific words in them or a year, or log files and a specific time range, etc. I know when I've narrowed it down enough, but grep doesn't really work well with that.

Also, whatsup with all the web apps and their in-tree builds. Node-modules directory, the build dirs, try to grep -r anything and you get results where a single line is trillion screen lines long. Not fun. So then you start descending further into the directories to avoid it, when practically you almost never want to grep the built artifacts.

I understand XenOS's point that you should be allowed to search all files textually, I'm not planning on removing that ability, I just think that 99.9% (scientifacally proven) of the time that's not what you want, you want something way more narrow. As a developer it's my job to figure out how to do that in a way that's pleasing for the user.

Posted: **Sat Aug 10, 2019 1:23 pm**

Schol-R-LEA wrote: Considering that it just happened again (when I added the part where I asked if you had anything to say about my post concerning Oberon), you may have a point.

I don't really have much of an opinion on Oberon. I've been aware of it for years, and read a bit about it, and mostly liked the basic ideas. Never really had the time to really try it out or get more deeply into it though.

I'm hoping my plan (yes, hoping about my plan, time for osdev in the real world is hard) is to get enough time to get enough done, so that after that I can go back and try all the different OS's (OS/2, Oberon, Amiga, etc) and see what I can pilfer. I used Amiga back in the day, but it's so long that I hardly remember anything besides having multiple floppy drives.

I try not to comment on things I have no real knowledge about, but would like to point out that I do appreciate your in-depth knowledge of the many things. I've noticed you often try to bring quite a bit of historical context (<-- see) to the topics, and I for one do appreciate it.

Posted: **Sat Aug 10, 2019 2:04 pm**

LtG wrote:You'd rather hand code the "context" every time?

No. If it's a thing I do frequently, I save the command(s) as a shell script.

LtG wrote:The tool wouldn't be significantly more complex, just some interface for allowing new formats to be understood by the tool.

That's exactly what makes it complex - you need to add something for every format you want to handle.

LtG wrote:I'd rather use the whole IDE, granted most (all?) of them are slightly bloated, but I'd still rather get all the benefits the offer.

That's a matter of choice and use case. For my use case IDE would not have the benefits I actually need.

LtG wrote:I simply objected to you comparing one tiny feature (grep) with the whole IDE, because you don't develop with grep alone, you probably use the CLI too, and vim, and a compiler and other stuff. So the bloat is still there, just pieced out, and possibly smaller.

Of course I need an editor and a compiler, but I don't need a GUI for either of them.

LtG wrote:So you do provide the context, with CWD, which is exactly what it is. Early on they realized it would be kind of neat if you could have an implicit context.

I can have implicit context as well. For projects where I use VIM I sometimes create a VIM script that does the cwd for me and opens some files. And there's my project context.

LtG wrote:Because you want control over it?

If I want control over it, it's not automatic. Your demands on the tool contradict each other.

LtG wrote:First your argument was that vim can handle large files, now it's that when it can't, those files are too large and it's the files fault. So by definition vim is perfect, because when it fails it's somebody else's fault.

You're mixing things up. I said that I had no problems when we were talking about filed of "a few MiB". And even those I rarely open with VIM or any other editor unless I have to. But several 100MiB or GiB in an editor is a doubtful way of solving some task.

LtG wrote:You call that simple? I call that convoluted. Assume it's a 1GiB log file, how many times does your proposed solution go thru all the data?

Once. Only the first filter stage goes through all the data. The second stage goes through the output of the first stage (which is significantly less, if I order the filters in such a way that the first one keeps the least amount of data), and so on.

LtG wrote:For a once off search of something I have to write a Perl script? And you don't find anything wrong with that picture?

I don't have to write a completely new Perl script for every single search. As I said, it's an alternative method. I'd write a Perl script if I have to deal with similar searches another time, just with different search criteria (date, time, function name...).

LtG wrote:Aside from that, it seems your argument seems to be that because it can be done in a CLI, then there's no reason for a GUI (or anything better).

That is neither what I said nor my line of reasoning. If the CLI has advantages over the GUI, then there is reason to use the CLI instead of the GUI. And I mentioned such advantages in my previous posts (less resources etc.).

LtG wrote:By that line of reasoning we could all use brainf*ck the programming language, I wouldn't. I want something better.

By my line of reasoning you would use brainf*ck if it has some advantage over other languages, for the particular use case.

LtG wrote:I usually have some high level task I need to complete, like fixing a problem, to do that I need to figure out why it happens, so I need to look at the logs, and at that point with CLI and grep I immediately end up dealing with minutia, instead of staying high level and asking the log what happened with a particular request, or what has happening at a particular time, etc.

If that suits your task then go for it. That's not my use case, though.

LtG wrote:If we reduce CLI to mean textual, then the title question of this topic is:
Can you live without a GUI (including browsers)? For me that's a no.
Can you live without a CLI? Yes. If the necessary changes were made I'd do so happily.

My answer would be the opposite. Again, for my use case.

Schol-R-LEA wrote:I believe that LtG's point (correct me if I am wrong) is that the structure of it as a text source file is implicit, rather than structured as (e.g.,) a syntax tree or some other data structure which can be rapidly traversed without additional parsing. The relevant point being that some IDEs - not all, or even most, but some - will have already generated such a structure upon loading the file, building a collection of annotated symbol tables of the various identifiers with metadata regarding what it refers to and the locations it appears in, a partial AST of the code (if the IDE is language-aware enough, whether internally or through an extension script), tries for auto-completion, pre-loaded buffers of source files of code which the current file references, etc.

Of course, but that's neither an argument In favor or against a GUI. I can use, e.g., VIM with something like YouCompleteMe and semantic completion, so it will do all these things in a pure TUI as well. Of course it needs more resources than just a plain text editor + grep. And it does make sense to go this way if one wants to work on a project and perform several operations on this AST. But that's not the use case I mentioned as an example what I use grep for. My use case was a single search, and for that I don't have to parse all source files into an AST, even those functions which are completely irrelevant for my search.

Schol-R-LEA wrote:While these operations do expend resources - execution time and memory - and may not be needed in a given session, they do speed these operations up from the programmer's perspective, as they are done in the background at points when the programmer's attention is elsewhere (this relates to the oft-debated assertion that mouse motions are in fact faster than keyboard commands, but the key commands seem faster because the time spent is mostly in recollecting of the command combination, while the time used on mouse actions is more of the muscle-memory variety and thus more evident to the programmer's perception). One can argue that the speed-up is illusory, but since the cycles and memory 'wasted' on them would otherwise be unused, they actually do provide a real gain in working performance.

If I have as many resources as I want and need to use such a tool, then of course it does not matter how many resources this tool needs. But if I work in a restricted environment - devices with low memory, working over a slow internet connection - resources do matter, and it might simply not be possible to run such things in the background. But again, this has nothing to do with the GUI vs. TUI question.

Schol-R-LEA wrote:I think that much of this argument comes down to matters of perspective. LtG is thinking in terms of the high-level problem ("How do I find all instances of this function being called") while XenOS is focused in the lower-level ("what substrings in this file match the function name"). The former isn't really any 'better' than the latter, or vice versa, but they are not the same question, and do not result in the same answer.

That's exactly my point. One day I might be searching for functions being called, and of course I can have some IDE that creates an AST which answers this question faster than grep. But on another day I might be searching for links in a HTML file. Of course I can also parse that first into some kind of markup AST, but I need a different parser for that. Yet another day I might be searching a log file, and there might not even be any parser for that. That's why want a generic tool that solves these tasks by reducing them to a common, low level task, and in a minimal, resource limited environment that generic tool might be all I can use instead of specialized parsers for every single task I might encounter.

Schol-R-LEA wrote:It is a good deal more sophisticated - and more resource-intensive - than either of you seem to realize. I know that this may not sound relevant, but my point is that the 'simpler' tool in question is anything but, and both of you are basing part of your argument on the same fallacy; even if you are both using it for opposing points, the fact remains that it is a fallacy.

I'm not saying that tools like grep use no resources at all. I just avoid unnecessary resource usage (such as displaying a GUI) when they can be used more wisely (such as the actual search task grep or any other program performs), and I don't have unlimited resources to do both.

Posted: **Sat Aug 10, 2019 2:59 pm**

XenOS wrote:
LtG wrote:The tool wouldn't be significantly more complex, just some interface for allowing new formats to be understood by the tool.
That's exactly what makes it complex - you need to add something for every format you want to handle.

That's not the same thing as complex. Plus, you already have to have it, otherwise the recipient of the data wouldn't be able to understand it. I'm suggesting we also share it with the search.

XenOS wrote:
LtG wrote:I'd rather use the whole IDE, granted most (all?) of them are slightly bloated, but I'd still rather get all the benefits the offer.
That's a matter of choice and use case. For my use case IDE would not have the benefits I actually need.

So you're not using any IDE? What are the benefits you need that IDE's don't meet? Have you tried good IDE's?

XenOS wrote: Of course I need an editor and a compiler, but I don't need a GUI for either of them.

So a CLI that brings compiler and editor together and an IDE that brings them together with the same RAM/CPU footprint is bad? If it makes even one thing faster/easier/more convenient, then it's a win. After that, keep on piling the wins.

I'm not saying it can't be done on the CLI, but rather that we should strive for something better. And given that there are millions of programmers, we get something similar to compound interest.

XenOS wrote:
LtG wrote:Because you want control over it?
If I want control over it, it's not automatic. Your demands on the tool contradict each other.

No they don't. Tell me about a single Linux tool that doesn't depend on UI (config files, prog args, etc) to some extent. Of course I want control over a tool.

XenOS wrote:
LtG wrote:First your argument was that vim can handle large files, now it's that when it can't, those files are too large and it's the files fault. So by definition vim is perfect, because when it fails it's somebody else's fault.
You're mixing things up. I said that I had no problems when we were talking about filed of "a few MiB". And even those I rarely open with VIM or any other editor unless I have to. But several 100MiB or GiB in an editor is a doubtful way of solving some task.

You said there's no problems with vim and large files.

Solving a task by looking at logs of GiBs is a doubtful way? Large enterprises have large logs.

XenOS wrote:
LtG wrote:You call that simple? I call that convoluted. Assume it's a 1GiB log file, how many times does your proposed solution go thru all the data?
Once. Only the first filter stage goes through all the data. The second stage goes through the output of the first stage (which is significantly less, if I order the filters in such a way that the first one keeps the least amount of data), and so on.

I thought sed (to remove Java stack trace line-feeds) goes thru the the entire data set, then grep goes thru all of it again? Wastefully.

XenOS wrote:
LtG wrote:For a once off search of something I have to write a Perl script? And you don't find anything wrong with that picture?
I don't have to write a completely new Perl script for every single search. As I said, it's an alternative method. I'd write a Perl script if I have to deal with similar searches another time, just with different search criteria (date, time, function name...).

I said once off search, yes, I've created and used (created by others) "incantations" multiple times, but I've also been in the situation where I need to do it just for once. I've also deemed it reasonable to write a C program to go thru the data because it was easier than try to figure out how to get the same thing done with awk/sed/grep. Like parsing a routing table and streamlining it (combining subnets if the super-net has the same next hop, etc).

The routing "engine" already knows if they belong together, yet I have to do a poor mans version of the understanding/semantics to get the job done. I just don't have access to it, so I have to recreate it, and due to practical concerns I have to cut corners.

XenOS wrote:
LtG wrote:Aside from that, it seems your argument seems to be that because it can be done in a CLI, then there's no reason for a GUI (or anything better).
That is neither what I said nor my line of reasoning. If the CLI has advantages over the GUI, then there is reason to use the CLI instead of the GUI. And I mentioned such advantages in my previous posts (less resources etc.).

So do you use a Linux system without a GUI? If not, then the resources are already expended anyway, which is part of my point. Using a CLI on top of that doesn't save resources.

A search for "typoed" would still be written in text, but if I can be more specific (eg. pertaining to project X, within a std::string, etc), then I'd happily reap the benefits.

I'm saying a CLI itself brings nothing to the table. It only restricts in a bad way.

XenOS wrote:
LtG wrote:By that line of reasoning we could all use brainf*ck the programming language, I wouldn't. I want something better.
By my line of reasoning you would use brainf*ck if it has some advantage over other languages, for the particular use case.

But earlier you were using grep as useful in the _general_ case, not specific (particular).

XenOS wrote: If that suits your task then go for it. That's not my use case, though.
LtG wrote:If we reduce CLI to mean textual, then the title question of this topic is:
Can you live without a GUI (including browsers)? For me that's a no.
Can you live without a CLI? Yes. If the necessary changes were made I'd do so happily.
My answer would be the opposite. Again, for my use case.

In what way is it your use case? Specifics help me (us) to better understand what is needed from a system.

And honestly, replying to forum.osdev.org, can you say you don't need a browser?

XenOS wrote: Of course, but that's neither an argument In favor or against a GUI. I can use, e.g., VIM with something like YouCompleteMe and semantic completion, so it will do all these things in a pure TUI as well. Of course it needs more resources than just a plain text editor + grep. And it does make sense to go this way if one wants to work on a project and perform several operations on this AST. But that's not the use case I mentioned as an example what I use grep for. My use case was a single search, and for that I don't have to parse all source files into an AST, even those functions which are completely

YouCompleteMe does it exactly (more or less) as any IDE does it, now that overhead is acceptable?

For your use case for a single search you have to provide grep with the relevant context on _how_ to search for something. That knowledge already exists, you shouldn't have to duplicate it.

As for the AST, that too already exists (intermittently), why can't it exist all the time and be taken advantage of?

XenOS wrote: If I have as many resources as I want and need to use such a tool, then of course it does not matter how many resources this tool needs. But if I work in a restricted environment - devices with low memory, working over a slow internet connection - resources do matter, and it might simply not be possible to run such things in the background. But again, this has nothing to do with the GUI vs. TUI question.

How much are those resources? Are they unreasonable? For a "average" C++ project I'd guesstimate them to be in the order of MiB's. When dev'ing, I don't work over a slow Internet, so a moot point. I'm not suggesting we should dev on an IoT. My laptop has 64 GiB of RAM, my next laptop will have same or more.

Granted Google mono-repo (good or bad) has billions of lines of code, but I would argue that it's largely redundant, and also that their devs are able to deal with it with normal laptops. So it's a non-issue, even for one of the largest code-bases in the world.

In addition, because of structured code (as opposed to magical text), I would argue the size is smaller, the same names (variable, function, etc) don't need to be repeated in memory.

XenOS wrote: That's exactly my point. One day I might be searching for functions being called, and of course I can have some IDE that creates an AST which answers this question faster than grep. But on another day I might be searching for links in a HTML file. Of course I can also parse that first into some kind of markup AST, but I need a different parser for that. Yet another day I might be searching a log file, and there might not even be any parser for that. That's why want a generic tool that solves these tasks by reducing them to a common, low level task, and in a minimal, resource limited environment that generic tool might be all I can use instead of specialized parsers for every single task I might encounter.

HTML = "some kind of markup AST"

I'm suggesting only using "some kind of markup AST", so we reap the benefits. No parsing required. Resource constrained also love it.

XenOS wrote:
Schol-R-LEA wrote:It is a good deal more sophisticated - and more resource-intensive - than either of you seem to realize. I know that this may not sound relevant, but my point is that the 'simpler' tool in question is anything but, and both of you are basing part of your argument on the same fallacy; even if you are both using it for opposing points, the fact remains that it is a fallacy.
I'm not saying that tools like grep use no resources at all. I just avoid unnecessary resource usage (such as displaying a GUI) when they can be used more wisely (such as the actual search task grep or any other program performs), and I don't have unlimited resources to do both.

Avoiding "unnecessary" resource usage may be a form or premature optimization. By not using a AST every IDE/compiler/editor will have to understand C++ (to a point), instead of indexing your HDD you have to do a full brute force search (10 TiB vs. few MiB). I'm arguing paying the upfront cost is often simpler and more cost efficient.

PS. I'd also like to thank you too for continuing with this thread, even if it at times it might seem futile. At least we both have to consider both sides.

Posted: **Sun Aug 11, 2019 12:24 am**

LtG wrote:That's not the same thing as complex. Plus, you already have to have it, otherwise the recipient of the data wouldn't be able to understand it. I'm suggesting we also share it with the search.

The "recipient of the data" does not necessarily have to understand the input format. If I grep Youtube links from a HTML page and pipe them into a media player, the media player does not understand any HTML, so there is no use in creating a HTML markup AST. Hence, I don't need a HTML parser here at all.

LtG wrote:So you're not using any IDE? What are the benefits you need that IDE's don't meet? Have you tried good IDE's?

I've said this more than once: using less resources. I have tried Eclipse, KDevelop and Dev-C++. I don't know which ones you consider "good".

LtG wrote:So a CLI that brings compiler and editor together and an IDE that brings them together with the same RAM/CPU footprint is bad?

As I also already said multiple times: it is not the same footprint if the IDE requires a GUI (and hence some graphics framework, or at least a graphical desktop) while the CLI does not, and if the IDE does some work that the editor does not.

LtG wrote:If it makes even one thing faster/easier/more convenient, then it's a win. After that, keep on piling the wins.

Of course, if it makes things faster. But it does not make things faster if I have to power up a graphical environment just to find something in a file and change it.

LtG wrote:No they don't. Tell me about a single Linux tool that doesn't depend on UI (config files, prog args, etc) to some extent. Of course I want control over a tool.

But if I change the parameters every time or some config files, running the tool is not automatic anymore. And if I can control it on every run, then I can also adapt its task on every run. And if I can do that and interact with the tool on every run, then I can also inspect the input and output, which defeats your argument that one cannot "look and see" if the tool is automatic. So either I interact with it, or it's automatic (runs from a cron job or script without user input on every run).

LtG wrote:You said there's no problems with vim and large files.

I said I have no problems with large files in VIM when you suggested to open a file of a few MiB, which is true. Then you moved on to several GiB, and still I have no problem, because I would not even open the file in VIM in the first place.

LtG wrote:Solving a task by looking at logs of GiBs is a doubtful way? Large enterprises have large logs.

No, and that's again not what I said. I said that opening such a file in an editor is a doubtful way, instead of filtering it and extracting the data that actually needs inspection.

LtG wrote:I thought sed (to remove Java stack trace line-feeds) goes thru the the entire data set, then grep goes thru all of it again? Wastefully.

How would you parse the log file into some kind of AST without going through the entire data set?

LtG wrote:So do you use a Linux system without a GUI? If not, then the resources are already expended anyway, which is part of my point. Using a CLI on top of that doesn't save resources.

Yes, I do, and not just one of them. I have several Raspberry Pis, for example, all of them running Linux, most of them without GUI. Also some embedded devices with Linux and no GUI. And some HPC servers / nodes.

LtG wrote:I'm saying a CLI itself brings nothing to the table. It only restricts in a bad way.

I already mentioned the advantage - see above.

LtG wrote:But earlier you were using grep as useful in the _general_ case, not specific (particular).

Exactly. My use case is working with lots of different text input formats (including C++, HTML, LaTeX, Markdown, plain text, log files) from different sources and extracting data from them. So either I need different, specific tools for each of them, or just one generic tool which solves all my tasks.

LtG wrote:In what way is it your use case? Specifics help me (us) to better understand what is needed from a system.

Also this I have mentioned multiple times already.

LtG wrote:And honestly, replying to forum.osdev.org, can you say you don't need a browser?

Of course I need a browser on the computer which I use for accessing the forum. But 1. that does not need a GUI (reading a posting here works fine with text only), 2. I don't need a browser on every machine I use (such as remotely managed RPi or HPC cluster).

LtG wrote:YouCompleteMe does it exactly (more or less) as any IDE does it, now that overhead is acceptable?

It does so without any graphical environment, so also on those machines where I have no GUI at all. So a GUI would be overhead which can be avoided.

LtG wrote:As for the AST, that too already exists (intermittently), why can't it exist all the time and be taken advantage of?

If it exists it can be used, but it does not necessarily have to exist at all (see my HTML link extracting example above) because it is not needed.

LtG wrote:How much are those resources? Are they unreasonable? For a "average" C++ project I'd guesstimate them to be in the order of MiB's.

An X server + some graphical environment takes definitely more than a few MiB.

LtG wrote:When dev'ing, I don't work over a slow Internet, so a moot point. I'm not suggesting we should dev on an IoT. My laptop has 64 GiB of RAM, my next laptop will have same or more.

I have not suggested to do all development work or programming on IoT. Of course I use a good machine for that as well. But that was not even the task I was talking about. It was about finding something in a file and fixing it.

LtG wrote:In addition, because of structured code (as opposed to magical text), I would argue the size is smaller, the same names (variable, function, etc) don't need to be repeated in memory.

Still they have to be presented to the user somehow, and I guess you want to show the names to the user (also multiple times) and not just the address of a pointer to the string with the name.

LtG wrote:HTML = "some kind of markup AST"

I'm suggesting only using "some kind of markup AST", so we reap the benefits. No parsing required. Resource constrained also love it.

I wonder how you would edit such AST without converting it to the user in some readable form, and converting user input back.

LtG wrote:Avoiding "unnecessary" resource usage may be a form or premature optimization. By not using a AST every IDE/compiler/editor will have to understand C++ (to a point), instead of indexing your HDD you have to do a full brute force search (10 TiB vs. few MiB). I'm arguing paying the upfront cost is often simpler and more cost efficient.

An editor does not have to "understand C++" in order to edit it. It just has to understand plain text. HDD indexing is a completely different thing which has nothing to do with GUI vs. TUI, so that is completely irrelevant here.

Posted: **Sun Aug 11, 2019 4:48 am**

LtG wrote:So you're not using any IDE? What are the benefits you need that IDE's don't meet? Have you tried good IDE's?

I am not the one you asked, but I have an answer as well.

I've used, at times, professionally:

VisualStudio
JBuilder
Eclipse

There are likely one or two I've forgotten, plus several others I gave a go privately.

Issues I had with IDEs:

Hiding critical configuration in submenues / complex configuration files where it is hard to figure out what exactly the exact overall configuration applying to a given translation unit actually is (VisualStudio is a beast in this regard).
Diff'ing those configurations tells you very litle.
Subtle compatibility issues between different versions / releases of an IDE (such as a certain MSVS solution configuring OK on one machine but not on another -- turned out one was an English version of MSVS, the other a German one).
To really "fly", you have to customize your IDE's settings. Again, this configuration is usually hidden away in submenues / complex configuration files, or (worse) gets intermingled with the project files (so your choices get back into the repository, and from there into other people's working directories).
Potential for bugs due to overly complex framework (such as the one time where we tried to track down a bug on our AIX build, in direct contact with IBM support, to eventually figure out that their remote-debugging-IDE-plugin didn't honor LIBPATH and thus was of no help whatsoever).
Time spent "learning the IDE" beyond the superficialities is mostly wasted when your next employer uses a different one.
IDEs are usually optimized for a very few specific things, like Java source, or C++ source. I have lots of other files to edit too -- documentation, configuration, test data. This either making do with sub-par support for the format in the IDE, or switching to a completely different environment for the editing (further defeating the benefits of the IDE).
I tend to work on several different machines, with heterogenous software installations. Getting full-blown IDEs installed on e.g. a production server is a no-go. So I'd have to switch between the IDE on the development platform, and something-else-entirely on the production server.

There's more, but I think this is starting to paint a picture. Compare that with the benefits of using plain Vim (which I do):

You work with plain Makefile / CMakeLists.txt. Everything "config" is in there, in plain view, shared among all who compile that source.
Diff'ing different configs (Makefiles / CMakeLists.txt) shows you the changes someone did deliberately (as opposed to "clicking around in a GUI") right away.
IDE / Vim configuration is done / kept in ~/.vim*, i.e. your personal $HOME. Transferring configuration from one machine to another is trivial, and configuration does not get saved in the work directory.
Editing source is completely separate from compiling source. If someone wants to use a different editor, go right ahead. But don't put your .project file or whatever into the repository. Editors, or their versions, don't matter. Editors are also disjunct from compiler choices.
I use the same tool to edit Java source, C++ source, Makefile source, LaTeX source, XML config files, test data. I use the same command language, same macro features. I display different formats in the same editor session. If I change a class name and do a global search, I'll get that class name highlighted in the documentation as well as the source, so the two don't start to diverge.
There is a huge amount of overlap between Vim, sed, grep, and several other "old Unix" tools (simply because they share so much heritage -- Vim and sed, especially). Most of what I learn about one tool, I can apply to others.
One very much underrated feature of Vim is the ability to select a section of text, and filter it through another tool or program (i.e. that program gets the selected text as input, and Vim replaces the selected text with the tool's output). This feature alone has saved me many hours of manual editing, replaced by a Perl 5-liner and a one-shot macro).
I use the same tool regardless of whether I am working locally on my smartphone, netbook, desktop, on the development machine, or the production server. (Getting support to install Vim on a production server is a piece of cake. Usually it is already installed, because it's what they use...)

I admit that IDEs bring some things to the table that require some preparation for a Vim user -- autocompletion, to name the most prominent. But A) it can be set up on Vim as well, and B) I actually don't consider it to be that helpful in the first place. (But that might be because I did mostly maintenance coding, i.e. I edit more than I write from scratch.)

I am quite aware that you can make similar points the other way round, showing what GUI IDE's bring to the table that plain editors don't. I don't say that my points above are more important, that I am "right" and you are "wrong". But there are strong points that can be made for the "classic" toolchain -- which you have to master to actually realize its strong points. Which is probably why it is slowly sinking into oblivion today, as "the kids" get those sparkly IDEs thrown at them and never really see what could be done without them.

Posted: **Sun Aug 11, 2019 6:08 am**

@Solar..

I've used Eclipse for C++, but didn't really like it.

I like IntelliJ, which does support pretty much all the formats you mentioned.

You can use CMake with MSVS.

I don't need IDE on a production server, I don't think you should ever modify code in production.

I agree that most IDE's are bloated (including IntelliJ), but not using an IDE slows things down too much. And RAM is cheaper than developer time.

Posted: **Sun Aug 11, 2019 6:41 am**

In fact, most of the points Solar mentioned apply to my use cases as well, I just didn't remember how much trouble I had with an IDE... The scattered configuration files are definitely something I was annoyed about, and every IDE using a different format for them, so it was impossible to switch from one IDE to another. So thanks for this nice summary!

Let me just add a few points:

Focus on application development: The ones I used for C++ were designed to develop applications that run on the same OS as the IDE itself. So when you create or import a project, it creates a "build" and a "run" step, and the "run" step wants to execute an executable file produced in the "build" step. That turned out to be useless for my OS kernel project, when there is no such executable, and I need to boot an ISO file in an emulator instead. Figuring out how to change the "smart" default to running that emulator took ages, just to find that those IDEs support only one "run" configuration - which is again nonsense if I want to test my kernel against different emulators.
No support for dynamic configurations: My OS project targets multiple platforms, including x86 and ARM. Both have virtual memory, so I have a namespace called "Pager" with functions that map / unmap physical to / from virtual memory. There is one header file with function declarations, which is included by the high level kernel code, and multiple C++ files where the map / unmap functions are implemented (one for each target architecture, in different subfolders). Now a "smart" IDE has some "go to implementation" context menu if I click on a function name - but if there is more than one implementation, where should this take me? How would the IDE know whether I want the ARM or the x86 file opened? Even if there was a way to tell it "right now I work on x86" because I want to correct a wrong flag in the implementation - there's still two different ones, for 32 and 64 bits, and I want both of them.
Outdated standards: My kernel development always uses the latest GCC version with the latest C++ standards (C++17 right now, and GCC 9.2 is expected to come out tomorrow). Last time I used an IDE it did not know any of the latest C++ syntax features I wanted to use and showed me lots of bogus errors. A few years ago it even complained about C++11 fixed size integers, marking a single source file as having more than 20 errors, because it did not honor #include <cstdint>. That killed it for me.

In the end, I spent more time to figure out how to tell the IDE what it should do than with actually coding, and so it wasted time instead of saving it. So how do I cope with these things using VIM, hand written config files (I use autoconf/automake, which some people say is not the best for OS development, but works fine for me), GCC and the command line (which took a lot less time for me to configure, while saving a lot of time while working)?

Custom build and run steps: I have created my own keyboard shortcuts in VIM, which are saved in a VIM script, which I also use to load the project files, window configuration etc. They do exactly what I want because I told them to - run the configure script, run make, run QEMU, run Bochs. Some of them even give me a menu where I can choose whether I want to run QEMU with one virtual CPU or multiple, so I can test SMP code. (There's even more I added, like different shortcuts for "git commit" which add different github icons to the commit message, depending on whether I added a feature, changed something, removed something...)
VIM supports tag files to specify jump locations, which can be generated with a program called "exuberant ctags", and which is again very flexible.
VIM syntax highlighting comes as VIM scripts, which are text files. So if some part of the newest C++ syntax is not yet implemented, I can just add it. YouCompleteMe used Clang in the background for syntax aware completion, so I can simply use the newest version of Clang (whose feature completion is almost the same as GCC) and YCM.

And, before you ask - no, there is no GUI needed here. Keyboard shortcuts, syntax highlighting, completion, smart status line with compiler output - all work in a TUI as well. Note that even QEMU works without a GUI and can output my kernel messages via an emulated serial port directly into the terminal (and I can even have that terminal inside of VIM and use the output). Plus, it allows to run the whole build and test cycle on a headless remote development server, because the whole build configuration is where I want it to be and in a format which also the tools on that server understand.

Posted: **Sun Aug 11, 2019 6:45 am**

XenOS wrote: I've said this more than once: using less resources.

I'm not saying bloat is good, but what are you getting in practice with less resource usage? The laptop has X amount of RAM, use it or don't, you don't save anything.

XenOS wrote: Of course, if it makes things faster. But it does not make things faster if I have to power up a graphical environment just to find something in a file and change it.

I keep the graphical env up all the time.

XenOS wrote: No, and that's again not what I said. I said that opening such a file in an editor is a doubtful way, instead of filtering it and extracting the data that actually needs inspection.

If you know what you're looking for, sure, you can grep it. Wouldn't it be nice if you could also open the file instantaneously regardless of it's size?

XenOS wrote: How would you parse the log file into some kind of AST without going through the entire data set?

I wouldn't parse it, I think parsing is evil. I would keep it in binary format, store the "AST". So for log entries there might be a timestamp and a message, maybe a severity level, etc. Then you can search it.

XenOS wrote: Yes, I do, and not just one of them. I have several Raspberry Pis, for example, all of them running Linux, most of them without GUI. Also some embedded devices with Linux and no GUI. And some HPC servers / nodes.

So how do you interact with the RPi without a GUI? Or do you actually SSH into the RPi from your laptop GUI? I'm not saying it can't be done, but in practice I'd expect most people to always work on a GUI and then just connect to the other devices.

So the RPi doesn't need a GUI or a CLI, unless it needs to be operated directly.

XenOS wrote: Exactly. My use case is working with lots of different text input formats (including C++, HTML, LaTeX, Markdown, plain text, log files) from different sources and extracting data from them. So either I need different, specific tools for each of them, or just one generic tool which solves all my tasks.

I think using text as the interchange format was a bad idea. If every format was registered with the system, then you could still use one search tool and it would be better than grep+text.

XenOS wrote: If it exists it can be used, but it does not necessarily have to exist at all (see my HTML link extracting example above) because it is not needed.

If you only ever used HTML to grep it for youtube links, then true, you don't need to have HTML "AST" on the system. In that case you, the person, are the consumer of HTML and nothing else on the system needs to understand HTML, but you still do.

More practically you would likely have something on the system that does need to consume HTML (like a browser, or maybe some automated tool, maybe apt-get (not sure if it uses HTML or something else)), so in practice most systems need it anyway. Of course you don't have to install some fileformat understanding to a system that doesn't need it. So a IoT device might be very trimmed down OS.

XenOS wrote: An X server + some graphical environment takes definitely more than a few MiB.

I've never been a fan of X, but as I said, Win95 could manage with just a few MiB for a GUI. Of course today you need more memory simply due to higher resolutions alone. The code to deal with GUI compositing, etc, doesn't have to be huge, given that Win95 managed with a few MiB total.

XenOS wrote: Still they have to be presented to the user somehow, and I guess you want to show the names to the user (also multiple times) and not just the address of a pointer to the string with the name.

Yes, there would probably be a textual representation for the user. I haven't been able to figure out anything better for that.

My argument is that we shouldn't store it as text though, because then we have to parse it, instead store it as "code" (binary AST, etc), and produce the human friendly text when needed.

XenOS wrote: An editor does not have to "understand C++" in order to edit it. It just has to understand plain text.

No it doesn't, clearly, since we use text editors to edit C++ files.

But that offloads the burden to us users. If you want anything better, like refactoring, syntax highlighting, etc, then the editor (or plugin) does have to understand the language. And the file alone is useless, so you practically also need a compiler, which has to understand it.

So I'm not adding bloat, rather splitting the understanding from the compiler away to it's own service so all the tools can use it. The editor uses it, search/grep uses it, etc.

If I have a log file (binary/"AST") with just timestamps and messages, how would I grep that? What would be the flag for timestamps? What representation that I have to remember correctly? In a GUI I can just open the file and immediately sort by field or filter, there's discoverability and the GUI helps me with the date format, maybe with convenient buttons for day/hour/etc.

What if I want to search for specific function calls by name? What's the grep flag for that? Of course this could all be done on the CLI too, but it would mean a ton more flags for grep. I don't think it would work as well as it would within a GUI.

Posted: **Sun Aug 11, 2019 7:19 am**

Now you're trying to use grep to solve a problem it is not intended to be used for. You don't use grep for structured data. Use clangd for the source code case. For logs, note that journald's logs are already binary, still CLI tools have no problems handling those. Structured/non-structured storage is completely orthogonal to the CLI discussion and hence OT.

Posted: **Sun Aug 11, 2019 10:05 am**

LtG wrote:I'm not saying bloat is good, but what are you getting in practice with less resource usage? The laptop has X amount of RAM, use it or don't, you don't save anything.

As I said multiple times before: I don't have unlimited amounts of memory on every device I use. I cannot use more memory on a limited platform than it has, or more network bandwith than my connection supports to run a remote UI. So resource footprint does matter. Besides - even on my 64GB RAM office computer I minimize resource usage of most tools because I want to use the memory for what I actually need it (like running memory intensive calculations and caching their intermediate results, so the more memory I give to that and the less I use for unnecessary things, the faster my calculation runs and the more time I save - yes I do use all memory on that machine, yes it has all slots full, no I won't buy a bigger one just to run an IDE at the same time).

LtG wrote:I keep the graphical env up all the time.

Most of my machines don't even have one.

LtG wrote:If you know what you're looking for, sure, you can grep it.

...which was exactly the example scenario you asked for - find all stack traces involving a particular function call.

LtG wrote:Wouldn't it be nice if you could also open the file instantaneously regardless of it's size?

No. I would definitely not want to read and inspect an unfiltered file of several GB. And even if I wanted to have a look at it (or show it to someone who really begs for that experience), I would not use an editor for that, but a pager (like less on Linux).

LtG wrote:I wouldn't parse it, I think parsing is evil. I would keep it in binary format, store the "AST". So for log entries there might be a timestamp and a message, maybe a severity level, etc. Then you can search it.

That would require that you have written the program yourself that creates the output, so that it writes log files in AST instead of text files. Also it requires that every user who wants to understand the log file must use some program that understands and displays that particular binary AST format. The advantage of plain text is that you can inspect it without such a specific tool, just with a text viewer (+ filters such as grep if you want to find something specific).

LtG wrote:So how do you interact with the RPi without a GUI? Or do you actually SSH into the RPi from your laptop GUI? I'm not saying it can't be done, but in practice I'd expect most people to always work on a GUI and then just connect to the other devices.

Either SSH or a serial console. And an SSH client doesn't need a GUI either. Remember that laptop I mentioned with the broken graphics adapter working only in text mode? I used that quite often for such purposes.

LtG wrote:I think using text as the interchange format was a bad idea. If every format was registered with the system, then you could still use one search tool and it would be better than grep+text.

That means that "the system" has to know every format you might ever want to use. What if you have a new program that gives you log files in a custom format, because it needs to add some data that cannot be stored in the ones known to the system? Even if this program comes with its own log file viewer - what if you want to show it to someone else, who doesn't use that program, and maybe even uses a different OS that is not supported by the viewer? The big advantage of text files as interchange format is that one doesn't need any special purpose tool for every different format to view and edit them. A text editor works for every kind of text file.

LtG wrote:If you only ever used HTML to grep it for youtube links, then true, you don't need to have HTML "AST" on the system. In that case you, the person, are the consumer of HTML and nothing else on the system needs to understand HTML, but you still do.

More practically you would likely have something on the system that does need to consume HTML (like a browser, or maybe some automated tool, maybe apt-get (not sure if it uses HTML or something else)), so in practice most systems need it anyway. Of course you don't have to install some fileformat understanding to a system that doesn't need it. So a IoT device might be very trimmed down OS.

Even if I have, e.g., apt-get installed on a system (it doesn't understand HTML, just the HTTP protocol, and even that via some library), I don't have to load it into RAM and execute it if I just want to grep for links. There is a difference between having a tool or library installed on a system, and having it actually loaded in RAM and executing.

LtG wrote:I've never been a fan of X, but as I said, Win95 could manage with just a few MiB for a GUI. Of course today you need more memory simply due to higher resolutions alone. The code to deal with GUI compositing, etc, doesn't have to be huge, given that Win95 managed with a few MiB total.

The code size shouldn't depend on the resolution at all. On systems where I do use a GUI, I keep it minimal - XMonad or LXDE, applications that prefer plain X / lightweight libraries over heavy frameworks. Again that pays off on a limited device (such as an old tablet with little RAM, but perfectly usable in such an environment).

LtG wrote:My argument is that we shouldn't store it as text though, because then we have to parse it, instead store it as "code" (binary AST, etc), and produce the human friendly text when needed.

See my argument above - your files are not readable without a special tool.

LtG wrote:No it doesn't, clearly, since we use text editors to edit C++ files.

But that offloads the burden to us users. If you want anything better, like refactoring, syntax highlighting, etc, then the editor (or plugin) does have to understand the language. And the file alone is useless, so you practically also need a compiler, which has to understand it.

So I'm not adding bloat, rather splitting the understanding from the compiler away to it's own service so all the tools can use it. The editor uses it, search/grep uses it, etc.

That's not a new idea. I mentioned VIM + YouCompleteMe. The latter uses libclang to "understand" C++, so it does already what you suggest - splitting the understanding from the compiler, make it available in the editor.

LtG wrote:If I have a log file (binary/"AST") with just timestamps and messages, how would I grep that? What would be the flag for timestamps? What representation that I have to remember correctly? In a GUI I can just open the file and immediately sort by field or filter, there's discoverability and the GUI helps me with the date format, maybe with convenient buttons for day/hour/etc.

So now you don't only need a special tool to view the log file, but you also need a GUI? What if you are working in a CLI only environment? You cannot even view log files anymore. That sounds horrible to me.

LtG wrote:What if I want to search for specific function calls by name? What's the grep flag for that? Of course this could all be done on the CLI too, but it would mean a ton more flags for grep. I don't think it would work as well as it would within a GUI.

Code: Select all

pcregrep -M 'FunctionName\s*('

...since a function call should be followed by optional whitespace (possibly newlines, hence the use of pcregrep with -M multiline option) and opening bracket. Just a quick example.

Posted: **Sun Aug 11, 2019 5:09 pm**

LtG wrote:
XenOS wrote: Yes, I do, and not just one of them. I have several Raspberry Pis, for example, all of them running Linux, most of them without GUI. Also some embedded devices with Linux and no GUI. And some HPC servers / nodes.
So how do you interact with the RPi without a GUI? Or do you actually SSH into the RPi from your laptop GUI?

I do, and I can't really figure what's so "actually" about it. Actually I did spend most of the last twenty years of my professional carreer SSH'ed to some Solaris / Linux / AIX machine of some description.

I'm not saying it can't be done, but in practice I'd expect most people to always work on a GUI and then just connect to the other devices.

Err... what? "SSH into the RPi from your laptop GUI" and "work on a GUI and then connect to the other device" are the same thing...?!?

LtG wrote:So the RPi doesn't need a GUI or a CLI, unless it needs to be operated directly.

It's at points like this where I think you're a bit fuzzy about the definition of some of the terms you're using.

A system without a CLI can't be SSH'ed into.

But you can launch a GUI app on a system you're connected to via SSH, using your local machine as X server (i.e., application runs remote, but you get to see / use the GUI locally).

Posted: **Mon Aug 12, 2019 6:50 am**

Solar wrote:
LtG wrote: So how do you interact with the RPi without a GUI? Or do you actually SSH into the RPi from your laptop GUI?
I do, and I can't really figure what's so "actually" about it.

The actually is about _actually_ using a GUI, on the laptop side.

You can use the RPi directly (with keyboard + display):
- with a GUI
- with just text mode, but still graphics stack needed because the fonts get rendered by Linux
- with just text mode, without graphics stack (not sure if RPi supports it)

If you connect to the RPi from your laptop, then the above also applies on your laptop.

I think most people are running GUI on their laptop (or at least in graphics mode), so much if not all of the cost is already paid.

I don't think you need to run a GUI/CLI on the remote end, except in some cases (like virtual desktop).

Solar wrote:
I'm not saying it can't be done, but in practice I'd expect most people to always work on a GUI and then just connect to the other devices.
Err... what? "SSH into the RPi from your laptop GUI" and "work on a GUI and then connect to the other device" are the same thing...?!?

Yes, the two are the same. See above. I expect the cost of at least one GUI already paid in almost all cases where a _user_ is interacting with the systems.

Solar wrote:
LtG wrote:So the RPi doesn't need a GUI or a CLI, unless it needs to be operated directly.
It's at points like this where I think you're a bit fuzzy about the definition of some of the terms you're using.

A system without a CLI can't be SSH'ed into.

Well, the local CLI and the SSH CLI are separate, a server could be running headless and without any memory use for graphics (including the LFB), and still allow SSH.

However I was talking about the RPi not needing GUI/CLI at all for remote control. You need something for remote control (assuming you want to control it remotely), but neither GUI/CLI is a requirement. So the remote control wouldn't happen over SSH.

If you need to interact with the RPi directly (keyboard + display) then you do need something like CLI/GUI.

I haven't figured out how I'm going to do remote control/desktop, so I don't know which parts happen locally and which remotely. For instance TeamViewer (TV) seems to send picture snapshots, while MS RDP seems to share semantic information allowing for better UX.

I am planning on allowing the remote end be controlled without having to establish a remote desktop. For instance being able to start/stop services, shutdown/reboot the system, etc. Of course you could also remote desktop into the system (creating a remote UI) and do those same actions from there.

Assuming you are using a system with a GUI, then you only pay for GUI once in the first case above, and twice in the second case above.

OSDev.org

Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?

Re: Are command line interfaces for an OS obsolete?