Programming "Views"

[Brendan]

Hi,

Imagine you've got a project consisting of several processes and plenty of other resources (sounds, pictures, data files for internationalisation, etc).

You open the project. What does the IDE show you?

Mostly you'd want to see an overview of the project as a whole - which processes communicate with each other, which resources are used by which processes, etc. This could be (e.g.) boxes connected with arrows; where the colour of a box tells you what sort of thing it is (e.g. yellow boxes for processes, green boxes for data, orange boxes near the edge for external pieces like the OS's VFS and GUI, etc); and the arrow tells you the dependencies between the boxes.

From this view you'd want to be able to drag and drop new boxes from a toolbar into the project (to add new files to the project), and be able to click on things like pictures to start an image editor, etc. You'd also want to be able to delete anything that isn't used; and maybe right-click to view/edit the properties (change the file's name, see some stats on it's contents, etc).

You click on something that represents a process to "zoom in" on it. Now what does the IDE show you?

For me, processes are split up into "units" (which take the place of files). You're going to want to see units and the relationships between them (more yellow boxes connected by arrows). Just like before you're going to want to be able to create new units and delete units that aren't being used; and right-click to view/edit the properties (change the unit's name, see some stats on it's contents, etc).

You click on something that represents a unit to "zoom in" on it. Now what does the IDE show you?

Did you guess "more coloured shapes connected by arrows"? I hope so. A unit consists of functions, global data, structure/type definitions, etc. Each of these could be a circle/ellipse. Maybe light blue for functions, light green for global data, etc. In addition you'd have arrows going to things in other units that might be drawn as greyed out circles/ellipses. Just like before, you're going to want to be able to create new things and delete things that aren't being used; and right-click to view/edit the properties.

You click on something that represents a function to "zoom in" on it. Now what does the IDE show you?

This one is easy - you probably want some sort of text box showing the function as lines of code (with syntax highlighting and other goodies). For things like global data and structure/type definitions you might want something else, like a dialog box, where you can add members, change existing members, etc. You'd also want to be able to click on things like function calls to be taken to the function being called, and have a "back" button so that you can navigate through the unit that way (and follow the code to other units, etc).

Now; it's very important for all these views to reflect the actual code. There's no point having pretty diagrams that contain information that quickly becomes obsolete and needs to be manually fixed to avoid being useless (this is probably the single biggest problem with things like UML modelling tools). Is this possible?

Because my source code exists in a pre-tokenised state, there's a symbol list and a type list built into the file (where one source code file contains all the units for one process). It doesn't take much work to scan the tokens, find "symbol references" and "type references", and use the symbol list and type list to figure out what each piece within each units depends on and where it is. It's only the highest level view (showing the files that make up the project and their dependencies) that are a problem.

To solve that, you can have a "project file" that acts as a cache of the information needed. For each file, you want the file's name plus a list of other files it depends on. Data files don't depend on anything. Any time the source code for a process is changed you'd scan through it looking for dependencies (file names) and update the cached information in the "project file".

Because it's sort of easy to get all the information for each of the views, the higher level views could show more details. For example, (for the highest level "project view") rather than just having a yellow box with a name on it representing a process, you could have a yellow box with a name that has a little diagram inside it, and as you zoom in the little diagram gets larger and larger until you reach the "process view" (where rather than just seeing a yellow box for each unit, you see a yellow boxes with little diagrams inside them).

Also, if we're caching information in the "project file" we can cache other sorts of information too; like thumbnails for pictures, icons for sound files (or maybe a thumbnail of a screenshot of the sound editing application?), etc. In some cases these can be "zoom-able" too. E.g. you zoom in on the thumbnail for a picture and it gets larger and larger until you realise you're inside an image editor.


Cheers,

Brendan

[Thomas]

Dear Brendan ,

You might be able to hack up a Visual Studio plugin as proof of concept ( Some COM knowledge would be helpful). Or even an Eclipse or NetBean plugin.
But it makes me wonder, what specific application do you have in mind ? It is really hard to come with something general purpose with what you describe.

--Thomas

PS : http://en.wikipedia.org/wiki/Microsoft_BizTalk_Server . See if BizTalk suits you ? . Since it is used in enterprises for nearly the same thing you described.

[Brendan]

Hi,

You might be able to hack up a Visual Studio plugin as proof of concept ( Some COM knowledge would be helpful). Or even an Eclipse or NetBean plugin.
But it makes me wonder, what specific application do you have in mind ? It is really hard to come with something general purpose with what you describe.

This is ideas for a future IDE, for my source file format (which isn't "plain text"), for my language, for my OS.


Cheers,

Brendan

[Kevin]

I think before you start describing what the IDE should look like, you need to define the goals and the different use cases it should accommodate. In general I think the high-level diagrams are mostly interesting for people who are new to the project (or to a specific part of the project) and need to get an overview. Once you're reasonably familiar with the project, you know the big picture and just want to work on the source code level without worrying about diagrams. So I think it's indeed of utmost importance that all the arrows in the diagram are not entered by the user, but automagically generated and updated from the source code.

One thing that I imagine challenging is distinguishing important arrows from minor ones. If you display all dependencies in a big project, you end up with a lot of not so interesting arrows. There is probably a danger that you won't see the forest for the trees if all of them are displayed. But I guess it's something that needs to be tried out with data from a real project.

For things like global data and structure/type definitions you might want something else, like a dialog box, where you can add members, change existing members, etc.

No, please not! This is one of the major reasons why input sucks so hard in UML tools. You can't just write down your structure fields, but you have to click through dialog boxes, in the really bad tools switching between mouse and keyboard multiple times per field.

This is one general problem I would have as a user of your IDE: It's a misconception that I would want to click anywhere. Using a mouse, and even more so switching between keyboard and mouse, is inefficient. Just give me a good keyboard interface. You may, of course, have the mouse functionality additionally, but don't make it mandatory to use it.

[Combuster]

Brendan,

I'm probably going to be a bastard with a lack of imagination for saying this, but make a prototype and a youtube video demonstrating that it works, preferably better than what I do with my editor of choice. Because in all seriousness, there's hardly any shared foundation between ways of working to allow any form of meaningful comparison as it stands.

[Thomas]

Dear Brendan ,

This is ideas for a future IDE, for my source file format (which isn't "plain text"), for my language, for my OS

You can make a visual studio plugin to read binary files as well. Since these platforms already solve 70% problems you describe , it might be well worth your time re use than re invent.

I worked as "senior" engineer for a company. In all honesty , seniority does not equate to competence , but it does make me more equipped to deal with corp bureaucracy and how to deal with people in general. New languages does not really solve any of the engineering issues in my opinion , As a maintenance engineer my experience has shown that even with Java , C# etc I still have the same engineering issues to deal with and have not considerably eased by job. A well written piece of C code is always easier to read , debug and fix -- ie more maintainable.

One more design issue you might want to ponder over is , is it good a really idea to tightly couple the language with the IDE ? Time has shown that such languages do not really stand the test of time . Examples VB6 , Actionscript are obsolete now. The languages themselves are not bad but for some reason they tend to become obsolete sooner.

So I think it's indeed of utmost importance that all the arrows in the diagram are not entered by the user, but automagically generated and updated from the source code.

There are many plugins and features in modern ide's that already does it.

You might want to check out the old RosAsm win32 assembler where you edited the .EXE file directly and you had drag n drop functionality . Too bad that it is not supported anymore. Kudos to the author who has put in significant amount of work.

http://web.archive.org/web/201005270832 ... osAsm.html

--Thomas

[Brendan]

Hi,

I'm probably going to be a bastard with a lack of imagination for saying this, but make a prototype and a youtube video demonstrating that it works, preferably better than what I do with my editor of choice. Because in all seriousness, there's hardly any shared foundation between ways of working to allow any form of meaningful comparison as it stands.

I think you're right - some sort video/pictures would make this a lot easier to understand. Sadly, I'm a very bad artist and wouldn't even know how to start creating a video (it'd actually be easier for me to slap together a bootable "320*200 real mode" demo, but that'd be awkward for other people and too much work for me).


Cheers,

Brendan

[Combuster]

If you have a camera, you can just point it at a live screen+keyboard with some example (read: cheap mock up that's only capable of doing the things you're demonstrating - You might even get away with a powerpoint of fake screenshots where you can "click" on). Getting anything functional for us to test in person is quite a big step further given the lack of infrastructure, even if it's still a mock-up.

[Brendan]

Hi,

This is ideas for a future IDE, for my source file format (which isn't "plain text"), for my language, for my OS

You can make a visual studio plugin to read binary files as well. Since these platforms already solve 70% problems you describe , it might be well worth your time re use than re invent.

I've never used visual studio and don't have a copy on any computer; and I have never done windows programming of any kind (at least none that I'd willingly admit to). The time it would take for me to "re-use rather than invent" sounds like many months of work, and (given that I only want to slap together a little example to help people understand what I'm describing) that sounds extremely excessive.

One more design issue you might want to ponder over is , is it good a really idea to tightly couple the language with the IDE ? Time has shown that such languages do not really stand the test of time . Examples VB6 , Actionscript are obsolete now. The languages themselves are not bad but for some reason they tend to become obsolete sooner.

It's not that the language itself is tightly coupled with the IDE. It's that the source file format includes things that require some sort of tool to manage (e.g. a type list, a symbol list, etc). It's possible for 20 different people to implement 20 very different IDEs that all handle the same source file format. It's also possible for an IDE to make the language look a lot like C (or Python, or BASIC, or Pascal); because the source file format stores tokens and not text - language syntax is left up to the IDE.


Cheers,

Brendan

[bluemoon]

Brendan, your idea assume a strict hierarchy of objects. How to you represent interfaces or extends (as of objective-C)? Do you purposely restrict those patterns?

[Brendan]

Hi,

I think before you start describing what the IDE should look like, you need to define the goals and the different use cases it should accommodate. In general I think the high-level diagrams are mostly interesting for people who are new to the project (or to a specific part of the project) and need to get an overview. Once you're reasonably familiar with the project, you know the big picture and just want to work on the source code level without worrying about diagrams. So I think it's indeed of utmost importance that all the arrows in the diagram are not entered by the user, but automagically generated and updated from the source code.

You're right. The goals would be:

• help a designer to design the project (possibly weeks before anyone writes any code at all)
• help programmers implement all the little pieces of the design by giving them diagram/s they can refer to that describe what the pieces are supposed to do and how they're supposed to fit together.
• help people that are new to the project understand it quickly.

So I think it's indeed of utmost importance that all the arrows in the diagram are not entered by the user, but automagically generated and updated from the source code.

They'd be entered by the designer; then automatically modified/corrected as programmers implement things (as the design inevitably morphs into something radically different from what the designer originally designed ). Of course if there is no up-front design (e.g. "agile" rather than "waterfall") then it'd naturally morph from "nothing" into "something".

One thing that I imagine challenging is distinguishing important arrows from minor ones. If you display all dependencies in a big project, you end up with a lot of not so interesting arrows. There is probably a danger that you won't see the forest for the trees if all of them are displayed. But I guess it's something that needs to be tried out with data from a real project.

At the highest level you're not going to have too many processes (or dependencies between processes and other things). As you go deeper towards the lower levels the total number of dependencies increase, but the fraction of the total you're looking at decreases, so it should (at least partially) cancel out.

If you like, think of it a bit like google maps. If there's a country with 1 million roads; when you're looking at a map of the entire country you only see the major roads connecting cities and don't see 1 million roads, and when you zoom in enough to see minor roads you're only looking at a specific area and don't see 1 million roads.

For things like global data and structure/type definitions you might want something else, like a dialog box, where you can add members, change existing members, etc.

No, please not! This is one of the major reasons why input sucks so hard in UML tools. You can't just write down your structure fields, but you have to click through dialog boxes, in the really bad tools switching between mouse and keyboard multiple times per field.

I did some checking. I downloaded the Xerox Alto's user manual today and had a look to make sure; and I can confirm that keyboard shortcuts are as old as the GUI itself.


Cheers,

Brendan

[Brendan]

Hi,

Brendan, your idea assume a strict hierarchy of objects. How to you represent interfaces or extends (as of objective-C)? Do you purposely restrict those patterns?

I'ts not a hierarchy of any one thing. The highest level view would show processes and data files, the next level down shows "units", and the next level down shows functions.

It's not intended for OOP.

Let me see if I can explain it better with some ASCII art....

Imagine you're implementing a simple text editor. Naturally this is 2 processes - one process is the "front end" (to deal with the user), and the other process is the "back end" (to deal with the file you're editing). The front end also talks to the GUI (an external thing, not part of the project), and the back end talks to the virtual file system (another external thing).

So, for the highest level view you might see something like:

Code: Select all

    _ _ _ _      ________      ________       _ _ _ _ 
  :        :    |        |    |        |    :        :
  :  GUI   :<-->| Front  |<-->|  Back  |<-->:  VFS   :
  :_ _ _ _ :    |  End   |    |  End   |    :_ _ _ _ :
                |________|    |________|

When the user clicks on the "front end" box they go down to the next lower level; which shows the individual units in the front end process.

The front end process might have 5 units - one main unit containing a message handling loop and "event dispatch", one unit that contains code to handle communication with the back end, one unit that has user input handling code, and one unit that has video output code. This might be displayed like this:

Code: Select all

                             _______
                            |       |
    ________     _______    |       |
   |        |   |       |   |       |    _______  
   |        |-->|       |-->| Video |<--|       |
   |        |   | Input |   |_______|   |       |
   |  Main  |   |       |               |       |
   |        |   |       |-------------->| Back  |
   |        |   |_______|               | Comms |
   |        |                           |       |
   |        |-------------------------->|       |
   |________|                           |_______|

The user clicks on the "back comms" box they go down to the next lower level; which shows the data and functions contained in that specific unit.

The "back comms" unit might have a function to handle messages from the back end process (that calls other functions in the unit); plus several functions to ask the back end to do various things (open/read a text file, delete some characters in the current text file, insert a character, save the file, etc). It might also have some global data (but I can't think of anything it'd need). This might be displayed like this (using rectangles instead of circles because circles are too hard to draw with ASCII art):

Code: Select all

           _____________________
          |                     |
   Main-->| sendOpenFileRequest |
          |_____________________|

           ____________________     ___________________________
          |                    |   |                           |
          |                    |-->| handleFileOpenedStatus    |
          |                    |   |___________________________|
   Main-->| backMessageHandler |    ___________________________
          |                    |   |                           |
          |                    |-->| handleScreenContentsReply |-->Video.updateScreen
          |____________________|   |___________________________|

           ___________________________
          |                           |
  Input-->| sendScreenContentsRequest |
          |___________________________|

           _______________________
          |                       |
  Input-->| sendDeleteTextRequest |
          |_______________________|

           _______________________
          |                       |
  Input-->| sendInsertTextRequest |
          |_______________________|

I hope that makes things a little clearer.


Cheers,

Brendan

[Kevin]

I would love to see what this looks like with a project of the size of, say, qemu. I suspect having only three levels of diagrams, one of which is reserved for processes, means that at least the level containing the units would be overloaded.

[Brendan]

Hi,

I would love to see what this looks like with a project of the size of, say, qemu. I suspect having only three levels of diagrams, one of which is reserved for processes, means that at least the level containing the units would be overloaded.

That's probably a good thing - if there's too many units in a process, then you're probably not using enough processes.

Note: I still think emulators should use a "process per emulated device" model (with a standardised "virtual PCI device" messaging protocol to allow different virtual devices to be written by anyone and used in any emulator).


Cheers,

Brendan

[Kevin]

Well, convert each supported device and backend into a separate process, then your process diagram is overloaded.

A lot of these units don't really exist in any running process anyway, they are mostly alternative. So for your disk backend, you usually have either a raw image, or a qcow2 image, or a VMDK image, or an NBD connection, etc. but rarely you use all of them at the same time. Similarly, you have either PC hardware or some ARM board or an old PPC Mac platform, but you never need all the devices at the same time.

The other thing is that for perfomance reasons you'll likely want to have the device implementation in the same thread as the backend (I assume for purpose of this IDE, thread and process are mostly equivalent). So a running instance will have an IDE device and a qcow2 backend in one process, and the SCSI device and a raw image in a second process. So you actually have a lot of processes/threads involved, but from the perspective of this IDE they are just one type of process, which would contain all block backends and all block device emulations as its units, right?