Page 3 of 3

Re: 512B (Selfer)

Posted: Mon Mar 25, 2013 2:50 am
by dozniak
Shikhin wrote:Erm, no. I've got only two Gists in total
I meant the gists in general. They're like git-backed pasties usually.
Shikhin wrote:could be a fork of the original one, yes. I was stupid and forgot to do that, but, Gist still allows you to do it (even if I didn't).
That could probably solve it.
Shikhin wrote:If more and more people use Gists for this purpose, maybe GitHub includes it in your main profile, and more people also make sure to check people's Gists out?
That may be, but it isn't at the moment.

Re: 512B (Selfer)

Posted: Mon Mar 25, 2013 2:57 am
by shikhin
Hi,
dozniak wrote:
Shikhin wrote:could be a fork of the original one, yes. I was stupid and forgot to do that, but, Gist still allows you to do it (even if I didn't).
That could probably solve it.
Yes, could solve it -- but now the new one has 2 revisions, so I'm too lazy to fix it. You still can live with it. And it isn't stopping you from donating me money. :lol:
dozniak wrote:
Shikhin wrote:If more and more people use Gists for this purpose, maybe GitHub includes it in your main profile, and more people also make sure to check people's Gists out?
That may be, but it isn't at the moment.
Thus, I'm trying to make it so.

Regards,
Shikhin

Re: 512B (Selfer)

Posted: Mon Mar 25, 2013 8:41 am
by dozniak
Shikhin wrote:And it isn't stopping you from donating me money. :lol:
I spent a lot on Kickstarter, so I'll skip for now.

Re: 512B (Selfer)

Posted: Mon Mar 25, 2013 5:32 pm
by DavidCooper
Hi Shikhin,

Testing has begun. I've put selfer 3072 sectors into a multifloppy partition which starts at the 3GiB point on a 4GB flash drive - that puts it at the start of the second 1.5MB subpartition with my own OS in the first subpartition. One of the photos below shows Selfer being selected for booting from the first page of 125 subpartitions. The second photo shows the second sector of Selfer after it's been loaded in on boot having previously been saved using s, the code in it being test code for k, m, p and r (which all worked fine). All the F6s are simply values that were already on the flash drive - it would be easy to clear them to zeros, but I wouldn't want to save the result using w due to the excessive wear. The colourful stripe across it is my real-mode debugger, called into action by running a CC Int 3 instruction - this offers a direct root to reboot my OS from there (the VBR being stored further up memory along with the debugger). [Edit: I should have pressed h to switch the debugger into hex mode, but I can't be bothered making a replacement picture so I'll just leave it as it is.]

For Selfer to be used practically on a flash drive, it really needs to be in subpartitions of a multifloppy partition so that it can be saved to many different locations, thereby creating backups and retaining old versions to return to if the latest version (of Selfer plus the code being built on top of it) breaks beyond repair. If Selfer is in the first subpartition instead of my OS, it will boot initially from there, and by changing the LBA stored at 502h it could save itself (and any code that's being built with it) to a new subpartition, though it needs new code to do that saving in a manner that avoids excessive wear (which would be written in the second sector). It would also need new code (again in the second sector) to boot up a different version of itself from another subpartition, but the result would be that each version would be able to boot any other. It would need to create a dummy partition table entry for DS:SI to point at, specifying the limits of the subpartition for Selfer to work within. Further code could automate various aspects of the relocation process and show a list of available versions to boot and available places to put a new copy after loading in the MBR to find the limits of the multifloppy partition.

For multifloppy partitions, I'm currently using 82h to label it as a multifloppy partition (sharing that value with CP/M). It would also make sense to put the name Selfer in at 03h in the first sector if there's enough room for that so that the name of whatever's there for each subpartition can be displayed. With my OS, it displays whatever's at that location unless it recognises a FAT12 disk in which case it uses the disk name further down the BPB instead (if it isn't blank), and with my own disk format it finds the disk name near the end of the bootsector next to the date and time of the last save, and 12 bytes of wear variables which count how many times the location on the drive has been written to (using 4 bytes for every half megabyte block of flash memory). It isn't necessary (or even possible) for Selfer to go to the same lengths - just displaying the name at 03h would do, and that's why putting the name Selfer there would be useful - the user could change it to the name of the OS or tool they're building, or to version numbers, or dates. Personally I would just keep my own OS in the first subpartition and use it like a fancy MBR to boot different versions of Selfer, but a 2nd sector extension to Selfer should ideally be designed to do the same job so that Selfer is capable of being completely independent.

Re: 512B (Selfer)

Posted: Tue Mar 26, 2013 12:18 am
by DavidCooper
Hi Shikhin,

I've just run through your code with my debugger (which can replace selected instruction bytes ahead in the code with Int 3 instructions and then restore them when it gets to them) and I've found a little bug. You store the LBA at 502h, but you also store the k copy location at 501h, so one of the bytes of that overlaps with the LBA and makes loading and saving unreliable.

I haven't found anything more serious than that, but I was trying to work out why when I loaded Selfer from the third subpartition it ended up writing 8192 copies of itself to the flash drive, one every 8 sectors up from it's initial location. [Edit: I've just found out why - I think it does that every time I get the dummy partition table entry's LBA wrong, but that'll be fixed when I automate its setup.] When I originally loaded Selfer in from the second subpartition, the first time it didn't appear to load anything in at all (either that or it loaded in random junk from elsewhere on the drive), but on subsequent attempts it behaved perfectly, so I don't know what happened then either. More experiments will be required. [Edit: doubtless the same cause - I think I forgot to set up the dummy table entry altogether the first time. Whenever that happens, pressing x to look at the second sector sends the cursor to the right instead and it sometimes writes the bootsector to the flash drive 8192 times. Anyway, I think that can be avoided from now on, and fortunately I was using QEMU which seems to do all the saves in one go with very little wear, unlike the way it works with the BIOS when running directly on real hardware (though the saves were made to real hardware - I wasn't working with an image).]

Other possible issues: how safe is the first call you make to the BIOS to check for extensions? Could it mess up DS:SI or DL? It would be safer to save DL and the LBA first. Also, how safe is it when you get key input? Are Int 16h BIOS calls guaranteed not to mess up registers?

I wasn't looking for places to reduce the size of the code, but there's one bit that jumped out where you prepared to test for a nop at 7E00 using the code mov bx,bp followed by add bh,0x2. Those could be replaced with mov bx,0x7E00. It's only two bytes saved though.

Re: 512B (Selfer)

Posted: Tue Mar 26, 2013 5:17 am
by shikhin
Hi,
DavidCooper wrote:I've just run through your code with my debugger (which can replace selected instruction bytes ahead in the code with Int 3 instructions and then restore them when it gets to them) and I've found a little bug. You store the LBA at 502h, but you also store the k copy location at 501h, so one of the bytes of that overlaps with the LBA and makes loading and saving unreliable.
I think I fixed that in the latest revision of the Gist. The LBA is now stored at 0x503, while the block (for int 0x13) is stored at 0x507. Unfortunately, I still haven't gotten into the habit of creating virtual drives, putting a MBR on it, and putting Selfer into a VBR. It took a lot of experimentation to get it working with a pen drive in the first case -- right now, it's a bit too tough to do it again. Could you test if it's completely fixed, now?
DavidCooper wrote:I haven't found anything more serious than that, but I was trying to work out why when I loaded Selfer from the third subpartition it ended up writing 8192 copies of itself to the flash drive, one every 8 sectors up from it's initial location. [Edit: I've just found out why - I think it does that every time I get the dummy partition table entry's LBA wrong, but that'll be fixed when I automate its setup.] When I originally loaded Selfer in from the second subpartition, the first time it didn't appear to load anything in at all (either that or it loaded in random junk from elsewhere on the drive), but on subsequent attempts it behaved perfectly, so I don't know what happened then either. More experiments will be required. [Edit: doubtless the same cause - I think I forgot to set up the dummy table entry altogether the first time. Whenever that happens, pressing x to look at the second sector sends the cursor to the right instead and it sometimes writes the bootsector to the flash drive 8192 times. Anyway, I think that can be avoided from now on, and fortunately I was using QEMU which seems to do all the saves in one go with very little wear, unlike the way it works with the BIOS when running directly on real hardware (though the saves were made to real hardware - I wasn't working with an image).]
I have no clue on that error -- since I assume the state of a lot of registers in Selfer (e.g. "this was set to X here, that was set to X there, this xor that is zero, and oh! neat, it's also required in a BIOS call, and the BIOS call isn't expected to touch that register, so wait, I can inc it here, and I get 1 in the register!"), a small little error in the dummy entry could cause very much unexpected behavior.
DavidCooper wrote:Other possible issues: how safe is the first call you make to the BIOS to check for extensions? Could it mess up DS:SI or DL? It would be safer to save DL and the LBA first. Also, how safe is it when you get key input? Are Int 16h BIOS calls guaranteed not to mess up registers?
For checking int 0x13 extensions, RBIL says that DL and SI aren't messed up with. I could, however, the general structure goes as follows: "check for int 0x13 extensions; clear AX to set registers; use AH which is cleared up to set video mode;", and I was a little too lazy to change that a bit. According to RBIL, AH and AL should be modified only.

Some people go as far as to not trust the BIOS at all. I've seen people pushing everything before a BIOS call, and restoring it back. I neither do that in my main OS, nor in Selfer. I think I'll change it a bit in my main OS, since you shouldn't be trusting the BIOS so much. However, for Selfer, we don't have the space to not trust the BIOS.
DavidCooper wrote:I wasn't looking for places to reduce the size of the code, but there's one bit that jumped out where you prepared to test for a nop at 7E00 using the code mov bx,bp followed by add bh,0x2. Those could be replaced with mov bx,0x7E00. It's only two bytes saved though.
That totals up to 5 free bytes in Selfer, and 9 free bytes in Selfer (extended). =D>
DavidCooper wrote:Testing has begun. I've put selfer 3072 sectors into a multifloppy partition which starts at the 3GiB point on a 4GB flash drive - that puts it at the start of the second 1.5MB subpartition with my own OS in the first subpartition. One of the photos below shows Selfer being selected for booting from the first page of 125 subpartitions. The second photo shows the second sector of Selfer after it's been loaded in on boot having previously been saved using s, the code in it being test code for k, m, p and r (which all worked fine). All the F6s are simply values that were already on the flash drive - it would be easy to clear them to zeros, but I wouldn't want to save the result using w due to the excessive wear. The colourful stripe across it is my real-mode debugger, called into action by running a CC Int 3 instruction - this offers a direct root to reboot my OS from there (the VBR being stored further up memory along with the debugger). [Edit: I should have pressed h to switch the debugger into hex mode, but I can't be bothered making a replacement picture so I'll just leave it as it is.]
Heh, I thought you had written an extension to Selfer to add a neat debugger. A little disappoint, there. :)
DavidCooper wrote:For multifloppy partitions, I'm currently using 82h to label it as a multifloppy partition (sharing that value with CP/M). It would also make sense to put the name Selfer in at 03h in the first sector if there's enough room for that so that the name of whatever's there for each subpartition can be displayed. With my OS, it displays whatever's at that location unless it recognises a FAT12 disk in which case it uses the disk name further down the BPB instead (if it isn't blank), and with my own disk format it finds the disk name near the end of the bootsector next to the date and time of the last save, and 12 bytes of wear variables which count how many times the location on the drive has been written to (using 4 bytes for every half megabyte block of flash memory). It isn't necessary (or even possible) for Selfer to go to the same lengths - just displaying the name at 03h would do, and that's why putting the name Selfer there would be useful - the user could change it to the name of the OS or tool they're building, or to version numbers, or dates. Personally I would just keep my own OS in the first subpartition and use it like a fancy MBR to boot different versions of Selfer, but a 2nd sector extension to Selfer should ideally be designed to do the same job so that Selfer is capable of being completely independent.
I'm not too sure what standard specifies having the name of the partition at 0x03? Or is that simply a suggestion to make storing multiple versions of Selfer neater?

As for a fancy MBR to do that, I suppose one could extend Selfer such that it does that itself. You appear to be a born machine code programmer, so I'd imagine you can do it, right? :D

Amazing screenshot, anyhoo.
dozniak wrote:
Shikhin wrote:And it isn't stopping you from donating me money. :lol:
I spent a lot on Kickstarter, so I'll skip for now.
You mean to say you spent a lot of money on interesting, yet totally nonviable projects planned for the future, by people who have no job other than posting said projects on funding websites; however, you got no money to spend on a small little child writing real and viable programs for the education of others? :o

Regards,
Shikhin

Re: 512B (Selfer)

Posted: Tue Mar 26, 2013 5:13 pm
by DavidCooper
Hi Shikhin,
Shikhin wrote:I think I fixed that in the latest revision of the Gist. The LBA is now stored at 0x503, while the block (for int 0x13) is stored at 0x507.
I can't find the original info, but I've found something else that suggests the byte at 500h is part of the BDA, stating that it holds a PrtScr key status byte http://computer-programming-forum.com/4 ... 3602b7.htm. It may be that on some machines pressing that key could overwrite the stored drive number, so if I was you I'd move the storage array up by a byte, which will align all the contents more neatly into the bargain (though the microscopic speed gains in accessing them won't be recognisable).
Unfortunately, I still haven't gotten into the habit of creating virtual drives, putting a MBR on it, and putting Selfer into a VBR. It took a lot of experimentation to get it working with a pen drive in the first case -- right now, it's a bit too tough to do it again. Could you test if it's completely fixed, now?
I didn't test it before as I didn't want any bug writing to the flash drive if things went wrong, but as you've changed it it should be fine now. I'll test it now, but I can already see that it will work as that bug's gone. I wouldn't use the LBA version with virtual drives at all as it would take up a lot of space storing it all - the idea is to use it on real hardware which is a much more rewarding environment to work in, just so long as you don't damage it. QEMU lets you test everything in an emulator from a real flash drive first though, so you can get the best of both worlds.

What I do is start with a new flash drive and go into the MBR using a hex editor. I look at the partition table, and usually there's just the one entry specifying a FAT32 partition. I reduce the size of the FAT32 partition and create a new partition to sit in the reclaimed space. Bytes 05h-07h in the FAT32 partition's entry need to be changed to a new top CHS address for the partition. Bytes 0Ch-0Fh also need to be changed to the start LBA of the new partition minus the value that's already sitting in bytes 08h-0Bh. With larger flash drives (e.g. 16GB) the new CHS value is likely to be FE FF FF.

The second table entry will start with 80h followed by the CHS for the start of the new partition, then perhaps 82h (any value that Windows doesn't care for will do) followed by the CHS for the end of the new partition. The next four bytes are the LBA for the start of the new partition and the four bytes after that are the size of that partition. Use the original top of the original partition as the new top of the new partition - don't move it upwards even if the size of the drive suggests there's more space above there as there may be a good reason for that space not being used.

The next step is to open the drive as a logical drive instead of a physical drive and to put in a new number of sectors for the FAT32 partition at 20h of the BPB (this will be the same value as used in the last four bytes of the first partition table entry). The FAT32 partition can then be formatted in Windows to make it fit the reduced space properly. The only other thing you need is standard MBR code to boot up the bootable partition and point DS:SI at the correct table entry. Oh, and you obviously have to paste Selfer into the right place as well.

The new partition should start at an LBA location which is a multiple of 1024 so that wear can be measured more easily. Some flash drives take seconds to save when you use the BIOS, so if the one you're using is unresponsive, it's really best to avoid using it for anything that saves through the BIOS. I've tried a number of different types and Sandisk ones respond well: I prefer the type that only stick out a few millimetres as they're 99.9% less likely to be leant on sufficiently hard by the user to do damage. 200MB would be a good size for a multifloppy partition, though I use nearer a gigabyte. It really depends on how much you intend to use Selfer: for building a few tools on top of it you won't need more than 200MB, but for a serious attempt to build an OS with it you'd want a lot more. With my OS I tend to plug in two or three flash drives plus an SD card and sometimes a USB floppy disk drive as well, and the BIOS allows me to boot my OS from any of these and then for the booted one to boot up any other version of itself from any of the other drives (which often means that they boot using drive numbers 82h, 83h and 84h, 81h being used for the internal hard drive). I can also use any version of my OS to boot up an MBR on the same or another drive, and to do so with the debugger active. That kind of functionality could also be added to Selfer, though of course it always depends on the BIOS of the machine you're using as to whether all these things are going to be possible on it - some may restrict you to using a single flash drive.
I have no clue on that error -- since I assume the state of a lot of registers in Selfer (e.g. "this was set to X here, that was set to X there, this xor that is zero, and oh! neat, it's also required in a BIOS call, and the BIOS call isn't expected to touch that register, so wait, I can inc it here, and I get 1 in the register!"), a small little error in the dummy entry could cause very much unexpected behavior.
I still don't understand the problem so I'll have to look at it carefully. The only thing wrong in the dummy table entry was the LBA, so it should simply have loaded in the wrong sectors without causing any other problems, and yet the result was that it saved sectors and moved the cursor to the right when x was pressed. I'll have to explore it to find out exactly why that happened because putting in the right LBA value shouldn't be able to cure it, even though it appears to do just that. It's just possible it was loading in something with a nop at 7E00 and leading to it running something that caused havoc.
However, for Selfer, we don't have the space to not trust the BIOS.
Moving the saving of DL and the LBA ahead of it would still be safer, and if it takes more bytes to do so, there are some that can be won back:-

Code: Select all

push bp
push si
push ds
push es
 
call bx
 
.RetRun:
pop es
pop ds
pop si
pop bp
BP and SI can be done in one go with a single byte, along with all the other registers, and restored in the same manner. I see that pushad gets a prefix and takes two bytes, so I'm guessing there's a pusha or pushaw instruction that doesn't generate the prefix.

Also, if you look above the cmp al, 'M' part of the program, there are five jumps to .Next which use three bytes for their jump rather than two. If you put in a label just above the jmp .Next in the .IncrementPointer part of the program, the five jumps to .Next higher up can be diverted to that new label instead which they can reach with just two bytes instead of three, thereby saving five bytes.
Heh, I thought you had written an extension to Selfer to add a neat debugger. A little disappoint, there. :)
It could be added to Selfer - it was written in machine code and a modified version could be typed in while using Selfer. Alternatively it could be treated as a binary blob and code could be added to Selfer to load it into the place in memory where it's designed to sit (00080000h). That would probably be the best way to do it as it would be easier to upgrade it to later versions (the current version can only handle real mode and a text screen as was designed primarily to help debug MBRs and VBRs) - there's probably no need to be too much of a purist about everything being written through Selfer when it's effectively been written in the same manner already and so there's no harm in avoiding the pointless task of typing it all in again.
I'm not too sure what standard specifies having the name of the partition at 0x03? Or is that simply a suggestion to make storing multiple versions of Selfer neater?
If names are to show up in the list, they have to go somewhere so a place must be chosen. If there's no BPB, there's no telling where any name will be, but I just decided always to put 11 bytes of some kind of name in at 03h with all my VBRs to make it easy for anyone looking at them through a hex editor to find out what they are. 8 bytes or even just 6 bytes of a name would do the job though, and if there are only 6 they could start at 05h instead and have exactly enough room for a far jump to go in front of them to make CS=0, so no space is wasted. The bytes at 03h and 04h would also conveniently be zeros and would therefore display neatly as spaces instead of random mess.
As for a fancy MBR to do that, I suppose one could extend Selfer such that it does that itself. You appear to be a born machine code programmer, so I'd imagine you can do it, right? :D
You don't need a fancy MBR to do it - the MBR only needs to boot up the first subpartition and doesn't need to know anything about subpartitions at all. It would then be for extra code in the second (or some later) sector of Selfer to provide the means to display a list of subpartitions and their contents while offering to boot any of them which are bootable. At that point, Selfer behaves like a fancy MBR when booting up other versions of itself.
Amazing screenshot, anyhoo.
Thanks, but it's really fairly simple code which could potentially be added to Selfer, so what I might now try to do (if it doesn't take up too much time) is build a second sector for Selfer which does all that stuff, relying on using autorun (putting a nop at 7E00) to patch a jump into the two nops in Selfer (which are currently still in the wrong place) to extend the functionality.

Re: 512B (Selfer)

Posted: Mon Apr 01, 2013 5:04 pm
by DavidCooper
I've found the original bit I read about the byte at 500h. It's in the Wikipedia entry on MBRs, just past half way down in a section called "Programming considerations".
While the MBR boot sector code expects to be loaded at physical address 0000h:7C00h, all the memory from physical address 0000h:0501h (address 0000h:0500h is the last one used by a Phoenix BIOS)[12] to 0000h:7FFFh (and sometimes up to 0000h:FFFFh) is available in real mode.
I've just written a Web page with a JavaScript program in it which automates all the partition table entry calculations, so you can just fill in the details of the existing partition and select a size for a new multifloppy partition to go after it. It's designed for use with the new version of my own OS (which isn't available yet), but the program will also work for Selfer or any other floppy-disk sized OS capable of booting from a partition on a flash/hard drive.

[There is also a link to my multiboot MBR code which should work for up to four bootable partitions (it certainly did, but I've just modified it and haven't yet checked it again fully), but the extra capability to boot two different subpartitions (either the first or the most recent in a contiguous chain of progressively later versions) only works with my own OS, so there's no advantage in using it with Selfer (unless you somehow have enough room to put "Vb" at 0B-0Ch and store the time and date just before the 55 AA at the end in CMOS clock format: s, m, h, weekday, d, m, y, 20h).]

Re: 512B (Selfer)

Posted: Wed Oct 23, 2013 9:29 am
by shikhin
Hi,

I got busy in various other things, and couldn't work much on Selfer in the interim period. Anyhow, I've setup a Git repository for Selfer, now, and you can locate it on my GitHub profile. I'll first work on Selfer so as to meet the satisfaction of DavidCooper (who is probably the only person seriously following up on the project), after which, I had plans for a "512-byte Selfer module & application" environment -- it will be nothing more than a fun extension planned for it.

Regards,
Shikhin

Re: 512B (Selfer)

Posted: Wed Oct 23, 2013 11:15 am
by DavidCooper
I don't think it wants a lot more done to it really. Tidy up the few little problems with it and then move on to something else like USB. These old machines are going to be in museums soon, so what's really needed is something minimal that can load and save itself from/to a flash drive after booting via EFI. It would be a lot more fun to see how few sectors are required for that.

Re: 512B (Selfer)

Posted: Thu Oct 24, 2013 6:33 am
by shikhin
Hi,
DavidCooper wrote:I don't think it wants a lot more done to it really. Tidy up the few little problems with it and then move on to something else like USB. These old machines are going to be in museums soon, so what's really needed is something minimal that can load and save itself from/to a flash drive after booting via EFI. It would be a lot more fun to see how few sectors are required for that.
For a flash drive with BIOS, I think the version of Selfer using extended int 0x13 fits. However, I'm not too sure if I'm willing to write a 512-byte (or multiple thereof) Selfer-clone for EFI machines, mainly because there are more complexities added via being in protected mode, the bigger address space (and figuring out what part of memory is free), the (comparitively) more complex calling convention, and so on.

Regards,
Shikhin

Re: 512B (Selfer)

Posted: Thu Oct 24, 2013 10:29 am
by DavidCooper
I'm not suggesting you should do an EFI version. I was just trying to say that the BIOS is heading for extinction and you don't want to put a lot more work into anything that depends on it.