Standard Libraries

Programming, for all ages and all languages.
SoulofDeity
Member
Member
Posts: 193
Joined: Wed Jan 11, 2012 6:10 pm

Standard Libraries

Post by SoulofDeity »

In one of my other topics, I complained about the messiness and complexity of all the libraries we use today, so I've decided that for my operating system I'm going to create all libraries entirely from scratch. I'll be posting my concepts here and would like your opinions and constructive criticism.
Keep in mind that it's incomplete.

Memory Management Functions

Code: Select all

void *alloc(u64 size)
	Allocates memory, returns pointer to data.
void *realloc(void *data, u64 size);
	Allocates or reallocates memory, returns pointer to data.
void free(void *data)
	Frees memory.
Filesystem IO Functions

Code: Select all

u32 mount(Device *device, char *mntpoint, char *path)
	Mounts a device, at mntpoint, returns a value stating if it's
	successful or an error code.
void unmount(Device *device)
	Unmounts a device


u32 openDirectory(Directory *dir, char *path)
	Opens a directory, returns a value stating if it's
	successful or not.
void closeDirectory(Directory *dir)
	Closes a directory.
u32 endOfDirectory(Directory *dir)
	Returns whether or not the end of the directory has been
	reached.
u32 firstFile(Directory *dir, DirectoryEntry *entry)
	Returns the information for the first entry in a directory.
u32 nextFile(Directory *dir, DirectoryEntry *entry)
	Returns the information for the next entry in a directory.
u32 fileExists(char *path)
	Returns whether or not a file or directory exists.
u32 copyFile(char *srcpath, char *dstpath)
	Copies a file or directory, returning if it's successful.
u32 moveFile(char *srcpath, char *dstpath)
	Moves a file or directory, returning if it's successful.
u32 renameFile(char *path, char *newName)
	Renames a file or directory, returning if it's successful.
void deleteFile(char *path)
	Deletes a file or directory.


u32 openFile(File *file, char *path, u32 flags)
	Opens a file, returns a value stating if it's successful or
	an error code.
void closeFile(File *file)
	Closes a file.
u32 endOfFile(File *file)
	Returns whether or not the end of the file has been reached.
u64 filePosition(File *file)
	Returns the position of the file.
u64 fileSeek(File *file, u64 pos, u32 seekStart)
	Seeks to a position in a file.
u64 fileRead(File *file, void *buffer, u64 size)
	Reads bytes from a file into a buffer, returning the number
	of bytes actually read.
u64 fileWrite(File *file, void *buffer, u64 size)
	Writes bytes to a file from a buffer, returning the number
	of bytes actually written.
char *fileReadLine(File *file, void *buffer, u32 length)
        Reads a line from a file, returns a pointer to the string.
u8 fileReadByte(File *file)
	Reads a byte from a file, returning the value.
void fileReadHWBE(File *file, void *buffer)
	Reads a halfword (16-bits) from a file in big endian.
void fileReadHWLE(File *file, void *buffer)
	Reads a halfword (16-bits) from a file in little endian.
void fileReadWBE(File *file, void *buffer)
	Reads a word (32-bits) from a file in big endian.
void fileReadWLE(File *file, void *buffer)
	Reads a word (32-bits) from a file in little endian.
void fileReadDWBE(File *file, void *buffer)
	Reads a doubleword (64-bits) from a file in big endian.
void fileReadDWLE(File *file, void *buffer)
	Reads a doubleword (64-bits) from a file in little endian.
void filePrintf(File *file, char *format, ...)
        Prints formatted text to a file.
void fileWriteLine(File *file, void *buffer, u32 length)
        Writes a line from a file.
void fileWriteByte(File *file, u8 byte)
	Writes a byte to a file.
void fileWriteHWBE(File *file, void *buffer)
	Writes a halfword (16-bits) to a file in big endian.
void fileWriteHWLE(File *file, void *buffer)
	Writes a halfword (16-bits) to a file in little endian.
void fileWriteWBE(File *file, void *buffer)
	Writes a word (32-bits) to a file in big endian.
void fileWriteWLE(File *file, void *buffer)
	Writes a word (32-bits) to a file in little endian.
void fileWriteDWBE(File *file, void *buffer)
	Writes a doubleword (64-bits) to a file in big endian.
void fileWriteDWLE(File *file, void *buffer)
	Writes a doubleword (64-bits) to a file in little endian.
Antti
Member
Member
Posts: 923
Joined: Thu Jul 05, 2012 5:12 am
Location: Finland

Re: Standard Libraries

Post by Antti »

I agree that current libraries are messy and complex. However, it may well be that your own implementations will turn out messy and complex too. Things may be simple and elegant at first. Feature additions, bug fixes, workarounds, etc. can change this.
User avatar
thepowersgang
Member
Member
Posts: 734
Joined: Tue Dec 25, 2007 6:03 am
Libera.chat IRC: thePowersGang
Location: Perth, Western Australia
Contact:

Re: Standard Libraries

Post by thepowersgang »

I admire your concept of a complete redesign, but I doubt it's going to be useful (and the inclusion of the WriteHLE etc functions are not the best idea). Mostly because it mirrors (with minor naming changes) the existing stdio.h/stdlib.h functions. Despite some eccentricities of the argument orders of the stdio functions (fread/fwrite having the handle last unlike the rest of the f* functions) these are standard, meaning that if I write code for my C library on my OS, it would work on GNU/Linux, Windows, MacOS, ... it would just work. (Assuming I only use the C standard functions)

Let's also remember, the C stardard functions were designed by a committee who knew what they were doing, with years of programming expirence with older languages to call upon.
Kernel Development, It's the brain surgery of programming.
Acess2 OS (c) | Tifflin OS (rust) | mrustc - Rust compiler
Currently Working on: mrustc
SoulofDeity
Member
Member
Posts: 193
Joined: Wed Jan 11, 2012 6:10 pm

Re: Standard Libraries

Post by SoulofDeity »

Antti wrote:I agree that current libraries are messy and complex. However, it may well be that your own implementations will turn out messy and complex too. Things may be simple and elegant at first. Feature additions, bug fixes, workarounds, etc. can change this.
I've been thinking about that as well. One I idea I had is to make all libraries abstract, running on underlying classes. That would allow me to make system specific changes without have to adjust the libraries in any way.
thepowersgang wrote:I admire your concept of a complete redesign, but I doubt it's going to be useful (and the inclusion of the WriteHLE etc functions are not the best idea). Mostly because it mirrors (with minor naming changes) the existing stdio.h/stdlib.h functions. Despite some eccentricities of the argument orders of the stdio functions (fread/fwrite having the handle last unlike the rest of the f* functions) these are standard, meaning that if I write code for my C library on my OS, it would work on GNU/Linux, Windows, MacOS, ... it would just work. (Assuming I only use the C standard functions)

Let's also remember, the C stardard functions were designed by a committee who knew what they were doing, with years of programming expirence with older languages to call upon.
The addition of fileWriteHWLE (edit here: I meant HWLE not HLE) and the other similar functions were added for the sake of portability. The user wouldn't have to be concerned with the endianness of the machine their using when attempting to read data from a file. Obviously, I don't expect others to adopt the standard libraries that will be used by my operating system, but I don't want to be intimidated into thinking I shouldn't innovate simply because such-n-such says that everyone should adhere to what they say is standard.
Last edited by SoulofDeity on Mon Nov 26, 2012 1:51 am, edited 1 time in total.
User avatar
bluemoon
Member
Member
Posts: 1761
Joined: Wed Dec 01, 2010 3:41 am
Location: Hong Kong

Re: Standard Libraries

Post by bluemoon »

SoulofDeity wrote:I don't want to be intimidated into thinking I shouldn't innovate simply because such-n-such says that everyone should adhere to what they say is standard.
You are free to revolutionize the standard, people are just pointing out what's common issue to be face/resolve.
SoulofDeity
Member
Member
Posts: 193
Joined: Wed Jan 11, 2012 6:10 pm

Re: Standard Libraries

Post by SoulofDeity »

bluemoon wrote:
SoulofDeity wrote:I don't want to be intimidated into thinking I shouldn't innovate simply because such-n-such says that everyone should adhere to what they say is standard.
You are free to revolutionize the standard, people are just pointing out what's common issue to be face/resolve.
Definitely a molehill :P (btw, by such-n-such I was referring to the committee not him. read that way it sounds rude...)

The way I see it though, operating system development been a 1 way direction for quite some time. It started with DOS and Unix, and all other operating systems simple added onto to those ideas, expanded little-by-little, and copied each other until the point we're at today. Worrying so much about backwards compatibility and what their competitors can and can't do leached away innovation, and what we know as standard today is simply a definition of what everyone does so they can look at each other, point, and say "hey, I can do that too."
If I focus on my goals and my ideas, and strive to be different, I may one day succeed in making something spectacular. Then maybe some time a few decades down the road, those big operating systems of today will strive to recreate what I've done.
Kevin
Member
Member
Posts: 1071
Joined: Sun Feb 01, 2009 6:11 am
Location: Germany
Contact:

Re: Standard Libraries

Post by Kevin »

SoulofDeity wrote:If I focus on my goals and my ideas, and strive to be different, I may one day succeed in making something spectacular.
I think the point people are making is that this draft isn't really different, it's just an obfuscated C standard library. Here a function renamed, there an u64 instead of a size_t, but that's about it. You add a few endianess-aware read/write functions, which may be nice in theory, but they aren't more than one-line wrappers around fread/fwrite (and I practice I don't write single dwords, but whole structs, so it becomes much less useful - and solving the problem with structs is probably not possible in pure C).

The concepts are exactly the same. And as long as this is true, you're probably better off writing a C standard library. I would suggest that for each function that you specify, you try to add the pros and the cons compared to the standard functions. In the end it should be clearer whether it's really worth it or if it just says "it's the same" for most functions.

Btw, I always hated this FindFirst/FindNext thing when I wrote Pascal code on DOS...
Developer of tyndur - community OS of Lowlevel (German)
User avatar
bluemoon
Member
Member
Posts: 1761
Joined: Wed Dec 01, 2010 3:41 am
Location: Hong Kong

Re: Standard Libraries

Post by bluemoon »

To some extend, standard C library are not meant to be fast and convenient, but portable. There are in fact some better, and totally different designs, and used by several OSes, for example the event-based IO (Completion IO, kqueue), IPC model (not just signal()), etc...

But in the end, it's so much benefit to build the LibC and POSIX layer, and many people decided to build that on top of their OSes.
SoulofDeity
Member
Member
Posts: 193
Joined: Wed Jan 11, 2012 6:10 pm

Re: Standard Libraries

Post by SoulofDeity »

Kevin wrote:
SoulofDeity wrote:If I focus on my goals and my ideas, and strive to be different, I may one day succeed in making something spectacular.
I think the point people are making is that this draft isn't really different, it's just an obfuscated C standard library.
Its file i/o, of course there's gonna similarities, and generally, this is just the beginning. I'm going to be coming up with new ways to handle internet protocol, audio playing, and pretty much recreating all common libraries such as truetype, libpng, jpeglib, etc.
Kevin wrote:Here a function renamed, there an u64 instead of a size_t
1. Not a function, it's a type definition
2. I used an unsigned 64-bit value for support for files > 4GB.
Kevin wrote:You add a few endianess-aware read/write functions, which may be nice in theory, but they aren't more than one-line wrappers around fread/fwrite (and I practice I don't write single dwords, but whole structs, so it becomes much less useful - and solving the problem with structs is probably not possible in pure C).
If you know the endianness of the machine you're using, which means you have to make your source code platform-specific, and dirty up your code with a bunch of if/defs. Just because you commonly use structs doesn't mean that the feature is pointless. That's like saying fprintf is pointless because you personally have never had to write a formatted string to a file. But take into consideration that that there are thousands of different file formats make use of big or little endian byte ordering (and some such as .blend that use both), It can be pretty valuable.
Kevin wrote:The concepts are exactly the same. And as long as this is true, you're probably better off writing a C standard library.
No they're not. Here's a list of differences right here:
  • Functions added for platform-independent field reading
  • File mode is u32 instead of a char *, which make parsing them and opening files faster (even if by only a couple of microseconds)
  • Arguments are more consistent
  • Function names are more easily recognizable
  • Devices and files can be mounted and unmounted at specific mount points. For example, you can mount "cdimage.iso" then use file i/o functions on it via mount point.
Of course it's not much, but like I said file i/o is file i/o. There's nothing that can really be done to improve how files are read aside from improving the api.
Kevin wrote:Btw, I always hated this FindFirst/FindNext thing when I wrote Pascal code on DOS...
What would you propose as an alternative?



EDIT:
Just a couple of ideas that came to me after what Blue moon said, I might adds some functions for asynchronous reading and writing, progress scanning, setting file attributes, creating symbolic links, and filesystem monitoring.
Last edited by SoulofDeity on Mon Nov 26, 2012 3:45 am, edited 1 time in total.
User avatar
Griwes
Member
Member
Posts: 374
Joined: Sat Jul 30, 2011 10:07 am
Libera.chat IRC: Griwes
Location: Wrocław/Racibórz, Poland
Contact:

Re: Standard Libraries

Post by Griwes »

While the POSIX layer is arguable, the *standard* C (and, of course, C++) libraries are a must for something you want to call "complete OS" at some point. No-one is going to learn that weird "standard" (sic) library of yours and move to your OS, if the only implemented library is not standard and would require rewriting the user code.

So here's the deal: write your own standard library implementation, but don't redesign it (poorly) and call it "standard", but implement what is defined in C and C++ standard (in my case, for example, I'm implementing C++ stdlib first, then C library on top of it - but you can do whatever you want), and design own libraries only for what is not in standard (and, in case of C++, is not in Boost either, maybe except crazy things like boost.log). Even Windows has standard-compliant C and C++ libraries - though they have their own.

And the last point. You were bitching about libpng before - what does it have to do with *standard* libraries?

----

Your post came before I posted, so here is the extended answer:
Its file i/o, of course there's gonna similarities, and generally, this is just the beginning. I'm going to be coming up with new ways to handle internet protocol, audio playing, and pretty much recreating all common libraries such as truetype, libpng, jpeglib, etc.
Do it! I'm all for you. But DON'T REINVENT *STANDARD* THINGS IN POOR WAY!
2. I used an unsigned 64-bit value for support for files > 4GB.
You do know that size_t (being standard) is often the same as uint64_t (again, I'm using standard name for that type), don't you?
Functions added for platform-independent field reading
lol
File mode is u32 instead of a char *, which make parsing them and opening files faster (even if by only a couple of microseconds)
I don't know what u32 is, so I cannot comment on this one.
Arguments are more consistent
You tell us that you are more competent library designer than Standard Committee, when you've sketched poor design in five minutes?
Function names are more easily recognizable
No, they aren't, because they are not standard.
Devices and files can be mounted and unmounted at specific mount points. For example, you can mount "cdimage.iso" then use file i/o functions on it via mount point.
Fine, but it has nothing to do with *standard*, as standard C and C++ have no notion of mountpoint (and look at the topic - *standard* libraries!).
Reaver Project :: Repository :: Ohloh project page
<klange> This is a horror story about what happens when you need a hammer and all you have is the skulls of the damned.
<drake1> as long as the lock is read and modified by atomic operations
User avatar
bluemoon
Member
Member
Posts: 1761
Joined: Wed Dec 01, 2010 3:41 am
Location: Hong Kong

Re: Standard Libraries

Post by bluemoon »

SoulofDeity wrote:If you know the endianness of the machine you're using, which means you have to make your source code platform-specific, and dirty up your code with a bunch of if/defs
No, you don't need a bunch of #if, there is htonl family macro.
Although I define "host-to-endian-little" family in some case to optimize for intel machines, as well as the 64-bit counterparts.
SoulofDeity
Member
Member
Posts: 193
Joined: Wed Jan 11, 2012 6:10 pm

Re: Standard Libraries

Post by SoulofDeity »

Griwes wrote:While the POSIX layer is arguable, the *standard* C (and, of course, C++) libraries are a must for something you want to call "complete OS" at some point. No-one is going to learn that weird "standard" (sic) library of yours and move to your OS, if the only implemented library is not standard and would require rewriting the user code.
My "sic" library is incomplete. This isn't even a basic draft or anything. This is a brainstorm session..
Griwes wrote:And the last point. You were bitching about libpng before - what does it have to do with *standard* libraries?
Nothing. It has to do with libraries in general. Like I said, I want to design my os from the ground up.

Griwes wrote:
Its file i/o, of course there's gonna similarities, and generally, this is just the beginning. I'm going to be coming up with new ways to handle internet protocol, audio playing, and pretty much recreating all common libraries such as truetype, libpng, jpeglib, etc.
Do it! I'm all for you. But DON'T REINVENT *STANDARD* THINGS IN POOR WAY!
Again, it's unfinished. This is just a brainstorming session.
Griwes wrote:
2. I used an unsigned 64-bit value for support for files > 4GB.
You do know that size_t (being standard) is often the same as uint64_t (again, I'm using standard name for that type), don't you?
Yep. That's why I said it. He called it a function, and I was just clarifying why I didn't choose some other data type.
Griwes wrote:
Functions added for platform-independent field reading
lol
File mode is u32 instead of a char *, which make parsing them and opening files faster (even if by only a couple of microseconds)
I don't know what u32 is, so I cannot comment on this one.
Unsigned 32-bit integer.
Griwes wrote:
Arguments are more consistent
You tell us that you are more competent library designer than Standard Committee, when you've sketched poor design in five minutes?
No, I'm said that the ordering of the arguments is more consistent. In the standard C library, sometimes the file point is first, sometimes its last, and there 2 fields for the size of the buffer in fread and fwrite which is pointless because you can simply multiply the count * sizeofthestructure.
Griwes wrote:
Function names are more easily recognizable
No, they aren't, because they are not standard.
That's just the stupidest thing I've ever heard. Thats like saying "fffid" is more understandable than "FindFirstFileInDirectory" simply because it's standard.


Honestly, I'm put off by the attitude. If you like your C standard library so much than stick with it. There's no sense it getting defensive like this if you don't actually believe I have the potential to make something better.
User avatar
Griwes
Member
Member
Posts: 374
Joined: Sat Jul 30, 2011 10:07 am
Libera.chat IRC: Griwes
Location: Wrocław/Racibórz, Poland
Contact:

Re: Pseudo-standard Libraries

Post by Griwes »

Yep. That's why I said it. He called it a function, and I was just clarifying why I didn't choose some other data type.
Here a function renamed, there an u64 instead of a size_t
Huh, "here" is one thing; then comes comma, then comes "there". He could've used list to say that:
- you renamed some function
- you used u64 instead of size_t

Your Englishfoo is too low for us, I think.
Nothing. It has to do with libraries in general. Like I said, I want to design my os from the ground up.
Then do it, but don't touch standard library design, because not having STANDARD (your is NOT standard) is going to make your OS dead from lack of users.

And "standard library" is not "library in general"; C and C++ standard libraries are standardized probably for longer than you are programming (and surely longer than I am), so stop arguing with smarter people's ideas.
Unsigned 32-bit integer.
Unsigned 32-bit integer is called `uint32_t` by all C, C++ and (I think) POSIX standards. Call it as such, we are in "standard libraries" thread after all.
That's just the stupidest thing I've ever heard. Thats like saying "fffid" is more understandable than "FindFirstFileInDirectory" simply because it's standard.
Again, look at YOUR thread topic. It says "standard".
1) Standard things are more understandable, because everybody either know exact arguments of that function or know exactly where to look for them.
2) Standard things are more understandable, because they are standard and everybody already uses them.
3) FindFirstFileInDirectory is a b i t c h* to write and read; fffid is much faster and shorter and has better capitalization style.
Honestly, I'm put off by the attitude. If you like your C standard library so much than stick with it. There's no sense it getting defensive like this if you don't actually believe I have the potential to make something better.
The point is, it's standard! You are free to submit standard proposals to C and C++ Standard Committees; then, the most experienced in field of standardization people will look at it and reject it, laughing at them.


---

* thanks, word filtering... no, seriously. You cannot write even that?
Reaver Project :: Repository :: Ohloh project page
<klange> This is a horror story about what happens when you need a hammer and all you have is the skulls of the damned.
<drake1> as long as the lock is read and modified by atomic operations
SoulofDeity
Member
Member
Posts: 193
Joined: Wed Jan 11, 2012 6:10 pm

Re: Pseudo-standard Libraries

Post by SoulofDeity »

I'm just gonna sum up everything in 1 sentence. "I don't care about what's standard, I care about what is most efficient and understandable."


I created this topic more or less to discuss with people where many improvements could be made in all libraries (including C standard libraries). Instead of just bashing my idea, tell me where improvements can be made.


btw, just noticed the topic name change :lol:
gerryg400
Member
Member
Posts: 1801
Joined: Thu Mar 25, 2010 11:26 pm
Location: Melbourne, Australia

Re: Pseudo-standard Libraries

Post by gerryg400 »

SoulofDeity wrote:I'm just gonna sum up everything in 1 sentence. "I don't care about what's standard, I care about what is most efficient and understandable."


I created this topic more or less to discuss with people where many improvements could be made in all libraries (including C standard libraries). Instead of just bashing my idea, tell me where improvements can be made.


btw, just noticed the topic name change :lol:
SoulofDeity, I'm interested in your memory interface.

I'm currently implementing my userspace memory allocation routines and am following the Posix spec. Do you have any improvements in mind ? The only difference I can see is using u64 for size. I can't use this because 64 bit ints are inefficient on some of the processors I target. Are there any other improvements you can suggest ?
If a trainstation is where trains stop, what is a workstation ?
Post Reply