Server OS in ten seconds flat

[NickJohnson]

Hi -- it's been a while.

I'm starting a new project, with a very concrete goal: to construct a usable Unix-like server OS in as little time as possible. The requirements for it to be complete are as follows:

- It supports a stock Unix-like userland.
- It has a TCP/IP stack adequate for serving HTTP requests.
- It can run Apache httpd.
- It can be run from a CD or hard disk.

The goal is for two developers (a friend and me) to complete this in under six months, starting next Monday. This seemed like an only mildly insane timeframe, such that it will be exciting even though the end result is yet another Unix-like system using a mostly stock userland.

It will, of course, be open source, with the kernel most likely under an OpenBSD-style license. Whether or not the source will be publicly available before release is still being decided, as is the name. Either way, I will try to keep this thread updated with some design details. Either this will serve as documentation on how to complete a functional OS in six months, or as a warning to those who would dare attempt such an audacious exercise.

[gerryg400]

Will you re-use some rhombus code ? Microkernel?

[NickJohnson]

The design is going to be a pure monolithic kernel, at least at first, because it's the easiest to construct. We had some interesting ideas involving microkernel-style drivers that communicated over the kernel's TCP/IP stack, but that is definitely out of the scope of the initial project. Basically, it's going to be architecturally boring and Linux-like; the whole point is the speed at which it is going to be built. Writing a TCP/IP stack is going to be enough of a challenge without any other complications.

It probably won't have much Rhombus code. We're thinking of making it 64 bit only, since that simplifies certain things, and we don't really care about changing video modes since it's a server system. It will also likely need to support kernel preemption, because it's monolithic, which makes it architecturally different enough from the Rhombus kernel that not much would transfer (because the Rhombus kernel is 50% the memory manager and the scheduler, and most of the rest is idiosyncratic IPC stuff.)

[cxzuk]

id love to see a diary, blog or epic summary go with this

[gravaera]

This will be interesting, and I wish you good luck :O

--Peace out
gravaera

[NickJohnson]

Well, we may end up using nginx. The only reasoning at this point behind using Apache is that on the default Arch install of both nginx and Apache httpd, Apache httpd uses a lot fewer shared libraries:

readelf -d /usr/sbin/nginx

Code: Select all

 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpcre.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.1.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.1.0.0]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

readelf -d /usr/sbin/httpd

Code: Select all

 0x0000000000000001 (NEEDED)             Shared library: [libpcre.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libaprutil-1.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libapr-1.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

The point is that it has to run a webserver of some sort.

[Brynet-Inc]

0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
..

"Standard" stuff.

0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
..

Why is nginx linked with libstd++ and libm Linux? it isn't on OpenBSD. Do they just link with every shared library for the hell of it?

0x0000000000000001 (NEEDED) Shared library: [libpcre.so.1]
0x0000000000000001 (NEEDED) Shared library: [libcrypt.so.1]
0x0000000000000001 (NEEDED) Shared library: [libssl.so.1.0.0]
0x0000000000000001 (NEEDED) Shared library: [libcrypto.so.1.0.0]
0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]
0x0000000000000001 (NEEDED) Shared library: [libz.so.1]

None of these are important, SSL can be disabled.. pcre can be linked statially (..or disabled), libdl is a linux thing, and I imagine libz can be disabled (..but it should be easy to port).

Still, good luck with the project.

[brain]

Have you run readelf against libapr and libaprutil? Both are dependency-heavy iirc...

[bluemoon]

And apache without php and other fancy stuff is not much useful, however they make giant dependency tree.
On the other hand, you may just write a simple httpd using sockets with virtually no dependency at all.

[JamesM]

Have you run readelf against libapr and libaprutil? Both are dependency-heavy iirc...

Having being part of a project that ported Apache, I can attest to this.

[turdus]

Well, if you're looking for a very simple http server, here's one I wrote for an embedded system. Note, it's not stupid-proof (not much error handling), neither performance oriented (uses fork), but it's only a few lines of C code with no library dependency at all. It handles GET and POST http requests. You can statically link it with php if you like.

Code: Select all

/* 
 *  This program is free software; you can redistribute it and/or modify 
 *  it under the terms of the GNU General Public License as published by 
 *  the Free Software Foundation; either version 2 of the License, or 
 *  (at your option) any later version. 
 * 
 *  This program is distributed in the hope that it will be useful, 
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of 
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the 
 *  GNU General Public License for more details. 
 * 
 *  You should have received a copy of the GNU General Public License 
 *  along with this program; if not, write to the Free Software 
 *  Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 
 */
#define __USE_GNU
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <netdb.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#include <time.h>
#include "libs.h"		//split(separatorlist,string,index)

#define	METHOD_GET	1
#define	METHOD_POST	2

char http_method=0;
char *http_req=NULL;
char *http_requesturi=NULL;
char *http_param=NULL;
char *http_module=NULL;
char *http_language=NULL;
char *http_remote=NULL;

int module_argc=0;
char **module_argv=NULL;

extern char *strndup (__const char *__string, size_t __n);
void http_request();

/************************************************************************** 
	parses http header information
	sets http_language and http_param and others
 **************************************************************************/ 
int parse_header(char *header)
{
	char *ptr=NULL,*ptr2=NULL;
	unsigned int i,j,postlen=0;

	if(!strncmp(header,"GET",3)) { http_method=METHOD_GET; header+=4; }
	else if(!strncmp(header,"POST",4)) { http_method=METHOD_POST; header+=5; }
	else return -1;
	while(header[0]==' ')header++;
	j=0;while(header[j]!=' '&&header[j]!='\r'&&header[j]!='\n'&&header[j]!=0)j++;
	http_requesturi=(char*)strndup(header,j);
	ptr2=++header;

	//get the browser default (first) language
	i=0;while(header[i]!='\r'&&header[i]!='\n'&&header[i]!=0)i++;
	for(;i<strlen(header);i++){
			if(!strncasecmp(&header[i],"accept-language",15)){
				i+=16; while(header[i]==' '||header[i]=='\r'||header[i]=='\n')i++;
				if(header[i]!='\n'&&header[i]!='\r'&&header[i]!=0){
					j=i; while(header[i]!=';' && header[i]!='\r' && header[i]!=',' && header[i]!='\n' && header[i]!=0)i++;
					http_language=(char*)strndup(&header[j],i-j);
				}//if got language
			} else //if accept language header
			if(!strncasecmp(&header[i],"content-length",14)){
				i+=15; while(header[i]==' '||header[i]=='\r'||header[i]=='\n')i++;
				sscanf(&header[i],"%d",&postlen);
			} //if content-length
	}//for
	if(http_language==NULL) http_language="en";
	
	while(*header!='?' && *header!=' ' && *header!=0) header++;
	http_module=split("/? ",ptr2,1);

	if(*header=='?') header++;
	ptr=header; while(*ptr!='\n' && *ptr!='\r' && *ptr!=0) ptr++;
	while(*ptr!=' ') ptr--;
	http_param=malloc(ptr-header+1);
	memset(http_param,'\0',ptr-header+1);
	memcpy(http_param,header,ptr-header);
	//if it's a post, we have more arguments waiting
	if(http_method==METHOD_POST && postlen>0){
		int len=strlen(http_param);
		http_param=realloc(http_param,len+1+postlen+1);
		if(len!=0) { http_param[len+1]=0; http_param[len]='&'; len++; }
		memset(http_param+len,'\0',postlen+1);
		i=fread(http_param+len,postlen,1,stdin);
	}
	return 0;
}

/************************************************************************** 
	add a key=value pair to arguments
 **************************************************************************/ 
void add_argument(char *name,char *value)
{
	if(value==NULL) return;
	module_argc++;
	module_argv=realloc(module_argv,module_argc*sizeof(char*));
	if(name!=NULL){
	    module_argv[module_argc-1]=malloc(strlen(name)+strlen(value)+2);
	    sprintf(module_argv[module_argc-1],"%s=%s",name,value);
	} else {
	    module_argv[module_argc-1]=malloc(strlen(value)+1);
	    module_argv[module_argc-1]=(char*)strndup(value,strlen(value));
	}
}
/************************************************************************** 
	parse url encoded arguments
 **************************************************************************/ 
void parse_arguments(char *param)
{
	int i=0,c=0,size=0;
	if(param==NULL || param[0]==0) return;
	module_argc++;
	module_argv=realloc(module_argv,module_argc*sizeof(char*));
	module_argv[module_argc-1]=NULL; size=0;
	while(param[i]!=0){
		if(param[i]=='&'){
			i++;
			if(!strncasecmp(&param[i],"amp;",4)) {
			    i+=4;
			    module_argv[module_argc-1]=realloc(module_argv[module_argc-1],size+2);
			    module_argv[module_argc-1][size]='&';
			    module_argv[module_argc-1][size+1]=0;
			    size++;
			} else {
			    module_argc++;
			    module_argv=realloc(module_argv,module_argc*sizeof(char*));
			    module_argv[module_argc-1]=NULL;
			    size=0;
			}
		}
		else if(param[i]=='%'){
			i++;
			sscanf(&param[i],"%02x",&c);
			i+=2;
			module_argv[module_argc-1]=realloc(module_argv[module_argc-1],size+2);
			module_argv[module_argc-1][size]=(char)c;
			module_argv[module_argc-1][size+1]=0;
			size++;
		}
		else if(param[i]!=0){
			module_argv[module_argc-1]=realloc(module_argv[module_argc-1],size+2);
			module_argv[module_argc-1][size]=param[i]=='+'?' ':param[i];
			module_argv[module_argc-1][size+1]=0;
			size++;
			i++;
		}
	}//while
}

/************************************************************************** 
	main function, a mini httpd
 **************************************************************************/ 
void mini_httpd(char *http_listenon)
{
	struct sockaddr_in server,connected;
	socklen_t length=sizeof server;
	fd_set rdfs;
	struct hostent *hent;
	int port=80;
	int webserver_sock,conn_sock=1,val=1;
	char *buf;

	if(httpd_listenon==NULL) httpd_listenon="0.0.0.0:80";

	//get listen parameters
	memset(&server,0,length);
	buf=split(":",httpd_listenon,1);
	if(buf!=NULL) port=atoi(buf);
	if(port<1) port=80;
	server.sin_port=htons(port);
	buf=split(":",httpd_listenon,0);
	if(buf==NULL||strcmp(buf,"0.0.0.0")==0){
	    server.sin_family=AF_INET;
	    server.sin_addr.s_addr=INADDR_ANY;
	} else {
	    if((hent=gethostbyname(buf))==(struct hostent *)NULL) {
		sys_log('C',"HTTPD: Unable to locate %s", buf);
	    }
	    if(hent->h_length>sizeof(hent->h_addr)) {
		sys_log('C', "HTTPD: Buffer overflow in gethostbyname()");
	    }
	    server.sin_family=hent->h_addrtype;
	    server.sin_addr.s_addr=((struct in_addr *)(hent->h_addr))->s_addr;
	}
	free(buf); buf=NULL;


	if((webserver_sock=socket(AF_INET,SOCK_STREAM,0))<0){
	    sys_log('C',"HTTPD: unable to create socket");
	}
	(void) fcntl( conn_sock, F_SETFD, 1 );
	setsockopt(webserver_sock,SOL_SOCKET,SO_REUSEADDR,&val,sizeof(val));
	
	if( bind(webserver_sock,(struct sockaddr *) &server,length)==-1 ||
		getsockname(webserver_sock,(struct sockaddr *) &server,&length)==-1 ||
		listen(webserver_sock,16)<0){
			sys_log('C',"HTTPD: couldn't listen on %s:%d",inet_ntoa(server.sin_addr),port);
	}

	sys_log('N',"HTTPD: listen on %s",httpd_listenon);
	//let's start accepting http connections
	while(1){
		FD_ZERO(&rdfs);
		FD_SET(webserver_sock, &rdfs);
		select(webserver_sock+1, &rdfs, NULL, NULL, NULL);
		if(FD_ISSET(webserver_sock, &rdfs)) {
			conn_sock=accept(webserver_sock,(struct sockaddr *)&connected,&length);
			//start a child to handle connection
			if(fork()==0){
				http_remote=inet_ntoa(connected.sin_addr);
				//redirect stdin/stdout to conn_sock
				dup2(conn_sock,0);
				dup2(conn_sock,1);
				close(webserver_sock);
				sys_log('D',"HTTPD: connected via minihttpd (%s)",http_remote);
				//ok we have a connection now let's read the HTTP request
				http_req=realloc(http_req,1);
				http_req[0]=0;
				while(1){
					http_req=realloc(http_req,strlen(http_req)+1025);
					memset(http_req+strlen(http_req),0,1025);
					//read the HTTP header
					if(fgets(&http_req[strlen(http_req)],1024,stdin)!=NULL){
						//two empty line means end of header
						if(strlen(http_req)>=4 && (
							!strncmp(http_req+strlen(http_req)-2,"\n\n",2) ||
							!strncmp(http_req+strlen(http_req)-4,"\n\r\n\r",4) ||
							!strncmp(http_req+strlen(http_req)-4,"\r\n\r\n",4)
						)) break;
					} else sys_log('E',"HTTPD: request read problem");
				}
				//handle the request
				parse_header(http_req);
				http_request();
				//close channel
				close(conn_sock);
				exit(0);
			} else close(conn_sock); //end fork
		} //IF input ready
	} //endless loop
	exit(0);
}

/************************************************************************** 
	handle a http request, do it as you like
 **************************************************************************/ 
void http_request()
{
	//here you can call php and pass variables to it
	//just like apache mod_php do.

	//http response header
	printf("HTTP/1.1 200 OK\r\n");

	//parse request params into array
	module_argc=0;
	module_argv=NULL;
	parse_arguments(http_param);
	printf("<html><body>called as: %s<br>Arguments:<br>",http_requesturi);
	for(i=0;i<module_argc;i++){ printf("%s<br>",module_argv[i]); }
	printf("</body></html>");
	fflush(stdout);
}

[brain]

Ugh turdus my eyes! They burn! You're right its not very neat or efficient but for what it does it sure is compact, could do with more comments and less instructions per line imho. reminds me of obfuscsted perl...

[Solar]

@ turdus:

The coding style in parse_header() is almost a capital offense. Would you believe me when I say that I have actually seen worse - once?

[thepowersgang]

@Solar:
Was it the irssistats source? Because that code is just evil.

[MasterLee]

Maybe you could use TNTnet.