$Id: elinux.txt,v 1.2 1999/12/01 17:54:34 bjornw Exp $

Programmers guide to the difference between eLinux and normal Linux
-------------------------------------------------------------------
BW 991028

eLinux is a Linux 2.0.33 kernel using the uC-linux project patches
for disabling the kernel memory management, our own work of porting
to the Etrax100/CRIS architecture, and our drivers for Etrax100
interfaces.

The memory management patches (hereafter referred to as the NO_MM
patches) have consequences both internally in the kernel and as well
externally for user programs. The Etrax100 porting has mainly internal
consequences but some are affecting the user-space as we will see below.

1. Removal of memory management
-------------------------------

The Etrax100 chip has no MMU (memory management unit) and is thus not
capable of the following neat tricks that modern OS'es make great use of:

[Definitions: page: the minimal unit of address space, normally 4096
or 8192 bytes. VMA: Virtual Memory Address. IPC: Inter Process
Communication.]

1) Remapping of pages from physical memory into a virtual memory, per
   process. Removes fragmentation issues of physical memory, since 
   it is possible to remap scattered pages into consecutive virtual
   memory segments for a process. It also removes the need to relocate
   linked programs, since they can be remapped to always reside at the
   VMA given to it at link-time.

2) Protecting pages against access. This makes it possible to swap out
   pages to secondary storage (like a harddisk) and recall them as they
   are referenced. It also enables a similar technique called "demand
   loading", where you map the pages of a file on the harddisk to a 
   process virtual memory. As the process accesses the pages, they are
   brought in to RAM as needed. This is used in combination with 1) to
   demand-load the actual code of programs. When a program is executed
   the file containing the program is mapped into the VMA assigned at
   link-time, and paged in as the thread of execution touches pages.
   Should the system run low on memory, pages containing file mappings
   can simply be flushed and re-used; they will be reloaded when their
   code/data is needed again.

3) Protecting pages against writes. This enables a very important
   optimization known as COW (Copy-on-write). It means that if a range
   of pages need to be copied (like when a process forks a child that
   should have its own independent copy of the data segment) it is not
   actually necessary to make a physical copy of the data in RAM; it is
   enough to duplicate the page table and mark the pages write-protected
   for both the parent and the child. That way they will continue 
   sharing the memory until one of them actually alters the data, at
   which point a physical copy of the touched page is made and each
   process gets its own copy of it.

   A variation of this technique is done when memory is allocated to a 
   process. Physical pages start out mapped to the systems zero-page
   (a page filled with zeros in physical RAM), and when they are 
   written to, real pages are allocated and replaces the mapping.


Most of the above mentioned tricks are done to optimize the use of 
physical RAM and to increase the overall system speed. Abandoning these
in eLinux produces a comparatively slower system but is not a show-
stopper. However, the lack of the basic feature of memory remapping,
incurs a subtle but fatal blow to the operation of one of the cornerstones
of UNIX - the fork() system call.

vfork() and fork() in eLinux
----------------------------

In UNIX, processes can only be created by splitting an existing process
in two. This is done with the fork() system call, which takes no arguments
and the only thing that differs the resulting processes, is the return-
value of fork(). The semantics of fork() say that the processes from the
point of the fork() are completely stand-alone. Nothing one process does
can affect the other one (without intentional IPC mechanisms of course).

This means that global data, as well as data allocated from the heap at
runtime (with malloc() for example) needs to be duplicated so that each
process gets its own copy. This is where we run into trouble. The actual
copying of the data is not a problem for a NO_MM UNIX (even if its done 
much more efficiently by the COW algorithm described under 3 above).
But since the data area now appears at two different physical addresses
(and we cannot do remapping of memory), somehow one of the processes
needs to change the way it accesses its memory, so that it finds its
own data.

This is astonishingly problematic. Normally a program is linked at
compile-time to access its data at a specific VMA, and we can still load
such a program without MM by relocating the linker symbols at load-time
to the new physical locations. But when the process splits its data
segment, the code cannot be relinked to access the new data addresses 
since the old (parent) process still needs to access the old data. 
Furthermore, we cannot even load a new copy of the code of the program
and relocate its data references to point to the new data area, because
the data _itself_ may contain explicit pointers into other parts of the
data. An example will illustrate this:

void main()
{
	static int myDataVar; // this resides in the data segment
	int *myDataPtr;       // this resides on the stackframe

	myDataPtr = &myDataVar;  // now we get a pointer to somewhere
				 // inside the data segment

	// now we fork and do some stuff in each process to illustrate
        // where bad things happen

	if(fork()) {
		   // in parent
		   *myDataPtr = 42;  // writes to the data segment
		   sleep(10);
		   printf("myDataVar is %d", *myDataPtr);
		   exit();
         } else {
	 	   // in child
		   myDataVar = 99;   // writes to our new copy, provided
			             // we have relinked the code
		   *myDataPtr = 24;  // possibly writes to the parents data!
		   exit();
         }
}

Here, clearly even if we make sure we copy the entire data segment and
make the code access the correct data segment globally (by reloading and
relinking the code for the child) we end up with old pointers to the
parents data area.

This can be solved by using a segment register - a base register from
which all data accesses are made relative to. It can be either implicit
inside the CPU (like the 8086 segment registers) or explicit by the
compiler generating relative accesses for all memory accesses.

It was decided that it was too costly to let the compiler generate
relative addressing for all data accesses. No quantitative study was
done though. It might be that it is afterall a viable solution (other
ports of Linux to MMU-less CPU's seem to have used it).

Since the problem could not be solved, we cannot support the real
semantics of fork(). Instead, fork() is deprecated and instead the
related function vfork() was implemented. The behaviour of vfork() is
that it forks the process, and lets the child share the parents 
resources until the child execs a new program or exits. The parent is
put to sleep until that happens. It is also undefined what happens if
the child tries to use any global variable, since it might be (and is,
in eLinux) shared with the parent.

Thus, you should use vfork() when you want to spawn a new program - by a
construction similar to this:

---

int pid;

pid = vfork();

if(pid == 0) {
       // in child
       const char *argv[] = { "/bin/myprog", "/bin/myprog", NULL };
       execve(argv[0], argv);
       // oops, exec failed! we need to exit instead.
       _exit(0);
}

// in parent. as soon as the child has execve'ed, the parent will resume
// execution here, and the child becomes a new process running the desired
// program. for example we could have run a shell-script instead of
// /bin/myprog.

// if you want to wait for the child to exit, do:

waitpid(pid, NULL, 0);  // wait for the child to finish

---

Please notice that in UNIX, _someone_ needs to wait for a process that exits.
The exit will release the allocated resources, but the process will stay in 
the process list as a Zombie until the parent or someone else waits for it.
If you don't want to wait for it synchronously like in the example above,
you can catch the SIGCHLD signal and let a signal handler wait for a process
when you get the signal (it is sent to the parent, when a child dies).

Now, there are obviously situations where we would like to use the real
fork() without the mandatory parent-sleep-until-child-exits. An example is
the sftpd FTP server. When an incoming connection is detected, it forks and
calls a subroutine that handles the new session, while the main-loop continues
and accepts new connections. Therefore, fork() is not disabled entirely in
eLinux; it works just like vfork() but without the sleeping of the parent. In
a way it is similar to creating a new thread, sharing the memory between the
parent and the child. However, that behaviour is NOT guaranteed in any way
and WILL lead to buggy software if you rely on it working. To be sure that a
fork() in eLinux works, you NEED to read and understand the following points:

* The normal behaviour of fork() is that the child gets its totally own
  memory-space, including copies of the parents global variables and data
  allocated from the heap using malloc(). Many programs _rely_ on this
  mechanism to work. A server program might have a dozen global variables
  and when a connection is accepted, it fork()'s and the children changes
  these into reflecting their own connection. But in eLinux, the children
  will NOT get their own memory spaces, so each child will overwrite their
  brethren's variables as well as their parents. If you port a program that
  uses this mechanism, you need to MAKE SURE that it does not access any
  global variables in children (or static variables inside functions).

* The child gets its own copy of the parent stack - thus, you CAN reference
  variables from the parents stack. However, be very careful if any of the
  variables is a data array, since the compiler might have cached an address
  into the array in a register, and that address is obviously not updated with
  the childs stack address after the fork(). 

  This is especially noticeable in this situation:

  void
  myfunction() 
  {
    char buf[16];
    buf = strcpy("/dev/ttyS0");
    if(fork() == 0) {
	    // in child
	    int fd = open(buf, O_RDONLY);
	    [do stuff with fd]
	    exit(0);
    }    
  }

  Here, since the parent is not slept by the fork(), it will exit the
  stack-frame containing the char buffer before the child even begins 
  executing. When it does, it will have cached the 'buf' pointer in a 
  register and tries to reference through that into the by now released
  stack-frame, and opens the wrong file => VERY UNDEFINED BEHAVIOUR!

  There is really only one case where you can fairly safely use fork()
  in eLinux and that is if you do like this (taken from a real program):

    	    if((fd = accept(master_fd, (struct sockaddr *)&sa, &salen)) < 0) {
		continue;
	    } else {
		/* we got a new connection, fork to handle it */
		int pid;
		pid = fork();
		if(pid < 0) {
		    perror("fork");
		} else if(pid == 0) {
		    /* in child */
		    handle_ftp_request(fd);
		    /* exit, this will cause a SIGCHLD to the parent and
		     * we return the fd as status code so it knows which.
		     */
		    exit(fd);
		}
		/* in parent - stay cool */
	    }
	
  Here, the child only relies on the input parameter 'fd' which is a simple
  integer. All other variables are on the stack inside handle_ftp_request
  and deeper, and are thus copies local to the particular child.

  Be aware that the popen() c-library function uses fork() and is therefore
  also subject to the above problems, although it could be rewritten to use
  vfork(). It is not like that in our c-library yet.

mmap in eLinux
--------------

Another memory-management related UNIX feature is the mmap() call, which 
maps either empty memory or a file into the virtual address space of the
calling process, at either a selected address or at an address picked by the
OS. This has numerous uses like reading from a file without having to first
read the data into a local buffer, removing a copy. The OS will load the
pages of the file into memory as they are accessed. It can also be used to
extend allocated data areas in a contiguous fashion, like if the stack needs
to be extended, you can mmap() new memory into the addresses just below the
stack.

We can obviously not do any of these tricks in full without an MMU. In
particular, we can't let the process pick a virtual address since all our
addresses are physical and might already be occupied by another process or
the OS itself. We cannot either demand-load an mmap'ed file, but have to
allocate pages and read the entire file into memory. But at least it works,
and using it makes sure your program works efficiently under a full Linux.




2. Consequences of running without swap, with fragmentation issues etc
----------------------------------------------------------------------

T.B.D., just some quick info so I can remember it first.

Both the buffercache and pagecache will use memory. In a normal Linux 
system it is normal for the caches to use up almost all memory which is not
in use by applications. Why pay for more memory if it's not used ? Memory
is released by the caches when it is needed for other things.

However, in eLinux without MMU, we rely on being able to allocate 
consecutive regions of memory, and this can be a problem since even if
we free up enough memory, it might have become severly fragmented. There
is no solution to this except restricting either the size or location of
the caches (for example, the caches can grow from one side of the memory
while normal allocations grow from the other side - this requires some
heavy modifications of the page_alloc buddy system though).

The mechanism that releases memory from the caches when necessary can be
confusing. It works like this:

The kswapd process (which lives in mm/vmscan.c) is a kernel thread in an
eternal loop. It is awoken by a timer function, which triggers when the
number of free pages (nr_free_pages) decreases past some limits. kswapd
then calls try_to_free_page until a desired free page count is reached (there
are a lot of parameters - see mm/swap.c for free_pages_low/high for example).

try_to_free_page has a couple of things it can try, but the important one
without an MMU is shrink_mmap (mm/filemap.c) which goes through all pages
in the system, checks if its used by the page or buffer cache or something
else, and frees it if possible. If the page was in the page cache it
can be freed directly from shrink_mmap; if it had buffers, try_to_free_buffer
is called.

try_to_free_buffer (fs/buffer.c) simply checks if all buffers on the page
are clean and unlocked - if so, the page is freed.

Now, for this to work, buffers obviously have to be clean. They are clean
when they were allocated for a read from a device, but as soon as they are
modified they become dirty and are not immediately written back to the 
device they belong to. Instead they wait until the bdflush kernel-thread
goes through the buffer cache and actually writes the buffers to the device
(or until an explicit sync() or unmount() is made on the device).



