In the process of making my own Operating System.

There are many different operating systems; which one will you choose?

In the process of making my own Operating System.

Post by centip3de on Sat Jun 18, 2011 1:53 pm
([msg=58700]see In the process of making my own Operating System.[/msg])

Why hello there Mr. HTS, I was wondering if you had any ideas for me.

I'm currently in the process of making my own personal Operating System, and as any OSDev would want, I want it to be different from the others. Currently my only ideas are portability, and modes, both of which I'll explain in detail below. Anywho, I have a simple SIMPLE kernel done, the bootloader done (though I think I might just use GRUB), and the linker done. So far all it says is "Well hey there" then it hangs there for the remainder, until you shut it off. Anywho, I was wondering if you fine people had any ideas of what I should add, besides what I'm planning to add. Also, if you could tell me if I'm completely crazy in my ideas, that would be wonderful.
Thanks
~C

Modes- Different modes that would be tasked with different things. It could be that when you're in a certain mode, you can only do certain things. For instance, when I'm in "USER MODE" I shouldn't be able to issue a "server start" command, nor should I be able to issue a "ping" command. Each of these following commands should only be able to be started in that certain mode. The current modes I have in mind are:

User - For your general file manipulations, applications, and maybe a few more things
Root - For editing system files
Networking - For things like "ping", "tracert" , and the like
Server - Can be set to "nm" which is "No monitor" but essentially turns the computer into a server, which can be broken after a simple key combo (cntrl + c ?")
Developer - Access to developing tools, such as making anything from "C" programs, to Python scripts, to ASM programs

Portability - When you're in Root mode, and had a flash drive greater than 16 gigs inserted into your machine, you would be allowed to issue the command "drive [drive letter] copy -os -a". This would copy your ENTIRE OS onto that flash drive. It would also change a few config files/boot loader files/kernel files so that it could be run from that flash drive. That way you could take your OS (or at least a copy of it) wherever you went. I was also thinking that if you had your normal computer running in Server mode, you could sync up all the changed files to the server, so that all your files would be updated on your main computer.

This would be my summer project, and am hoping that I would get to the point where I can switch modes by the end of said summer. I know that I will not complete this entire thing for at LEAST 2 years, but hey, I got nothing better to do. ;)

So, tell me what you think! If you'd like to be involved in this, just PM me, if not and you want to flame me for under-taking such a daunting task, flame away, and if you'd like to be nice and tell me some ideas you think would be cool, well... That'd be cool. I'll provide info on this thing below.

Coding in- FASM/Free Basic
For- 32 bit CPU's
Getting my info- OSDev.org
If you want the code, just PM me, it's nothing special or new... Fow now :D
Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning. -Rick Cook
User avatar
centip3de
Moderator
Moderator
 
Posts: 1431
Joined: Fri Aug 20, 2010 5:46 pm
Blog: View Blog (0)


Re: In the process of making my own Operating System.

Post by conscience on Sat Jun 18, 2011 4:47 pm
([msg=58703]see Re: In the process of making my own Operating System.[/msg])

First, your modes idea seems to be a bit awkward. The goal of any operating system is to make the computer usable, manage system resources and provide some abstraction layers between the hardware and user applications. Restricting the usage in some aspects based on such modes will force the user to switch modes constantly which is much more an annoyance than a useful feature, and additionally you'd never be able to implement it without mixing up some of these "operation mode" codebases -> going into this all makes no sense. Within your system you'll have to tend the other way: generalize system resources as much as possible. Any restrictions made to the system functions should be based on the current user, maybe a bit on workflow, but not on predestinated modes.

A root and even a simple user will eventually nned things like ping for example. Another example for your server mode is that server software is userspace (CPL3) so that any computer with a networking interface will operate as server if needed no matter whether you run other threads simultaneously or not (i.e. there are other aspects of functioning).
A (full-blown) developer will definitely need everything the machine is capable of as well as a user or a root (or anybody) could eventually need some dev tools (which should also all be userspace software as otherwise nothing could stop a buggy application crushing your kernel or even your hardware)

Also note that too much effort to make something (noobishly - sorry dude, no offense, I just didn't find a better phrase; my bad) standalone can easily drown to failure as the increasing degree of incompatibility with other systems and software inversely lowers usefulness and sometimes usability. For example, rolling your own filesystem instead of implementing a commonly used one will introduce you trouble as no system will support your one as well as yours won't support theirs rendering data exchange between your OS and common systems impossible (at least by means of a physical storage device). Adopting known-good-and-effective implementations will save you a lot of time and energy as well.

I guess you're pretty new to OSDev :) Whatever decision you take, good luck!

I'd recommend for you to stick to the kernel design itself (leave thing like tools, text editors, etc. shipped with your system for now as they are simply applications which need an already working OS themselves). You'll first need (assuming x86)
  • A bootloader (a decision on native FS could take place) - I'd recommend experimenting on FAT12 floppies first
  • A physical memory manager and memory handling design
  • A virtual memory manager if you decide to use VM (make a stance of whether you use paging or not)
  • A device driver interface design
  • User API - System calls design
  • A very stable and carefully written exception handling (You'll love your first 237 triple fault :twisted: ) -- Stack Fault, Page Fault, Double Fault are critical. If you fail to handle a double fault or your system stack is corrupted, it'll triple fault, and again, and again, and again ... (the horribly slow murderer with the extremely inefficient weapon :mrgreen: )
  • A process manager (fire up, suspend, terminate, etc.)
  • A scheduler if you decide to implement multitasking (what type of MT would you use and with what design?)
  • And more what I don't wanna go into for now, but if you need any help, I'm always open for PMs and stuff

Don't forget that OS development is pure fun. Enjoy doing it! Love it! I do so.
Again, good luck!
Let him who has understanding recount the number of the beast, for it is a human number: His number is 0x029A.
conscience
Poster
Poster
 
Posts: 250
Joined: Thu Jan 08, 2009 9:05 pm
Location: 127.0.0.1
Blog: View Blog (0)


Re: In the process of making my own Operating System.

Post by centip3de on Sat Jun 18, 2011 6:28 pm
([msg=58707]see Re: In the process of making my own Operating System.[/msg])

conscience wrote:First, your modes idea seems to be a bit awkward. The goal of any operating system is to make the computer usable, manage system resources and provide some abstraction layers between the hardware and user applications. Restricting the usage in some aspects based on such modes will force the user to switch modes constantly which is much more an annoyance than a useful feature, and additionally you'd never be able to implement it without mixing up some of these "operation mode" codebases -> going into this all makes no sense. Within your system you'll have to tend the other way: generalize system resources as much as possible. Any restrictions made to the system functions should be based on the current user, maybe a bit on workflow, but not on predestinated modes.

A root and even a simple user will eventually nned things like ping for example. Another example for your server mode is that server software is userspace (CPL3) so that any computer with a networking interface will operate as server if needed no matter whether you run other threads simultaneously or not (i.e. there are other aspects of functioning).
A (full-blown) developer will definitely need everything the machine is capable of as well as a user or a root (or anybody) could eventually need some dev tools (which should also all be userspace software as otherwise nothing could stop a buggy application crushing your kernel or even your hardware)

Also note that too much effort to make something (noobishly - sorry dude, no offense, I just didn't find a better phrase; my bad) standalone can easily drown to failure as the increasing degree of incompatibility with other systems and software inversely lowers usefulness and sometimes usability. For example, rolling your own filesystem instead of implementing a commonly used one will introduce you trouble as no system will support your one as well as yours won't support theirs rendering data exchange between your OS and common systems impossible (at least by means of a physical storage device). Adopting known-good-and-effective implementations will save you a lot of time and energy as well.

I guess you're pretty new to OSDev :) Whatever decision you take, good luck!

I'd recommend for you to stick to the kernel design itself (leave thing like tools, text editors, etc. shipped with your system for now as they are simply applications which need an already working OS themselves). You'll first need (assuming x86)
  • A bootloader (a decision on native FS could take place) - I'd recommend experimenting on FAT12 floppies first
  • A physical memory manager and memory handling design
  • A virtual memory manager if you decide to use VM (make a stance of whether you use paging or not)
  • A device driver interface design
  • User API - System calls design
  • A very stable and carefully written exception handling (You'll love your first 237 triple fault :twisted: ) -- Stack Fault, Page Fault, Double Fault are critical. If you fail to handle a double fault or your system stack is corrupted, it'll triple fault, and again, and again, and again ... (the horribly slow murderer with the extremely inefficient weapon :mrgreen: )
  • A process manager (fire up, suspend, terminate, etc.)
  • A scheduler if you decide to implement multitasking (what type of MT would you use and with what design?)
  • And more what I don't wanna go into for now, but if you need any help, I'm always open for PMs and stuff

Don't forget that OS development is pure fun. Enjoy doing it! Love it! I do so.
Again, good luck!


Hahahaha no need to feel bad for calling me a noob, it's what I am :) . I'd have to agree with you on the modes, that they seem inefficient and as it would be too much of a hassle to code. Though I do want to implement a FAT16 or FAT32 file system, as it seems to be the most supported. I also want to add my "server mode" after I add implementation for a programming language. Also, it's meant to be for those only on that operating system, as it seems that it would make it a bit more secure, than if it was open for all operating system. Though I had a couple questions:

1. I was wondering what order I should go in? A basic kernel, bootloader, and linker are all done and ready, but I don't really know where to go from here, help?

2. Do you know any good books on Operating System Development, or Operating System Theory? I was checking out This one but I would like to have more, help?

Thanks!
~C
Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning. -Rick Cook
User avatar
centip3de
Moderator
Moderator
 
Posts: 1431
Joined: Fri Aug 20, 2010 5:46 pm
Blog: View Blog (0)


Re: In the process of making my own Operating System.

Post by neuromanta on Sat Jun 18, 2011 7:11 pm
([msg=58710]see Re: In the process of making my own Operating System.[/msg])

I think the best book on operating systems is from Prof Tannenbaum. You can study the development and source code of Minix, that could show you the way, although I never tried to implement an OS myself, so I'm not so knowledgable in the field. One question though: what are the features your kernel currently?
User avatar
neuromanta
Poster
Poster
 
Posts: 302
Joined: Mon Nov 30, 2009 9:29 am
Location: Hungary
Blog: View Blog (0)


Re: In the process of making my own Operating System.

Post by conscience on Sat Jun 18, 2011 7:46 pm
([msg=58711]see Re: In the process of making my own Operating System.[/msg])

centip3de wrote:Though I had a couple questions:

1. I was wondering what order I should go in? A basic kernel, bootloader, and linker are all done and ready, but I don't really know where to go from here, help?

2. Do you know any good books on Operating System Development, or Operating System Theory? I was checking out This one but I would like to have more, help?

Thanks!
~C


1: The term "basic kernel" can be interpreted many ways. On common is

Code: Select all
// we got some minimal RTL
print("Kernel is loading...\n"); // print calls putch()
for(;;){}


or alike. :) Which ones of the components listed in my previous post do you have already in your kernel? I think you'd like to kick off an exception handling and a memory manager first. On the top of the physical memory manager you can mount your VMM which you can implement kmalloc() and kfree() for (I like to extend K&R's malloc() to an open-end, dual linked list as it makes the implementation of kfree() more comfortable -- I mean merging free blocks).
You would also like to have an executable interface (ELF/PE/Other? -- don't overload yourself, pick only one for now) so you can load drivers from disk (count flat binaries out as you cannot relocate them) a bit later.
I don't know if you've enabled the A20 address line and reprogrammed the timer and the interrupt controller already. If not, go ahead an do it as the default vector for IRQ0 (timer) line of the PIT will fire a double fault immediately after CLI or even triple fault if you have not set up exception handlers yet. Once again, talking about exception handlers, they should be set up first as there is nothing else you can use for diagnosing errors. Test on a VMware so you can debug your code if something goes wrong - not just experiencing another pathetic reboot.
BTW a bit more info on your kernel would help to see where you are now and where you can go further.


2: I don't really know any books, although there are soem out there considered to be a very good resource. I've just collected the info I needed a bit of time ago from forums, wiki's, papers, etc. Not too much of info, but http://www.osdever.net and http://www.osdev.org may help a tiny bit and they also have references of other sites and books related to this. Also, Tannenbaum's papers are claimed to be very nice, however my personal preference is monolithic kernel, which he refused to appreciate. (It does not mean that I hate him and wanna take a crap on his feet, simply I'm not interested too much in an in-depth discussion of microkernels at the moment)

-- Sun Jun 19, 2011 2:52 am --

neuromanta wrote:...although I never tried to implement an OS myself, so I'm not so knowledgable in the field.


Really? Man, you don't know what you miss. ;)

-- Sun Jun 19, 2011 2:59 am --

aaaayyyy I almost forgot! Get a copy of the Intel 80386 Programmers Reference, you'll need it I think.
Let him who has understanding recount the number of the beast, for it is a human number: His number is 0x029A.
conscience
Poster
Poster
 
Posts: 250
Joined: Thu Jan 08, 2009 9:05 pm
Location: 127.0.0.1
Blog: View Blog (0)


Re: In the process of making my own Operating System.

Post by centip3de on Sat Jun 18, 2011 10:14 pm
([msg=58714]see Re: In the process of making my own Operating System.[/msg])

conscience wrote:Which ones of the components listed in my previous post do you have already in your kernel? I think you'd like to kick off an exception handling and a memory manager first. On the top of the physical memory manager you can mount your VMM which you can implement kmalloc() and kfree() for (I like to extend K&R's malloc() to an open-end, dual linked list as it makes the implementation of kfree() more comfortable -- I mean merging free blocks).
You would also like to have an executable interface (ELF/PE/Other? -- don't overload yourself, pick only one for now) so you can load drivers from disk (count flat binaries out as you cannot relocate them) a bit later.
I don't know if you've enabled the A20 address line and reprogrammed the timer and the interrupt controller already. If not, go ahead an do it as the default vector for IRQ0 (timer) line of the PIT will fire a double fault immediately after CLI or even triple fault if you have not set up exception handlers yet. Once again, talking about exception handlers, they should be set up first as there is nothing else you can use for diagnosing errors. Test on a VMware so you can debug your code if something goes wrong - not just experiencing another pathetic reboot.
BTW a bit more info on your kernel would help to see where you are now and where you can go further.


You know you need to study when you Googled essentially every word that you just said ( I currently have 56 tabs open, just to get a general gist of what you said). :shock:

Anyways, I'll post the kernel code, it's pretty much what you would find on the wiki under FreeBASIC, as I've had little knowledge of how to even start developing a kernel. I'll also post the linker code, right now though, my bootloader is GRUB, seeing as I also have no idea where to start. I also just finished with the GCC cross-compiling binutils.

Kernel
Code: Select all
DECLARE SUB PrintString(src AS Byte Ptr, x AS LONG, y AS LONG)
DECLARE SUB main ()

SUB multiboot ()
   const XALIGN = 1 shl 0             
   const MEMINFO = 1 shl 1             
   const FLAGS = XALIGN or MEMINFO
   const MAGIC = &h1BADB002       
   const CHECKSUM = -(MAGIC + FLAGS)
   const STACKSIZE = &h4000
   Asm
     .align 32
     .LONG MAGIC
     .LONG FLAGS
     .LONG CHECKSUM
         
     .comm stack, STACKSIZE     

     .global loader                 

     loader:
       lea   esp, [stack + STACKSIZE]
       push  eax                       
       push  ebx                       

       CALL  MAIN

       cli
       hlt                   
   END Asm

END SUB

SUB main ()
    CONST s = "Well hi there"

    PrintString CPtr(Byte Ptr, @s), 35, 12

END SUB

SUB PrintString(src AS Byte Ptr, x AS LONG, y AS LONG)

    DIM dst AS Byte Ptr
    DIM counter AS LONG

    dst = CPtr(Byte Ptr, &HB8000 + y * 160 + x * 2)

    counter = 0

    WHILE src[counter] <> 0
        dst[2 * counter] = src[counter]
        dst[2 * counter + 1] = 15
        counter = counter + 1
    WEND
END SUB


Linker
Code: Select all
OUTPUT_FORMAT("elf32-i386")
ENTRY (loader)

SECTIONS{
    . = 0x00100000;

    .text :{
        KERNEL_START = .;

        *(.text)
    }

    .rodata ALIGN (0x1000) : {
        *(.rodata)
        _CTORS = .;
        *(.ctors)
        _CTORS_END = .;       
    }

    .data ALIGN (0x1000) : {
        *(.data)
    }

    .bss : {
        SBSS = .;
        *(COMMON)
        *(.bss)
        EBSS = .;

        KERNEL_END = .;
    }
}


Now, prepare yourself for the bombardment of nooby questions.

1. I was messing around with FreeBASIC (I coded in VB.Net for ~3 years so I can read/write it all just fine) but it occurred to me that the majority of it relies heavily on WindowsAPI's, is there anyway for me to port them to my OS, or do I just have to avoid them? In the case of my avoiding them, how can I tell which ones rely on WindowsAPI's and which don't?

2. Where do I even start? I mean, I know what a Kernel is, but I just don't know where to start? I'm the kind of person who gets absorbed in tasks, but if I don't know where to start the task, then I most likely won't get absorbed in it.

3. Any good places besides osdev/osdever/google to help?

4. What do you suggest for a starting OSDev?

Thanks
~C
Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning. -Rick Cook
User avatar
centip3de
Moderator
Moderator
 
Posts: 1431
Joined: Fri Aug 20, 2010 5:46 pm
Blog: View Blog (0)


Re: In the process of making my own Operating System.

Post by conscience on Sun Jun 19, 2011 1:09 am
([msg=58723]see Re: In the process of making my own Operating System.[/msg])

centip3de wrote:You know you need to study when you Googled essentially every word that you just said ( I currently have 56 tabs open, just to get a general gist of what you said). :shock:


Dude! I'm sorry. So you have somewhat like a "Hello World kernel" right? So if I understand well, GRUB (I never used this for several reasons) loads your 32bit protected mode kernel and it is standard protected mode (without paging)
Your very first task will be to carry out your own GDT (research: protected mode tutorial -- there are tons of them out there) as the one created by GRUB is a) likely to be trashed, b) may not serve your purposes. It is common to have a hardcoded GDT (if you plan to use paging) as you'll probably never need to adjust its size. AFAIK you are loaded at 1MB mark with flat segments (starting from address 00000000h), and your new descriptors in the GDT will have to match this setting otherwise the CPU will triple fault, thus reset.
Once you have set up the GDT and its pointer (for the GDTR) you'll have to issue an LGDT machine instruction in order to load the special FWORD pointer into the GDTR and do a FAR JMP to flush the instruction pipelines and to reload CS from the GDT.
Usually:
Code: Select all
db 0EAh  ; FAR JMP instruction opcode
dw (offset @jump_right_after_this)
dw 08h    ; Commonly, the descriptor for CS is #1 which is 08h (8 bytes increment, so #2 is 10h which is usually for DS/ES/ETC)

Once you made the jump, load other segment registers with the data segment descriptor. From now on, you carry your own weight (i.e. your kernel uses only resources of itself). Don't forget to preserve important data GRUB gave you before using registers!!!

centip3de wrote:1. I was messing around with FreeBASIC (I coded in VB.Net for ~3 years so I can read/write it all just fine) but it occurred to me that the majority of it relies heavily on WindowsAPI's, is there anyway for me to port them to my OS, or do I just have to avoid them? In the case of my avoiding them, how can I tell which ones rely on WindowsAPI's and which don't?

2. Where do I even start? I mean, I know what a Kernel is, but I just don't know where to start? I'm the kind of person who gets absorbed in tasks, but if I don't know where to start the task, then I most likely won't get absorbed in it.

3. Any good places besides osdev/osdever/google to help?

4. What do you suggest for a starting OSDev?

Thanks
~C


1. Well, I'm not too familiar with Basic. It is about 10 years now since I've used some flavor of Basic and to be honest I never knew it well. In the Useful Stuff section of HTS (I guess) you'll find OllyDbg by which you can track DLL usage and API calls. The Windows API is a very heavy bloat, so although it's possible to port it, such project can never be expected to succeed for sure. There is a "freeWin" project, called Wine, which is now being developed for many years and it still has problems with running certain applications.
As one who writes her/his kernel in C has to carry out her/his own implementation of the C runtime library (at least the functions actually used) so do you (if your question was regarding the compiled apps' API dependency). If your question aimed to find out if you can port the IDE itself, then I have to answer "only God knows". No clue which APIs it might be using besides file handling calls (at least ATM).

2. Your first task is to present an own GDT as described above. After that I'd implement some basic exception handlers (e.g. print exception number and halt) as they are key elements of debugging your kernel at runtime as well as these stuff will be used later in system security aspects (not to be confused with the user account security and privileges as they are completely different and are way less privileged compared to the kernel: the kernel owns the whole machine, while a root user can only do things the kernel gives opportunity via the user interface -- again, check out protected mode's privilege levels).

3. No idea. Actually I've Googled all my way down the route. Sniffed around, pulled a few hundred megs of documents from the web for offline availability, mostly tutorials from sites like bona fide and the osdever wiki, and specifications, tech details about hardware, filesystems, file formats, etc.

4. Actually it is up to you what makes you satisfied. These projects can range from (I'll exclude real mode puffs) simple protected mode kernel with character shell (you'll need to write a parser then!) or 80x25 textmode windows, only one filesystem support and no device driver interface (just a few embedded drivers for the most important hardware like keyboard, video and disks), multi/monotasking app management to the full-blown, GUI-controlled, multi-user, multi-processing, multitasking/multihreading, Plug n Play capable, gigaprojects.

If you are happy with a character-based interface (lets say a windowed one), preemptive multitasking, single-CPU-core system supporting only one filesystem, etc. I'd recommend this one. You can hardcode you display driver into the kernel (put 80x25 char CRT management routines in) or even the user interface (you "draw" windows for each app running, all appz are somewhat fullscreen, use a keycombo to toggle windows, other keystrokes for basic functions like "close window and terminate task" or "shutdown" or "show main window" or "show settings panel", etc. I guess you know what I'm saying) and base device drivers. Protected mode for securing system against user apps, you use memory area between 1MB and 15MB which is (theoretically) guraranteed to be free, no paging. This is quite simple and later on you can extend as you like.

You'll need: GDT, an LDT per process (generally same as the GDT, but Local not Global), exception handlers and an IDT (again, check protected mode tutorials and the Intel 80386 Programmers' Reference), disk drivers (refer to Floppy and ATA programming manuals), a task management (you just assign metadata structures and some space where to save CPU state to running tasks), a scheduler (you save and restore CPU states to/from memory based on the timer tick IRQ0) and some file handling code, that just about wraps it up.

1. Have a GDT
2. Have an IDT and ISRs (Interrupt Service Routines) for exception/interrupt handling
3. Reprogram the PIC (to avoid collision with exceptions; put hardware interrupts above INT 1Fh) After this you can STI
4. Reprogram the PIT (set it up to generate 100Hz times ticks; this value is common)
5. Have a simple memory manager (your version of malloc() and free(); simply take care of contiguous 14MB)
6. Have task manager and scheduler
7. Have disk-, floppy-, display- and keyboard handling routines
8. Have filesystem handling routines on top of your freshly created "disk drivers"
9. Have a visual interface and a "keystroke/command parser" (read keystrokes from a queue put there by the keyboard ISR, translate it to ASCII via the keyboard driver i.e. translation routines, determine which task it belongs to and put the value on the task's queue or use it if the keystroke belongs to the user interface)
10. Pick a free interrupt line for system calls (Don't forget the IDT descriptor for it!). Determine the request type by EAX value passed by the app, the carry out the necessary action (e.g. read a file from disk into memory and return a reference of it in EAX)
11. So far you can load files, execute apps, display data driven by user input. Write somewhat-useful applications

That is the simplest partially-usable thing I can think of at the moment. Larger stuff are basically the same, but you get a map of the whole memory, parse it, set up memory manager according to this map, use paged virtual memory and have drivers dynamically linked to the kernel at runtime instead of being hardcoded into and the conform to a specification which lets these modules attach to the kernel. Furthermore, the user interface is a separate task (CPL3 afaik), a process does not necessarily have a task, each process can have multiple threads running, more supported filesystem, and alike...

Next question? ;)
Let him who has understanding recount the number of the beast, for it is a human number: His number is 0x029A.
conscience
Poster
Poster
 
Posts: 250
Joined: Thu Jan 08, 2009 9:05 pm
Location: 127.0.0.1
Blog: View Blog (0)



Return to Operating Systems

Who is online

Users browsing this forum: No registered users and 0 guests