Beginning Kernel Hacking

cheesysam

I'm going to be doing some kernel hacking as part of a final year compsci project.

What is the best way of starting this, for instance is there some sort of sandbox mode for running the kernel. My current attempts have been with Arch and slackware linux running in a VM but has proven to be a nightmare.

Where do I start with this?

Set_Killer

maybe try to recompile the kernel by hand. the best way to do that is to use Gentoo, but they are prepathcing the kernel before put it in the repository. i think you can do this with any other distro.

after that read the code of some modules and try to find bugss or howto abuse them. but be sure that the kernel is not written from newbie programmers.

am not so advanced. it would be interesting to hear the opiniion of the gurus.

ScHmItCh

Gentoo is a good place to start if you want to learn about the kernel. You can also do recompiles in other systems. I would recommend doing a few customizations before you go hacking away at the source code.

woboyle

First, visit www.kernel.org and www.tldp.org for useful how-tos and other kernel documentation.

Next, don't give up on running your experimental OS in a virtual machine. I have done a fair amount of kernel development for both Linux and Solaris with VirtualBox VM's. The important thing here is that the host needs to be a stable system - I use CentOS/RHEL 5 as my host (8 cores and 8gb ram) and run newer Linux clients (Gentoo, Ubuntu, et al) as well as Windows and Solaris x86 as guests.

timhoppen

Actually, in Gentoo, you can get a kernel.org kernel without any patches.
There are a variety of kernel packages available. Here are 3:

To use the kernel patched by Gentoo:

emerge gentoo-sources

To use the latest released kernel from kernel.org

emerge vanilla-sources

Or, to use the daily git snapshot from git.kernel.org

emerge git-sources

http://www.gentoo.org/doc/en/gentoo-kernel.xml

Or, as with any distro, you can just download the tarball from kernel.org and have fun. Gentoo (potentially) makes it easier because it is required to compile the kernel yourself anyway.

In addition, make sure that you know what areas of the kernel you want to hack at. Honing your focus will help a ton. The kernel is a big piece of code!

ford

I would take a look at GoboLinux. GoboLinux uses a kernel patch to hide the traditional filehierarchy's structure. Something that is rather simple like this would be a great start.

You may also want to look at some projects online such as Linux ABI and Longene.

amnesia

easiest thing to do would be get a linux distro with the config file exported to /proc as 'config.gz'

(by default) some distributions already have this enabled in the kernel before they release (slackware)

next step would be
cd /usr/src
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.x.xx.x.tar.bz2
tar xvjf linux-2.x.xx.x.tar.bz2
cd linux-2.x.xx.x.tar.bz2
zcat /proc/config.gz > $PWD/.config
make oldconfig;make menuconfig
>make oldconfig may or may not ask you if you'd like enable certain >new features
make menuconfig and edit in our out what you need and dont need
exit and save

cd src modules or whatever and look around.
change whatever you feel you can and are able to change really

cd /usr/src/linux-whatever

make modules; make modules_install

cp System.map /boot;cd arch/i386/boot/;cp bZimage /boot/kernel-whatever

add lines (if you use it) to lilo
image = /boot/kernel-whatever
root = /dev/hda1
label = whatever
read-only

lilo

reboot.

(not saying you don't know how to recompile a kernel, as kernel hacking goes -- you've got alot of lines to go through, have fun :P)

if anythings wrong, you'll know with a kernel panic :>

cheesysam

Hi everyone, I'm overwhelmed by the great responses, thanks!

Rubberman - Thanks for your motivation to keep me with VirtualBox. I'm using OS X 10.5 as a host at the moment which is as stable OS as I know. Presumably if that becomes a problem I can just backup the machine image and use it in Virtualbox on a different host?
Thanks for the links also. I'll get reading.

N.D, Josh, Tim - Gentoo seems like a common suggestion so I'll get that and get it installed.

Hi Bradford - Unfortunately I think hiding the file hierarchy will contradict the aims of the project but thanks for the suggestion. I will take a look at the projects you mentioned though

Ajay - Thanks for the little walkthrough. I have managed to get as far as compiling the kernel in a ubuntu distro but nothing seemed to be different after the edit/recompile/install/reboot process. I will try again with your advice once I get this system set up and stable!

My original post came across like i just want to throw in random bits of code and see what happens, not the case! Hopefully it will be a bit more planned than that.

As for where I'm focusing attention, I plan to look into doing some VFS modification for the benefit of simplification for the user when browsing directories and such-like. Purely experimental for the project I mentioned. When I get something working as intended I can post back here

In the meantime, thanks for the advice. I've got plenty to be getting on with but I'm sure I'll be back here for some more help at some point!