Open House Training On Linux Kernel Development - Part 1: Modules, Interfaces and Internals (Location: Chennai)

Open House Training On Linux Kernel Development - Part 1: Modules, Interfaces and Internals (Location: Chennai)

Duration: 35 hours (5 days @ 7 hours per day)

Course Description:

This training program intends to equip Linux developers with necessary skills to develop, debug and diagnose issues within the Linux kernel-space.

Target Audience (Who should attend):

  • Engineers working on Linux-based systems willing to get a
    deeper understanding of the Linux kernel.
  • Engineers intending to migrate onto Linux kernel for device
    driver development.
  • Architects intending to understand how the Linux kernel works
    to better leverage its capabilities into their projects.


  • Fair knowledge of C programming and data-structures using C.
  • Fair knowledge on UNIX/Linux systems and application programming.
  • Fair knowledge of computer hardware architecture.

Course Objectives:

  • Learn about Linux kernel architecture.
  • Learn and understand the design intricacies of the Linux kernel.
  • Learn about Linux kernel coding guidelines, best practices
    and approaches towards development on the Linux kernel.
  • Learn how to create custom patches and apply them on Linux
    kernel sources.
  • Learn about various Linux memory management,
    process management, filesystems, concurrency and
    synchronization mechanisms within the kernel and best
    practices for choosing them.
  • Learn about Linux kernel interfaces
    (procfs, sysfs, debugfs, syscalls)
  • Learn about Linux kernel modules (LKM) and learn how to
    create LKMs (for 3.x/4.x kernels)
  • Learn how to customize Linux kernel by source
  • Linux kernel debugging techniques covering kernel
    debugging configuration parameters (lock debugging,
    kmemleak detector), kprobes and remote debugging
    using KGDB.
  • Learn various Linux kernel data-structures
    and kernel-mode functions relevant to kernel

About the Instructor:

This training program will be handled by Mr. Chandrashekar Babu - an experienced FOSS technologist and corporate trainer with 20 years of experience in the IT industry. He has been a Linux enthusiast since late 1995 and has been a polyglot with diverse technology and programming language experience on Free Software/Open Source Software ecosystem.

He has been delivering training programs to some of the most prominent product companies on various technology domains (that also include Linux Kernel Internals / Device Drivers) since 2003.

You can know more about him by visiting the following URLs:

Training Cost: ₹ 15,000/- (inclusive of taxes)

How to register/enroll:

To enroll/register for this training program, kindly drop an
e-mail to and await reply.

Lab setup:

  • Participants are expected to carry-in their laptops
    with the following software installed and configured:
  • A laptop with high-end configuration (Intel Core i5 or
    above with atleast 4 GB of RAM) is recommended.
  • Participants must ensure that atleast 16 GB of
    free disk-space is available on their laptops - as this
    is required for installing a pre-configured Linux setup
    as a VirtualBox appliance for hands-on practice session
    in this training program.

What will be provided at the end of training program ?

  • At the end of the training program, participants can take
    personal copies on their laptops/external hard drives
    the following:
    • Complete recording of screencasts along with audio of the
      entire training sessions for future reference.
    • Presentations, notes and code samples for topics being covered.
    • The virtual machine appliance used during training sessions
      for practising hands-on.
  • A hard copy book on Linux Kernel Development by Robert Love
    for each attending participant.
  • Course completion certificate.

Demo screencasts

Training venue:

Slashprog Technologies, #C9 / C49, First floor, Mogappair Industrial Estate, Mogappair West, Chennai - 600 037.

How to reach venue:

  • The closest prominent landmark to the venue is Vavin bus-stop towards
    Ambattur Industrial Estate and HCL Technologies.
  • Take the road that leads towards Mogappair West depot from Vavin signal.
  • At approximately 350 metres, you would find Bata showroom, Grill Ministri and Fashion Look showroom.
  • Take the road that leads to the right opposite to Fashion Look.
  • Take the second right turn approximately 160 metres on this roads (opposite to Berton Tyres and Fosroc Chemicals).
  • Enter the third building at the left on this street and walk upstairs to reach the Venue.

You can alternatively search for the following address on Google Maps:
C-9 Jeswant Nagar, Ambattur Industrial Estate Chennai, Tamil Nadu 600037 and use directions to reach there.

Training schedule:

  • April 13th 2015 (Monday) to April 17th 2015 (Friday).
  • 9:30 AM to 5:30 PM [Lunch break between 1:00 PM to 2:00 PM]

NOTE: Lunch cannot not be arranged at the venue due to logistic reasons at this time. Please kindly co-operate.

Course Agenda

NOTE: The below agenda details proposed topics for this training program. The flow of topics and the coverage will depend on wavelength of majority of attending participants for this training program. The focus will be more on ensuring that the attending participants gain better clarity of the fundamentals before proceeding towards advanced topics.

Day 1

Introduction to Linux kernel

  • Introductory overview on Linux ecosystem
  • Design goals of the Linux kernel
  • Linux kernel architectural overview
  • Kernel-mode vs User-mode
  • Features and benefits of Linux kernel design
    • Monolithic kernel architecture
    • Processes and Files
    • "Everything is a file" paradigm
    • Virtual memory using flat memory model
    • Support for dynamically loadable kernel modules
  • Typical flow of a Linux system call
  • Linux kernel and process view of the system

Hands On

  • Identify some important documentation bundled with the Linux
    kernel source.
  • Understand the directory structure/organization of Linux
    kernel source tree.
  • From boot to init: a high-level code walkthrough of the Linux
    kernel source to understand initialization sequence of various
    subsystems during Linux kernel boot-up.

Day 2

Memory management within the Linux kernel

  • Memory management overview on Linux
  • Virtual memory, demand paging and swapping
  • Linux kernel data-structures to abstract memory
  • Lazy-TLB support on Linux
  • Zone allocator and related API
  • Memory management mechanisms - SLOB vs. SLAB vs. SLUB allocator
  • Using kmem_cache series of functions
  • Using kmalloc functions
  • Understanding the GFP flags
  • Using vmalloc functions
  • Creating custom emergency memory pools
  • Choosing the right memory management functions and best practices
  • Managing user-space memory access (copy_to_user(), copy_from_user() and related functions).

Linux kernel modules [Demonstration / Hands-On]

  • Linux kernel module organization
  • Loadable device drivers and feature enhancement modules
  • Managing Linux kernel modules

Linux kernel module development

  • Important header files.
  • Kernel module functions.
  • Compiling LKMs.
  • Exporting symbols from LKM.
  • User libraries Vs. Kernel mode library functions.
  • Module-specific macros and rudimentary functions.
  • Handling module parameters.
  • Using printk() series of functions, dump_stack(),
    BUG(), WARN() and panic().
  • Using dmesg command and /proc/kmsg to read kernel log messages.
  • Managing kernel logs using klogd/syslogd.
  • Tips, techniques and conventions used in writing programs within
    the Linux kernel.

Hands On

  • Using kernel modutils tools (lsmod, insmod, modprobe,
    depmod, modinfo).
  • Creating simple kernel modules (for 3.x kernels).
  • Using kernel mode memory management functions.
  • Simple kernel module examples emphasizing on basic kernel-space
  • A simple kernel module emphasizing on interfacing /proc

Day 3

Linux process model

  • Introduction to Linux processes.
  • Process states and process groups.
  • Processes Vs. Threads.
  • Linux kernel threads.
  • Linux kernel task structure.
  • An overview on CFS scheduler.
  • Process scheduling policies and mechanisms.
  • Process management primitives - fork(), exec(),
    vfork(), clone().
  • Scheduler related functions for managing processes and threads
    from within kernel-space.

Linux filesystems

  • Introduction to Linux filesystems and filesystem structure
  • Linux Kernel VFS layers and architecture
  • Linux file types, dentry and stat information
  • Linux device model and device files.
  • Pseudo filesystems (procfs, sysfs, debugfs)

Hands On

  • Write Linux kernel module to interface with /proc filesystem
  • Write Linux kernel module to interface with debugfs
  • Gather statistics/diagnostic information and manage processes
    using files exposed by the kernel via /proc interface.
  • Creating device files and understanding how they are wired to
    device drivers.

Day 4

Linux kernel execution contexts

  • Process context
  • Interrupt context
  • SoftIRQ context
  • Kernel functions - in_interrupt(), in_softirq(),
    in_atomic(), preemptible(), preempt_disable(),
    preempt_enable(), local_irq_disable(), local_irq_enable()

Concurrency mechanisms within the Linux kernel

  • Interrupt handlers
  • Threaded IRQ handlers
  • An overview on SoftIRQs
  • Timer functions and hrtimer functions
  • Using ktime, delay and sleep functions
  • Tasklets and High-priority Tasklets
  • Kthreads
  • Concurrency Managed Workqueues (and delayed workqueues)
  • Best practices for designing module to differ work within

Synchronization mechanisms within the Linux kernel

  • Memory barriers
  • Per-CPU variables and Atomic variables
  • Spinlocks, reader-writer locks
  • Seqlocks and RCU
  • Semaphores and mutexes
  • Completions and wait-queues
  • Notifier chains
  • Best practices for choosing the right synchronization
    mechanism for a given requirement.

Hands On

  • Simple kernel modules focussing on tasklets, timers,
    work-queues and kthreads.
  • Simple kernel modules focussing on synchronization
    mechanisms using wait-queues, semaphores and completions.
  • Using notifier chains within the Linux kernel
  • Gathering statistics and managing IRQ lines using /proc/irq/

Day 5

Debugging techniques on the Linux kernel

  • Constraints and limitations involved in debugging the
    Linux kernel.
  • Understanding Kernel Support for Debugging

Debugging using printing

  • Using printk() series of functions.
  • Using pr_debug() and dynamic printk functions.
  • Using early_printk(), dump_stack(), BUG(),
    BUG_ON(), WARN(), panic()
  • Redirecting console messages.
  • How messages get logged
  • Turning messages on and off
  • Rate limiting of console messages
  • Debugging faults within Linux kernel.
  • Deciphering Oops messages.

Using the Debug Filesystem

  • Mounting and analyzing debugfs.
  • Techniques and best practices for exposing kernel
    variables and data-structures using debugfs for inspection
    from user-land.

Debugging and instrumentation techniques using kprobe

  • Introduction to kprobes
  • Using kprobes, jprobes and kretprobes
  • Performance analysis and intrumentation techniques
    using kprobes


  • An overview on Linux tracers
  • Using function tracers to trace and instrument various
    kernel functions.
  • Using MMIO tracer to trace all MMIO operations on memory-mapped
    hardware registers.
  • An overview on blktrace, LatencyTOP, SystemTap and other tools
    built on-top of Linux kernel tracers.

An overview on KGDB

  • Configuring remote debugging using KGDB via serial
  • Using KGDB with KDB integrated support.
  • Setting breakpoints and analyzing kernel variables
    using GDB console.
  • Known issues and limitations with KGDB.
  • Using KDB interface and writing custom KDB plugins.

Other debugging features within the Linux kernel

  • Using the Magic SysRq key
  • Using hardware breakpoint interface to detect potential
    memory corruption.
  • Using kmemleak to check potential memory leaks.
  • Using kexec and kdump to analyze kernel crash dumps.

Building Linux kernel from the sources (Optional *)

* Based on time availability
  • Steps involved in deploying custom compiled Linux kernel from
    the available sources.
  • A quick walkthrough on Linux kernel source configuration parameters
  • Linux kernel optimization techniques
    • Speed/Performance optimization
    • Size optimization
    • Security optimization (Kernel hardening)
    • Feature optimization
  • Debugging options in the Linux kernel
  • Creating custom boot image with busybox user-land.

Hands On

  • Using printk() for debugging
  • Stack backtrace analysis from kernel crash
  • Using Magic SysRq
  • Writing simple module to expose variables and
    structures to user-mode using debugfs
  • Writing kprobe modules to analyze kernel
    function arguments and return values
  • Remote debugging using kgdb

Q & A