Bolt: Data Management of Connected Homes

Bolt is a data management system for emerging class of applications that helps IoT devices to interact and store data. The unique requirements of these applications such as support for time-series and tagged data, ability to share data between devices and assurance on data confidentiality & integrity have made the older platforms unsuitable. These platforms such as HomeOS, MiCasa Verde and so on provide high-level abstraction mainly for devices to interact and not for storage. The following paragraph elaborates the data manipulation characteristics of the IoT applications which stand as one of the main reasons for creating bolt.    

The observed data manipulation characteristics of the IoT applications are 1) a single writer exit, 2) always generate new data, 3) no random access to it, and 4) retrieve proximate records from the data streams. The traditional databases with support for transactions, concurrency control, and recovery protocols are an overkill for these data and file-based storage offers inadequate query interface as filesystem access happens in sequential order. In addition, data need to be shared between applications and secured while in transit and stored on a storage medium. It should also provide support for policy-based storage that helps minimize cost and efficient utilization of resources. Bolt supports the above data management characteristics, unlike the present storage abstractions. Next, we are going to explain the key techniques used by the bolt to tailor data management for the above applications.    

The four main key techniques are chunking, separation of index & data, segmentation, and decentralized access control & signed hash. Chunking is a process of grouping a contiguous sequence of records into chunks. It helps to increase the efficiency of the system by reducing the round trip delay incurred while data access (batching chunks). Data is accessed and stored at the granularity of chunks. Second, separation of index & data help us in two ways 1) index are queried locally 2) trust assumption of the cloud (data stored encrypted in the cloud and decryption happens only at the client side). Third, segmentation is the process of dividing data streams into smaller segments of users defined size. It helps to archive the streams as the amount of data in the stream increases.  Finally, bolt use decentralized access control and signed hash to provide confidentiality to data stored at the untrusted cloud storage. It encrypts the data with the owner’s secret key and distribute the keys via a trusted key server.  The subsequent paragraph gives an idea about bolt’s implementation.

Bolt API’s allow us to create a data stream which is of two types: ValueStream and FileStream. Former is used for writing small data value such as temperature reading and the latter for larger values like images or videos. The data is added to the stream as a time-tag-value using an append API. A stream consists of two parts – a log of data record   (DataLog) and an index that maps a tag to a list of data item identifiers. When a stream is closed, Bolt chunks the segment DataLog, compress and encrypts these chunks and generates a ChunkList. It then uploads the chunks, updated ChunkList and index to the storage server. The chunks are uploaded in parallel and application can configure the maximum number of parallel uploads. Finally, stream’s integrity metadata is uploaded to the metadata server. As mentioned in the previous paragraph, streams are encrypted with a secret key known only to the owner. If the owner wants to give access to other readers, it updates the stream metadata with secret key encrypted with the reader’s public key. In case of reading the data, it first checks the integrity of the metadata with the owner’s public key and the freshness using TTL in-stream metadata before downloading the index and DataLog.

In this paragraph, I am listing few drawbacks of bolt.  1) Fully dependent on control plane, 2) devices unable to subscribe a particular data stream generated from a device 3) each device has its own data stream (missing feature in bolt to merge data stream) 3) prone to pitfalls of the current IoT applications which leverage the cloud for storage (as bolt is using cloud storage), and 4) global scalability will be a challenge as bolt lack location-independent routing of segments. Bolt also uses custom IoT gateways, hence, can lead to interoperability issues.

The performance of bolt was evaluated in two ways: microbenchmark (compared with operating systems read and write: DiskRaw stream operations) and real-world use-cases. In the first approach, they took performance measurements for writes, reads, and scalability. The comparison was done for ValueStream, FileStream, and remote ValueStream. The ValueStream was compared to a single file in DiskRaw; the FileStream with multiple files. The results show ValueStream incurred higher overhead for local writes compared to DiskRaw. For remote streams, 64% of total time was taken to chunk & upload the DataLog; 3% went for index upload. In case of FileStream, its performance is comparable to DiskRaw for local writes. The storage overhead was compared for ValueStream over DiskRaw, it decreases with larger value size. The read performance of the local ValueStream was hindered by the index lookup and a data deserialization. The cost of download dominated for remote reads from ValueStream. The FileStream also have similar performance metrics.  The chunking of streams helped to improve the read throughput for temporal range queries. Finally, the time taken to open a stream depends on the time to build the segment index in memory and it grows linearly with the number of segments. The second part of the evaluation is explained in the next paragraph.

They conducted feasibility and performance analysis of bolt with three real world applications such as PreHeat, Digital Neighborhood Watch (DNW), and Energy Data Analytic (EDA). The results were compared with the performance of these applications while using openTSDB. In the first application, the average retrieval time from remote ValueStream decreases with increase in the chunk size. In DNW, chunks improve retrieval time by batching transfers even though it downloads additional data it might not require.  With respect to EDA application, a proportional increase in retrieval time for both bolt and openTSDB was observed. Bolt outperform openTSDB by an order of magnitude primarily due to the prefetching of data in chunks. The storage overhead of bolt is 3-5x lesser than openTSDB for all the above applications.

The experiments are excellent and show the benefits of bolt data management system. But, we found the following two drawbacks in bolt: 1) comparison between openTSDB and bolt may be incorrect as openTSDB is a relational database ( even though it supports time-series data ), 2) scalability is weakly tested while doing microbenchmark.

To conclude this summary, bolt is a perfect data management system for emerging class of applications which manage the IoT devices at home. It meets all the requirements of these applications which are unavailable on the existing platforms. The experiments carried out in this paper shows that compared to the openTSDB, bolt performs 40 times faster with 3-5x lesser storage overhead. The drawback highlights the challenges that need to be solved in order to deploy bolt in a highly scalable use case.

Leave a comment

Filed under bolt, Distributed Systems, IoT, Operating Systems, Storage

The Cloud is Not Enough: Saving IoT from the Cloud

The Internet of Things(IoT) represents a new class of applications which leverages the advantage of the cloud. This has allowed us to collect data from sensors and stream it to the cloud without worrying about the economic viability of storing and processing this data. But the current approach which is used to connect the IoT applications directly to the cloud has many drawbacks. The concerns regarding privacy, security, scalability, latency, bandwidth, availability and durability of data generated by these IoT applications has not been addressed. In order to overcome these drawbacks,  a data-centric approach has been adopted to create an abstraction between the IoT applications and the cloud.

The data-centric abstraction is called a Global Data Plane(GDP) which focuses on distribution, preservation, and protection of data. It supports the same application model as cloud while better matching the needs and characteristics of the IoT by utilizing the heterogeneous computing platforms, such as small gateways devices, moderately powerful nodes in the environment and the cloud in a distributed manner. The basic foundation of GDP is the secure single-write log and applications that are build on top of it are interconnected  through log streams rather than by addressing devices or services via IP.

The data generated by the IoT devices are represented as logs, also called as single-writer time series logs. This log is append only; mostly read-only and can be securely replicated and validated through cryptographic hashes.  The log-based approach deals with the issues of flexibility, access control, authenticity, integrity, encryption, durability and replication of data. These logs also need to be stored onto the infrastructure, current storage approach on the cloud doesn’t offer flexible placement, low latency or durability of information. To enable these this paper introduce Location-independent Routing in which packets are routed through an overlay network that uses Distributed Hash Table(DHT) technology. Dynamic topology change, pub/sub, and the multicast tree can be built over these overlay network in order to optimize latency and network bandwidth.  Although GDP can provide most of the functionality that are needed for applications, some applications may need additional support which can be provided by Common Access API(CAAPI). CAAPI is a layer above the GDP layer and plays a major role in replaying logs when a service fail. Checkpointing techniques can be used to avoid the overhead incurred due to log reply.

The data-centric approach used in this paper has help to overcome the pitfalls of today’s IoT applications. Though these problems are prevalent in web applications; when it comes to IoT space, it becomes more complex. I have written this writeup based on the paper “The Cloud is Not Enough: Saving IoT from the cloud”[1].


Leave a comment

Filed under Distributed Systems, Global Data Plane, IoT, Storage

Making a 64-bit Operating System from Scratch

A computer Operating System does lot of complex tasks like process management, memory management, I/O management etc and building one of its kind is the hardest task for a Computer scientist. But it will be always exciting to experience the process of building an Operating System. This blog post will walk through the steps in making the operating system.

You need to have a cross compiler install on the machine in which you are going to build the new operating system. I have explain this process in my blog “Building a cross compiler“.

First we have to create an assembly file boot.s, which will setup virtual addressing and long mode,


[BITS 32]
[SECTION .mbhdr]
[EXTERN _loadStart]
[EXTERN _loadEnd]
[EXTERN _bssEnd]
 ; Magic
 DD 0xE85250D6
 ; Architecture
 DD 0
 ; Length
 DD HdrEnd - MbHdr
 ; Checksum
 DD -(0xE85250D6 + 0 + (HdrEnd - MbHdr))
 ; Tags
 ; Sections override
 DW 2, 0
 DD 24
 DD MbHdr
 DD _loadStart
 DD _loadEnd
 DD _bssEnd
 ; Entry point override
 DW 3, 0
 DD 12
 DD EntryPoint
 DD 0 ; align next tag to 8 byte boundary
 ; End Of Tags
 DW 0, 0
 DD 8
 ; Hdr End Mark
[SECTION .boot]
[GLOBAL EntryPoint]
[EXTERN Stack]
 mov eax, Gdtr1
 lgdt [eax]
 push 0x08
 push .GdtReady
 mov eax, 0x10
 mov ds, ax
 mov ss, ax
 mov esp, Stack
 call SetupPagingAndLongMode
 mov eax, Gdtr2
 lgdt [Gdtr2]
 push 0x08
 push .Gdt2Ready
[BITS 64]
[EXTERN main]
 mov eax, 0x10
 mov ds, ax
 mov es, ax
 mov ss, ax
 mov rsp, Stack + 0xFFFFFFFF80000000
 ; If you later decide to unmap the lower zone, you will have an invalid Gdt if you're still using Gdtr2
 mov rax, Gdtr3
 lgdt [rax]
 mov rax, main
 call rax
 jmp $
[BITS 32]
 mov eax, Pdpt
 or eax, 1
 mov [Pml4], eax
 mov [Pml4 + 0xFF8], eax
 mov eax, Pd
 or eax, 1
 mov [Pdpt], eax
 mov [Pdpt + 0xFF0], eax
 mov dword [Pd], 0x000083
 mov dword [Pd + 8], 0x200083
 mov dword [Pd + 16], 0x400083
 mov dword [Pd + 24], 0x600083
 ; Load CR3 with PML4
 mov eax, Pml4
 mov cr3, eax
 ; Enable PAE
 mov eax, cr4
 or eax, 1 << 5
 mov cr4, eax
 ; Enable Long Mode in the MSR
 mov ecx, 0xC0000080
 or eax, 1 << 8
 ; Enable Paging
 mov eax, cr0
 or eax, 1 << 31
 mov cr0, eax
 DQ 0x0000000000000000
 DQ 0x00CF9A000000FFFF
 DQ 0x00CF92000000FFFF
 DQ 0x0000000000000000
 DQ 0x00A09A0000000000
 DQ 0x00A0920000000000
 DW 23
 DD TmpGdt
 DW 23
 DD TmpGdt + 24
 DD 0
 DW 23
 DQ TmpGdt + 24 + 0xFFFFFFFF80000000

To combine all the object files into a single executable we need a linker.ld file,


{    . = 0x100000;
     .boot :
         _loadStart = .;
         . = ALIGN(4096);
         Pml4 = .;
         . += 0x1000;
         Pdpt = .;
         . += 0x1000;
         Pd = .;
         . += 0x1000;
         . += 0x8000;
         Stack = .;
     . += VIRT_BASE;
     .text ALIGN(0x1000) : AT(ADDR(.text) - VIRT_BASE)
     .data ALIGN(0x1000) : AT(ADDR(.data) - VIRT_BASE)
     .rodata ALIGN(0x1000) : AT(ADDR(.rodata) - VIRT_BASE)
     _loadEnd = . - VIRT_BASE;
     .bss ALIGN(0x1000) : AT(ADDR(.bss) - VIRT_BASE)
     _bssEnd = . - VIRT_BASE;
     /DISCARD/ :

Next step will be to create a main.c file which will print “hello world” on to the screen.



static const uint8_t COLOR_BLACK = 0;
static const uint8_t COLOR_BLUE = 1;
static const uint8_t COLOR_GREEN = 2;
static const uint8_t COLOR_CYAN = 3;
static const uint8_t COLOR_RED = 4;
static const uint8_t COLOR_MAGENTA = 5;
static const uint8_t COLOR_BROWN = 6;
static const uint8_t COLOR_LIGHT_GREY = 7;
static const uint8_t COLOR_DARK_GREY = 8;
static const uint8_t COLOR_LIGHT_BLUE = 9;
static const uint8_t COLOR_LIGHT_GREEN = 10;
static const uint8_t COLOR_LIGHT_CYAN = 11;
static const uint8_t COLOR_LIGHT_RED = 12;
static const uint8_t COLOR_LIGHT_MAGENTA = 13;
static const uint8_t COLOR_LIGHT_BROWN = 14;
static const uint8_t COLOR_WHITE = 15;

uint8_t make_color(uint8_t fg, uint8_t bg)
       return fg | bg << 4;

uint16_t make_vgaentry(char c, uint8_t color)
       uint16_t c16 = c;
       uint16_t color16 = color;
       return c16 | color16 << 8;

size_t strlen(const char* str)
       size_t ret = 0;
       while ( str[ret] != 0 )
       return ret;

static const size_t VGA_WIDTH = 80;
static const size_t VGA_HEIGHT = 24;

size_t terminal_row;
size_t terminal_column;
uint8_t terminal_color;
uint16_t* terminal_buffer;

void terminal_initialize()
       terminal_row = 0;
       terminal_column = 0;
       terminal_color = make_color(COLOR_LIGHT_GREY, COLOR_BLACK);
       terminal_buffer = (uint16_t*) 0xB8000;
       size_t y, x;
       for ( y = 0; y < VGA_HEIGHT; y++ )
            for ( x = 0; x < VGA_WIDTH; x++ )
                   const size_t index = y * VGA_WIDTH + x;
                   terminal_buffer[index] = make_vgaentry(' ', terminal_color);

void terminal_setcolor(uint8_t color) 
        terminal_color = color; 

void terminal_putentryat(char c, uint8_t color, size_t x, size_t y) 
        const size_t index = y * VGA_WIDTH + x; 
        terminal_buffer[index] = make_vgaentry(c, color); 

void terminal_putchar(char c) { 
        terminal_putentryat(c, terminal_color, terminal_column, terminal_row); 
        if ( ++terminal_column == VGA_WIDTH ) 
             terminal_column = 0; 
             if ( ++terminal_row == VGA_HEIGHT ) 
                   terminal_row = 0; 
void terminal_writestring(const char* data) 
       size_t datalen = strlen(data); 
       size_t i; 
       for ( i = 0; i < datalen; i++ ) 
int main() 
       terminal_writestring("Hello, kernel World!\n"); 

The final step before build the new operating system is to create a “Makefile


ISO := os.iso
OUTPUT := kernel.sys

OBJS := boot.o main.o

all: $(ISO)

        cp $(OUTPUT) iso/boot
        grub-mkrescue -o $@ iso

$(OUTPUT): $(OBJS) linker.ld
        ld -nodefaultlibs -Tlinker.ld -o $@ $(OBJS)

        nasm -felf64 $< -o $@

        gcc -m64 -mcmodel=kernel -ffreestanding -nostdlib -mno-red-zone -c $< -o $@

        @rm -f $(OBJS) $(OUTPUT)

The final stage involves creating the bootstrap script, which will be now locate at “iso/boot/grub/“. The file grub.cfg will contain the following code,

menuentry "HelloOS" {
    multiboot2 /boot/kernel.sys

Now, from the directory in which the code resides, run the command,


This command will compile and create os.iso file.

In order to test the OS, you need to have a virtual machine, it can be either Qemu, VirtualBox or

I have tested it in qemu,

qemu-system-x86_64 os.iso

This will print “Hello World” on the screen.

Leave a comment

Filed under Operating Systems

DebUstav ’14

Debutsav ’14 is a version of the Debian’s developer conference conducted by Debian and Amrita University. Debian conducts DebConfs yearly where developers of the Debian operating system meet to discuss issues around the further development of the system. Debutsav ’14 is a mini version of the Debconf and it is gonna be held at Amrita University, Kollam. Debian is a free operating system, developed by a group of individuals known as the Debian project. Debian is one of the most popular Linux distributions for personal computers and network servers, and has been used as a base for several other Linux distributions. It is a distribution which is famed for its stability and which has some of the widest array of software in its archive serving almost all the Computer system hardware architectures known to mankind.DebUstav Log

DebUtsav is an amazing platform for any student who is passionate to experience the magic of open source. It is a great opportunity for developers, contributors and other interested people to meet the veterans in open source software development, develop professional connections,  chat with contributors who can guide you, present in front of a grand audience, and also comprehend the essence of open source development. This conference will be beneficial for developing key components of the Debian system, infrastructure and community. There are lots of free softwares which are not yet packaged in Debian, and the Debutsav might become a platform where the speakers act as a catalyst to have more interesting softwares come into Debian or to work together and try to solve the issues, or if not solve it then at least guide people to what could be the way forward.

The Debutsav ’14 is accepting proposals for talks, hands-on-sessions or workshops. The registrations will be open till the 7th of October. The topic/theme can be anything related to Free and Open Software. This is a grand rostrum for the students, speakers and delegates to talk, share, learn, discuss, debate and do software development. It is indeed a significant opportunity that will give you a strong foundation to build on! This great event is to be held on 17th and 18th of October.

So what are you waiting for? Register for being a speaker at the Debutsav ’14. Click here to register and be proud to be a part of it.. :)

1 Comment

Filed under Uncategorized

Building a cross compiler

Building a cross compiler is not a hard task, but still if you are not doing it in the right way it can waste lot of your time. This blog post is based on the steps I had followed in my machine with Ubuntu 14.04 installed on it.


First, make sure that you keep all the source in a directory “$HOME/src”.

mkdir $HOME/src
cd $HOME/src

tar xvf binutils-2.24.tar.gz
tar xvf gcc-4.9.1.tar.gz

You can change the version number according to the latest one. The version number can be obtained from the gcc website.

Install few supporting packages before you start the build process,

sudo apt-get install libmpc-dev
sudo apt-get install libcloog-isl-dev
sudo apt-get install libisl-dev
sudo apt-get install libmpfr-dev
sudo apt-get install libgmp3-dev

Hope these packages are also in other linux distros.

Now we need to decide on where to install our new compiler,  it is dangerous to install it in the system directory. So we can create a directory $HOME/opt/cross. If you want it to be global “/usr/local/cross” is the ideal location for placing the compiler.

Preparing to compile by setting the correct PATH variable, this can either executed as a shell command or it can be include in the ~/.bashrc file,

export PREFIX="$HOME/opt/cross"
export TARGET=x86_64-elf 
export PATH="$PREFIX/bin:$PATH"

Taking the first step by building Binutils, 

cd $HOME/src 
mkdir build-binutils 
cd build-binutils 
../binutils-2.14/configure --target=$TARGET --prefix="$PREFIX" --disable-nls --disable-werror 
make -j12 
sudo make -j12 install

Building GCC, 

mkdir build-gcc
cd build-gcc
../gcc-4.9.1/configure --target=$TARGET --prefix="$PREFIX" --disable-nls --enable-languages=c,c++ --without-headers
make -j12 all-gcc
make -j12 all-target-libgcc
sudo make install-gcc
sudo make install-target-libgcc

Now you have a new compiler which does not have access to C library or C runtime.

Leave a comment

Filed under linux, Operating Systems

Evolution of Virtual Memory

Before the development of higher level programming language, the programmers had to implement storage allocation methods into his application. The application was divided into different overlays and loaded into the memory one at a time as the size of the memory was confined. This methodology was easy as the programmers were aware of the machine details and his application. But things started changing with the launch of higher level programming language and growing size of application, which led to unmanageable overlays. To overcome this problem, Fotheringham in 1961 come up with the solution of  “Dynamic Memory Allocation” in which the task of memory allocation was done by the software and hardware. After the implementation of Dynamic Memory Allocation, the programmer can focus more on problem solving rather than spending time on understanding the hardware details. It also gives the illusion to the programmers that they have vast amount of memory for their use.


Now, let us see how this concept was implemented in Atlas computers,

The main memory in Atlas is divided into core store and drum store. The processor has immediate access to the core store, but it is limited in size as a result some data need to be stored in the drum store. The process of swapping data between the core store and drum store is done by SUPERVISOR, a housekeeping program which resides in memory. The memory contains 1 million 48-bit words and its registers are grouped into units of 512, but it is to be noticed that the 512-word block of information and 512-word unit of memory are different. The 512-word unit of memory in the core store is called ‘page’, and it can hold a block of information. Associate with each ‘page’ there is a 11-bit  page address register which stores the number of the block.

The “address” of a word in the main memory is 20-bit long, and consist of 11-bit block address and 9-bit position address within the block. The concept of “address” is distinct from the actual physical location, this distinction is a core idea in this system.  When a block needs to be loaded into the memory it is assigned an address within the 20-bit range and this address is notified to the SUPERVISOR. While the block is brought into the core store, SUPERVISOR arranges for that area of core store to have appropriate addresses. It also keeps track of  blocks in the drum with the help of  a table. During execution, if the program is unable to find a block in the core store, it sends interrupt to the SUPERVISOR which loads the required block into memory. In order to make room for this block, a block in the core store need to be written back to the drum. Drum Transfer Learning Program does this job of selecting which block need to be written back.

“Dynamic Storage Allocation in Atlas Computers, Including an automatic use of backing store” is the first paper on “Virtual Memory” concept in operating systems. A significant work on Virtual Memory was done by Peter J. Denning at Princeton University, his paper “Virtual Memory” extends the work done in this paper.

Leave a comment

Filed under Operating Systems

BitTorrent Ecosystem

I have been hearing about torrents for long and the only thing that I knew about it was that we can download movies from it. Rather than just downloading movies from torrents, have any one tried to understand how it works? Have any one thought how complex it is? Do you know that BitTorrent generates more that 40% of internet traffic?

BitTorrent had its beginning from a university where student had developed it to share resource among themselves. They never knew that it will make such a huge impact on the internet. Though it is a major technical discovery there have been wide spread protest against it from the movie industry to shutdown public torrent discovery site like utorrent, pirate bay etc to protect their interest.

torrent-new Let me dive to into the BitTorrent Ecosystem. The Ecosystem consist of three main component,

a) Tracker: Computer which helps the peers to find each other to form a torrent.

b) Torrent Discovery Sites: The place where people can find and upload torrent file.

c) Peers: The computers which forms a torrent.

“The collection of peers that participate in distribution of a specific file at a given time is called as a torrent.” 

The peers in a torrent can be classified into two groups, seeders and leechers. Seeders are the peers which contains the file completely and leechers contain the file partially. The files are distributed as chunks and the bittorrent client keep track of chunks which are downloaded into the peers. It assures that the complete file is downloaded from the peers in the torrent with integrity.

If a file need to be shared, it will be seeded locally by the peer which starts the torrent. The .torrent file which is yield after the seeding process can be uploaded into a  torrent discovery site. Those who need that particular resource can download the torrent file and open it in the bit-torrent client they have installed locally. The torrent file contain the metadata regarding the file which need to be distributed and the network location of trackers.

Peer A create a torrent to distribute the file movie.avi by first seeding that file locally and then registering itself with a tracker, it creates a movie.torrent file which it uploads into the torrent discovery site. If the peer B what to join this torrent, it downloads the movie.torrent file from the torrent discovery site and with the help of the bittorrent client which extract the details of the torrent from the movie.torrent file joins the torrent by contacting the tracker.

This peer-to-peer protocol has helped in the distribution of file with efficient use of bandwidth and resources.   But there has been various privacy and security concerns in the bittorrent ecosystem which need to be addressed.

Leave a comment

Filed under General, web