Index

  • My first Android app

    Posted on Monday, April 18th, 2016
    0 comments

    My first submission into the Google Play store isn't Python related, but demonstrates what sort of apps I release into the store. My applications will be licensed under the GPL2 license, but also charge an affordable price within the Play Store. I am opting to have my apps open source in the efforts to show potential privacy conscious consumers that my apps are going to be some of the safest to use within the Play Store.

    I recently got this idea from the abundance of Call Blocking/Screening apps in the play store. I was actually too scared to download these apps due to the requested permissions. They required Internet, the ability to dial phone numbers, and modifying my phone state. The last two are the only ones required for call blocking/screen to function. The Internet permission was being used for serving up ads, or so I would like to believe. I personally had no way to determine if the Internet permission was going something else to the information the app collected. I was also clueless on how it performed said blocking and screening functions... In the end, I never downloaded the app, instead I did some research and built my own.

    Feeling that other consumers might be in the same boat when it comes to installing various apps on their personal devices. I thought this would be an excellent market to enter. In a single sitting this evening, I opened a developer account, and created a full Android application. All of the source code is fully available as well, and marked clearly in the application, and metadata in the Play Store. I am charging a small fee of only $0.99 in the Play Store for the app, this is so that consumers can clearly support my motives, and allow me to spend more time further developing apps in the same spirit. Not all the apps I plan to release into the Play Store will charge, I only decided on this one as I wanted to see how well a privacy minded open source application would monetize in the play store.

    The app currently doesn't support separate white or black lists. In fact, the app is so simple and easy to use, that the whitelist is compromised of just your Contacts. I am also able to do this without actually requesting access to read your contacts. It also has the ability to blacklist all calls from North American toll-free numbers, in-case you have been receiving an excess of telemarketers or other spam from toll-free numbers. The settings screen itself is clear and easy to turn each feature either on or off.

    You can view the source for this project on my BitBucket project page. As it can be very informative to learn how this sort of application works on Android.

    The actual process of submitting an application was mostly easy, except for the need to draw graphics assets for the Google Play page... I'm not the best artist, so my icon and banner look rather awful. Perhaps if this app makes some money, I will hire an artist and have better assets drawn. Once the app is live, you can see it on the Play Store. If you have need for such an app, do purchase and download it. I'd be glad to take suggestions for other Android apps I may work on in the future. One idea is to convert over some of my SL4A Python tools into native Android, as my YouTube channel showing these tools got a fair amount of hits.

  • Remember the 6502 microprocessor?

    Posted on Sunday, April 3rd, 2016
    1 comment

    It has been a very long time since I talked about one of my side projects, Hacker's Edge. Over the weekend I found sometime to put in some work into it. The ending result is that Hacker's Edge now runs 6502 hosts in-game!

    While I'm not 100% complete yet with the integration, both the 6502 assembler and virtual machine are fully functioning within the game world. The end goal is to basically replace all my hardcoded Python routines with pure 6502 machine code, assembled using the in-game assembler. Why you ask? Because, this is a game about hacking, and my goal is to make the simulation as immersive as technically possible. This means that the 6502 virtual machine will take control of the game's mechanics as soon as possible. Here is a basic drill down of how a user log into to the game world will be like:

    1. Player successfully logs in, check to see if host is online.
    2. If host is offline, we initiate an empty memory address space for the VM, we attempt to load in KERNEL.SYS
    3. KERNEL.SYS will be entirely coded in 6502 code, and built specifically for Hacker's Edge.
    4. KERNEL.SYS will be responsible for loading the Shell program, which the player will finally interact with.

    There will be extensions to the 6502, not the op-codes or the assembly language. The extensions will all be present as either hardware interrupts(BRK) or through memory mapped I/O channels. Currently the Hacker's Edge TTY device is mapped to a memory address, any data written to this address is send out to the player as ASCII text. Since we are working with a TTY here, and not a framebuffer, there's no memory mapped video or text buffers available. All output to the player is done via a TTY device, AKA a serial link. Input from the player is done in a similar fashion, where keys the player enters are placed into a keyboard buffer, which then the 6502 code can read and parse.

    Other memory mapped I/O will include the ability to read from the virtual Hacker's Edge host's block device, and then there will networking I/O, which will allow the 6502 code to communicate with other in-game hosts. With all these devices combined, a complete working virtual operating environment based on real 6502 code can be created in Hacker's Edge!

    I originally wrote my own custom assembler and Virtual machine called Simple CPU simulator, which was used on a previous incarnation of Hacker's Edge. However, coding for this would require a bit of a learning curve for anybody, excluding myself... With the 6502, there is a vast amount of online documentation available, and lots of people still alive that know how to use the 6502 assembly language to make programs. The 6502 is also considered one of the easier microprocessors to code assembly on as well, making the overall learning curve for new players low.

    Hacker's Edge will hopefully be the epiphany of User-generated content, where players will be completely free to write any 6502 compatible assembly code and easily have a way to display those creations to the world. I know that people could basically just download and run a 6502 assembler and simulator/emulator on their local machine... However, the one big downside is that not many 6502 emulators are connected to a vast virtual Internet full of other 6502 hosts. What players will be-able to do in Hacker's Edge with a 6502 will just not be possible on a stand-alone configuration, and most definitely not as fun and entertaining. Here are some things I anticipate players to do with their virtual 6502 hosts:

    • Compile a custom KERNEL/Operating system
    • Build their own in-game hacking tools, to attack other in-game hosts
    • Create in-game games, which then other players can play
    • Ummm, perhaps create a multi-player game in the game itself... Like a MUD, or BBS doors
    • Emulate real-world TCP protocols in the game with 6502
    • Host in-game BBS systems, which can be used to say hide it from the real world Internet...
    • Develop viruses and worms to infect the virtual 6502 Internet of Things.
    • Attempt to build a Python interpreter in 6502 assembly(Would this even be feasible?) I've see LISP on the original Apple 2, and Python for the Apple 2GS model.

    As you could imagine, the possibilities will be endless, as countless programs were made on the VIC-20, Atari 800, Apple ][, Commodore Pet, and the Nintendo Entertainment System. After some research, it seems that the 6502 is still being widely used in embedded projects in it's CMOS form. The 6502 core, once fully integrated in Hacker's Edge should prove to the most unique part of the Hacker's Edge community.

  • Introducing PyCARS!

    Posted on Sunday, March 20th, 2016
    0 comments

    I recently purchased a second Raspberry Pi, the model 2, and a 7" touch screen. Originally I was going to use it to build a home automation HUD which I could place either on my wall or in my living room somewhere. However, after attaching the LCD to the Pi, I found it a bit too bulky for such an elegant task... My idea now is to build it out into a bedside HUD/alarm clock system. Of course, this HUD system will be built in the Python language, and so I am here to share my current implementation, and a progress report with my fellow Python peers.

    During the development, I needed to figure out some sort of UX/UI to use for this, and since I have a good passion for the Star Trek series, I thought using their LCARS design ideas would be a good base to start with. Currently, there really is no design or anything trek related(besides the sound effects) in the project. However, as the project matures, I am hoping to develop it into more of an app platform, where I can easily extend the functionality for new use-cases. Currently, it can act as a basic clock with a huge display. The display itself can be dimmed easily, and can also be completely turned off when not in use or when your asleep. There is a basic alarm system in place that cannot be configured without updating the Python code. I also built in some hue control functionality into it, so that I can easily control my bedroom and master bathroom lights with ease.

    To keep this open source project from copyright and legal issues, no assets, such as images or sound files will be included alongside the source code. Also, the project will never make mention to LCARS, only PyCARS, which isn't copyrighted or trademarked by CBS Interactive. CBS has been known to take down software with LCARS interfaces, such as the Tricorder app for Android. Although, they don't seem to take down every LCARS related fan project, which seems very weird... I am unsure why they target some projects, and leave others alone.

    You can view and download the source code repository from my BitBucket page PyCARS

    Since this page might be noticed by a CBS interactive lawyer one day, here are some other LCARS related projects which have been around for a lot longer, which legal action has not been taken yet:

    Just in case: Star Trek, Star Trek: The Next Generation are trademarks of PARAMOUNT PICTURES and CBS. In NO CASE is use of any copyrighted material, or any related, derived or inferred ideas intended as a claim of ownership to those copyrights/trademarks. I am in NO way associated with Paramount Pictures or CBS.

    Furthermore, I will not be making any money, nor selling any material copyrighted by CBS Interactive. This project will remain open source, under a license I still need to consider. This Python project will also never be distributed with copyrighted material, such as images and sounds created by CBS Interactive. If the end-user using this Python project so rightfully chooses to violate CBS Interactive copyright and trademarks and uses LCARS images, fonts, and sounds, the end-user will remain fully liable by law. However, I highly doubt CBS interactive will pursue a John Doe for using LCARS images and sounds in the personal confines on his own home. However, if this project ends up being used by a larger corporation for UI/UX(although I highly doubt that will ever happen), it is their duty to provide whatever image, font, and sound assets suite their particular interface requirements. I am being as clear as possible here, that I am absolutely not distributing any CBS Interactive copyrighted or trademarked material with this open source Python project. CBS Interactive can clearly download the entire codebase and fully examine it for any copyrighted or trademarked data. However, since the LCARS system used in the show isn't really a real computer program doing stuff, and I highly doubt that the LCARS system for the TV/movies is made in Python... I should be-able to fully rest my case that this project is absolutely and definitely 100% legal. So take your best shot CBS Interactive, you've got nothing on this project.

  • Building a Telnet client in my PyGame VGA Console project

    Posted on Tuesday, January 27th, 2015

    I am sure most of you recall my post on the VGA compatible framebuffer display in PyGame I released. I am equally sure that you also heard of my open source hacking simulation engine I released yesterday. Well, this evening I thought I'd create a new example app for my PyGame VGA framebuffer project, this app is in the form of a basic telnet client which can talk to my hacking simulation engine server. This client will eventually be used in my Hacker's Edge massively multiplayer online hacking universe game, or MMOHUG for short. While the server for the game world will be closed source, the desktop client will be open sourced and available to anyone to modify and improve or use for their own similar game. Here is a screenshot of how it currently looks:

    Here is how simple it was to create such a program using my powerful VGA framebuffer console for PyGame:

    import asynchat, sys, asyncore, socket, threading, time
    import vgaconsole, cmd, pygame, sys, shlex, subprocess
    
    class TelnetClient(asynchat.async_chat):
        def __init__(self, console):
            asynchat.async_chat.__init__(self)
            self.console = console
            self.set_terminator(None)
            self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
            console.stdio.write('Connecting to Hacker\'s Edge...')
        def handle_connect(self):
            self.console.stdio.write('connected!\n')
        def handle_close(self):
            self.console.stdio.write('Disconnected.\n')
            self.close()
        def transmit(self, data):
            self.push(data+'\r\n')
        def process_iac(self, iac):
            if iac[2] == chr(1):
                if iac[1] == chr(251):
                    self.console.mask_input = '*'
                elif iac[1] == chr(252):
                    self.console.mask_input = None            
        def collect_incoming_data(self, data):
            if chr(255) in data:
                iac = data.index(chr(255))
                self.process_iac(data[iac:iac+3])
                if iac == 0:
                    data = data[3:]
                else:
                    data = data[:iac]+data[iac+3:]
            self.console.stdio.write(data.replace('\r\n', '\n'))
    
    class ConsoleApp(vgaconsole.VGAConsole):
        cursor_klass = vgaconsole.AnimatedCursor
        def init(self):
            self.set_color(2,0)
            self.telnet = TelnetClient(self)
        def parse(self):
            self.stdio.write('\n')
            line = self.stdio.read()
            self.telnet.transmit(line)
    
    def main():
        pygame.display.init()
        screen = pygame.display.set_mode((640,400),0,8)
        pygame.display.set_caption("Hacker's Edge Single Player client")
        vga = ConsoleApp(screen)
        vga.init()
        vga.draw()
        pygame.display.update()
        try:
            vga.telnet.connect(('hackers-edge.com', 4000))
            client_thread = threading.Thread(target=asyncore.loop, kwargs={'timeout':1})
            client_thread.start()
        except socket.gaierror, e:
            vga.stdio.write('failed!\n%s' % e)
        while True:
            vgaconsole.clock.tick(30)
            events = pygame.event.get()
            for e in events:
                if e.type == vgaconsole.QUIT:
                    vga.telnet.close()
                    client_thread.join()
                    pygame.quit()
                    sys.exit()
                if vga.telnet.connected:
                    if e.type == vgaconsole.KEYDOWN and e.key == 13:
                        vga.parse()
                    else:
                        vga.handle_event(e)
            try:
                vga.draw()
            except:
                print sys.exc_info()
                pass
            pygame.display.update()
    
    if __name__ == '__main__':
        main()
    

    Isn't that easy? I hopefully made this VGA console easy to program and integrate into mostly any Python app that needs a cross-platform SDL console available. It's available via my Bitbucket page like every other open source project I have.

  • Open source hacking simulation game engine

    Posted on Sunday, January 25th, 2015

    Did you ever want to create your own text-based hacking simulation game that can run over telnet or in a web browser to share with friends? Now you can! Introducing an easy to use and expand open source hacking simulation game engine, the first of kind(I think). I licenses the game engine itself under the GNU Public License 2, so that many unique games can be made from the engine, while making every game made forkable. So... You release a new hacking simulation game, and another guy wants to expand on your ideas and make an even better experience. Your game can be easily forked, and enhanced by the next person. This process can continue until the Internet is littered with hundreds of cool micro hacking simulation puzzle games. Since it runs through telnet and uses very little resources, it's easy to place in any environment and forget about it. Here is the current example game that ships with the engine:

    The server located here will only accept at maximum of 5 concurrent connections, so if you experience a connection issue, try again after a couple minutes once the last person finished. You can also connect to the game via telnet: [hackers-edge.com:4000].

    As with the rest of my open source projects, this one is also located on my Bitbucket page: Hack RUN Open Source engine

    The source code is also a nice demonstration of how to build a functional telnet server using the standard asynchat module. So, if you were ever curious about this module or think twisted might be a bit too much, then give asynchat a shot, it's really nice to work with once you get to know it. I built this entire engine in a single days work.

  • Introducing VGA Console for Pygame

    Posted on Sunday, December 21st, 2014

    I just finished my initial work on a VGA Console emulator for Pygame. This is great, if say you need to add a retro style VGA Console into your game, or if you want to build an application that takes full advantage of the VGA video adapter. At the moment, I am releasing it as a Preview Demo, and it isn't ready for any production applications or games just yet. However, what is available in this preview is plentiful. You can download the full source from my Bitbucket page, which is linked in the right hand column of my blog. I will also include the source here and try to explain what it can currently so. I have also enclosed a screenshot of the demo app both in this blog post and in the bitbucket repository for those too lazy to download it and try it themselves in Pygame.

    import pygame, sys
    from pygame.locals import *
    import mmap
    
    clock = pygame.time.Clock()
    
    class VGAConsole(object):
        VGA_PALETTE = (
            (0,0,0),
            (0,0,168),
            (0,168,0),
            (0,168,168),
            (168,0,0),
            (168,0,168),
            (168,87,0),
            (168,168,168),
            (87,87,87),
            (87,87,255),
            (87,255,87),
            (87,255,255),
            (255,87,87),
            (255,87,255),
            (255,255,87),
            (255,255,255),
        )
        US_SHIFTMAP = {
            49: 33,
            50: 64,
            51: 35,
            52: 36,
            53: 37,
            54: 94,
            55: 38,
            56: 42,
            57: 40,
            48: 41,
            96: 126,
            45: 95,
            61: 43,
            91: 123,
            93: 125,
            59: 58,
            39: 34,
            92: 124,
            44: 60,
            46: 62,
            47: 63,
        }
        def __init__(self):
            self.vgabuf = mmap.mmap(-1, 4000)
            pygame.display.init()
            pygame.font.init()
            self.screen = pygame.display.set_mode((640,400),0,8)
            pygame.display.set_caption('VGA Console test')
            self.font = pygame.font.Font('VGA.ttf', 16)
            self.cursor = self.font.render(chr(219),0,self.VGA_PALETTE[7])
            pygame.mouse.set_visible(False)
            self.pos = [0,0]
            self.background = self.VGA_PALETTE[1]
            self.shift = False
            self.cframe = 0
            self.cframes = []
            for c in ('|', '/', '-', '\\',):
                self.cframes.append(self.font.render(c,0,self.VGA_PALETTE[15],self.background))
        def draw(self):
            self.screen.fill(self.background)
            self.vgabuf.seek(0)
            for y in range(0,25):
                for x in range(0,80):
                    attr = ord(self.vgabuf.read_byte())
                    fg,bg = 7,0
                    if attr > 0:
                        fg,bg = attr&0xf, (attr&0xf0)>>4
                    c = self.vgabuf.read_byte()
                    if ord(c) > 0:
                        self.screen.blit(self.font.render(c,0,self.VGA_PALETTE[fg],self.VGA_PALETTE[bg]), (x*8,y*16))
            self.drawMouse()
            self.drawCursor()
            pygame.display.update()
        def setXY(self, row, col, c):
            self.vgabuf.seek((80*row+col)*2)
            self.vgabuf.write(chr(0x1f)+chr(c))
        def type(self, c):
            if c == 13:
                self.pos[1] = 0
                self.pos[0] +=1
            elif c == 8:
                if self.pos[1] > 0:
                    self.pos[1] -=1
                    self.setXY(self.pos[0], self.pos[1], 0)
            elif c == 9:
                self.pos[1] += 8
            elif c == 27:
                pygame.quit()
                sys.exit()
            else:
                self.setXY(self.pos[0], self.pos[1], c)
                self.pos[1] +=1
            if self.pos[1] > 80:
                self.pos[1] = 0
                self.pos[0] += 1
        def write(self, text):
            for c in text:
                self.type(ord(c))
        def draw_ascii(self):
            row, col = 10,10
            for c in range(0,255):
                self.setXY(row,col,c)
                col +=1
                if col > 41:
                    col = 10
                    row+=1
        def draw_window(self, row, col, height, width, title=None):
            self.setPos(row, col)
            brd = chr(205)*(width-1)
            self.write(chr(213)+brd+chr(184))
            for y in range(row+1, row+height):
                self.setXY(y, col, 179)
                self.setXY(y, col+width, 179)
            self.setPos(row+height, col)
            self.write(chr(212)+brd+chr(190))
            if title:
                self.setPos(row, col+((width/2)-len(title)/2))
                self.write(title)
        def clear_window(self, row, col, height, width):
            for y in range(row, row+height+1):
                self.setPos(y, col)
                self.write(chr(0)*(width+1))
        def setPos(self, row, col):
            self.pos = [row, col]
        def clearScreen(self):
            self.vgabuf.seek(0)
            self.vgabuf.write(chr(0)*4000)
            self.setPos(0, 0)
        def mousePos(self):
            x,y = pygame.mouse.get_pos()
            return (y/16, x/8)
        def drawMouse(self):
            row,col = self.mousePos()
            self.screen.blit(self.cursor, (col*8,row*16))
        def drawCursor(self):
            self.screen.blit(self.cframes[self.cframe/3%4], (self.pos[1]*8,self.pos[0]*16))
            self.cframe+=1
        def main(self):
            self.draw_ascii()
            self.draw_window(9,9,9,33, ' ASCII ')
            self.setPos(0, 0)
            self.write('Welcome to VGAConsole!\rC:\>')
            self.draw()
            #pygame.event.set_blocked(MOUSEMOTION)
            while 1:
                clock.tick(30)
                for event in pygame.event.get():
                    if event.type == QUIT:
                        pygame.quit()
                        sys.exit()
                    elif event.type == MOUSEBUTTONUP:
                        oldpos = self.pos
                        self.clear_window(9, 9, 9, 33)
                        self.pos = oldpos
                    elif event.type == KEYDOWN:
                        if event.key == K_LSHIFT or event.key == K_RSHIFT:
                            self.shift = True
                        if event.key > 0 and event.key < 256:
                            c = event.key
                            if self.shift:
                                if c > 96 and c < 123:
                                    c-=32
                                elif c in self.US_SHIFTMAP.keys():
                                    c = self.US_SHIFTMAP[c]
                            self.type(c)
                    elif event.type == KEYUP:
                        if event.key == K_LSHIFT or event.key == K_RSHIFT:
                            self.shift = False
                self.draw()
    
    VGAConsole().main()
    

    As you can see from the screenshot, it's very complete looking, with full support for all those ASCII characters you knew and loved. The display itself is rendered from the same format as the VGA memory, where each character on screen takes 2 bytes. One byte stores some attribute information, such as the foreground and background colors, and the other byte is the ASCII character code. In the actual VGA memory, the attribute also supported a blink flag, which I omitted in this VGA Console, as it's annoying and I really don't want to see it used... So, rather than supporting blink, it supports a full 16 colors for both the foreground and the background. In the original VGA memory, it only supported 8-color backgrounds. Since the memory format for VGA is the same as located at &0xB800 memory segment on real hardware, you can technically use a memory dump file created by say BSAVE commands. I may enable native BSAVEing and BLOADing of the memory buffer in future version.

    During the draw() function, the VGA memory space is read and a surface is blitted. Currently, this surface is a display Surface object, but in the next release this will change to render to a normal Surface so that it can be blitted by the developer onto whatever Surface or display they want. This may cause issues with the mouse emulation however, I will need to look into that and see how the Pygame mouse works with Surfaces. So, yes, mouse emulation is also present. It displays a familiar block cursor we all used to remember. There is an API to grab it's location on the text display, so that you can determine if a text object or widget has been clicked.

    One of the major features which is missing is input and output buffering support. At the moment, the current API allows you to write directly to the VGA memory buffer to draw text onto the console. This isn't very ideal for several reasons. It will not work correctly with stdin/stdout applications, and taking input into a string isn't currently possible. Typing onto the console is currently possible, but the input is not buffered, and the user may edit anything on the screen. Proper stdin and stdout is supported in the next release. This will enable many standard Python functions which require either or both to actually work on the VGA console. This will really open the doors to what will be possible with VGA Console.

  • Planet Python now available on Gopher

    Posted on Saturday, December 13th, 2014

    In additional efforts to revive the yet to be dead Gopher protocol. I have created a Python script which I can either run manually or by using a cron job, which aggregates RSS feeds into a Gopher friendly format. This can enable browsers such as Lynx to easily read RSS feeds without requiring a separate RSS reader or by downloading huge amounts of data over HTTP. What more, is that all the articles it aggregates are all archived in Gopherspace, so they will always be available even if the blog hosting it dies. This is the advantage is not using a live feed. This act alone will immensely increase the overall content and information available over the Gopher protocol. Each day, there are more than 10 new Python articles. All these articles will also soon be indexed by the Gopher search server Veronica, adding to it's overall knowledge of information.

    Adding new RSS feeds is as simple as updating a standard Python list, so as a result, I also added other Planet feeds, such as Debian and KDE. Over time, I will add more interesting feeds that Gopher users would like to be-able to read from the comfort of a lightweight text-based browser. I have not yet decided if I will open source the script or not. I'd personally like to clean it up before the public eye sees it.

    For the moment, the script is run as a cron job once daily early in the morning hours, so while you are drinking your coffee and eating breakfast, you can open up Lynx or Overbite on your Android phone to quickly catch up on the latest Python, Debian, and KDE news.

    I will provide both the Gopher link and a link that goes through my own proxy for your convenience:

  • Building a Gopher client in Django

    Posted on Friday, December 12th, 2014

    Have you ever wanted to enable your users to access your Gopher server without the need of them installing a browser plugin, installing a standalone software, or using a third part proxy service? Now you can, by adding this simple Django app to any existing Django project. It doesn't depend on anything by my updated gopherlib.py, and the following app which I will explain here. I have already developed it and deployed it, so I will not be covering every single detail. For the most part, it's a very basic Django app that only uses function-based views to get the job done. So, this app might be good as a starting point for people new to Django, as there's nothing terribly complex going on here. You can see a fully working example here. First let's cover the urls.py:

    from django.conf.urls import patterns, url
    
    urlpatterns = patterns('djgopher.views',
        url(r'^$', 'root_menu', name='gopher_root_menu'),
        url(r'^RemoteDoc$', 'remote_document', name='gopher_remote'),
        url(r'^(?P<selector>.+)$', 'selector', name='gopher_selector'),
    )
    

    This isn't the most elegant way to grab the selector from the URL, but it gets the job done. This may have some security implications behind it, I am hoping that my code which handles this selector can't be easily exploited. Feel free to point out to me any possible security concerns and a way to fix it. Let's move onto the views.py:

    from djgopher.gopherlib import Gopher, GopherMenu, TextFile
    from django.shortcuts import render
    from django.http import HttpResponse, Http404, HttpResponseRedirect
    from PIL import Image
    from StringIO import StringIO
    
    def root_menu(req):
        g = Gopher('gopher.veroneau.net')
        menu = g.get_root_menu()
        return render(req, 'djgopher/menu.html', {'menu':menu.get_data(), 'title':'Root Menu'})
    

    Here is the first function based view, it's only task is to display the initial gopher menu to the user. I did hardcode my own Gopher server here, but for most use cases, you can actually put in 127.0.0.1 to refer to the local Gopher server. Here, we initialize the Gopher class, and obtain the root menu of that server. Then we send it off to a template which we will get into shortly.

    def selector(req, selector):
        if req.META['QUERY_STRING'] != '':
            selector += '?'+req.META['QUERY_STRING']
        gtype = selector[0]
        query = req.POST.get('query', None)
        if gtype == '7' and query is None:
            return render(req, 'djgopher/search.html', {'selector': selector, 'title':selector[1:]})
        g = Gopher('gopher.veroneau.net')
        data = g.get_selector(selector, query)
        if isinstance(data, GopherMenu):
            return render(req, 'djgopher/menu.html', {'menu':data.get_data(), 'title':selector[1:]})
        elif isinstance(data, TextFile):
            return HttpResponse('%s' % data, mimetype='text/plain')
        elif gtype == 'I':
            typ = selector[-3:].lower()
            if typ not in ('png', 'jpg', 'jpeg'):
                typ = Image.open(StringIO(data)).format.lower()
            return HttpResponse(data, mimetype='image/%s' % typ)
        elif gtype == 'g':
            return HttpResponse(data, mimetype='image/gif')
        elif gtype == '9':
            return HttpResponse(data, mimetype='application/octet-stream')
        else:
            return HttpResponse('Unhandled.', mimetype='text/plain')
    

    This function based view handles the actual rendering of the chosen selector. Here we check for a query string and append it to the selector so that the Gopher server can manage it. Then we check if the item type is a search, if so, then we display a special template if no query came in. This template is a basic form that will prompt the user for their search query. Finally, if we're still processing here, we grab our selector from the server and check what type we are handling. Technically, we should check if we support the type before even downloading any data to save bandwidth, but for now this should be fine. I believe I will update it shortly so that the item type is properly checked beforehand. Depending on the item type, we either display a menu, a text file, image, or perform a binary download. We also use the PIL library if the image file doesn't have a valid extension to make sure we serve the correct information to the browser.

    def remote_document(req):
        url = req.GET.get('uri', None)
        if url is None:
            raise Http404
        hostname, path = url.split('/',1)
        host, port = hostname.split(':')
        if path[0] != '0':
            return HttpResponseRedirect('http://gopher.floodgap.com/gopher/gw?gopher://%s:%s/%s' % (host, port, selector))
        g = Gopher(host, int(port))
        data = g.get_selector(path)
        if isinstance(data, TextFile):
            return HttpResponse('%s' % data, mimetype='text/plain')
        else:
            return HttpResponseRedirect('http://gopher.floodgap.com/gopher/gw?gopher://%s:%s/%s' % (host, port, selector))
    

    This final function based view is to enable users to retrieve remote documents, such as this one here. I only want to enable text document linking, not anything else to limit bandwidth and overall complexity. There are other clients and proxies out there that can easily grab resources from any Gopher server, so it's best to leverage those services for this.

    The Templates

    The templates are pretty dead simple, as this is a very basic client application, nothing too fancy is required to display Gopher content. Here is our base.html template:

    <html>
      <head><title>{{title}} | Django Gopher Client</title></head>
      <body>
        {{title}} | Django Gopher Client<hr/>
        {% block content %}{% endblock %}
      </body>
    </html>
    

    See? Nothing too complex, not even a stylesheet, although feel free to skin your own client all you want. Here is the template which renders a Gopher menu for us in HTML:

    {% extends "djgopher/base.html" %}
    {% load gopher %}
    
    {% block content %}
    <pre>{% for item in menu %}{% if item.type == 'i' or item.type == '3' %}{{item.type|icon:STATIC_URL}}{{item.name}}
    {% else %}<a href="{{item|selector}}">{{item.type|icon:STATIC_URL}}{{item.name}}</a>
    {% endif %}{% endfor %}</pre>
    {% endblock %}
    

    We are using pre tags here to ensure the content in the menu is rendered correctly. Lots of Gopherholes, including my own use ASCII art in the menus. Gopherpedia is another example of this. This template uses some custom template filters to enable us to dynamically place in URLs and icons using Python code. Like they say, the template is not for logic! Our final template is the search.html template:

    {% extends "djgopher/base.html" %}
    
    {% block content %}
    <form action="{% url gopher_selector selector %}" method="post">{% csrf_token %}
    Query: <input type="text" name="query"/><input type="submit" value="Go"/>
    </form>
    {% endblock %}
    

    The search template is pretty normal, just a basic HTML post form. Now, finally here is the template tag library to render those filters:

    from django import template
    from django.core.urlresolvers import reverse
    from django.utils.safestring import mark_safe
    
    register = template.Library()
    
    ICON_TYPES = {
        '0': 'text.png',
        '1': 'dir.png',
        'h': 'html.png',
        '9': 'binary.png',
        'I': 'image.png',
        'g': 'image.png',
        '7': 'search.png',
        '3': 'error.png',
        '8': 'telnet.png',
    }
    
    @register.filter
    def selector(value):
        if value['selector'][:5] == 'hURL:':
            return value['selector'][5:]
        if value['host'] != 'gopher.veroneau.net':
            uri = '%s:%s/%s' % (value['host'], value['port'], value['selector'])
            if value['type'] == '0':
                return '%s?uri=%s' % (reverse('gopher_remote'), uri)
            elif value['type'] == '8':
                return 'telnet://%s:%s/' % (value['host'], value['port'])
            else:
                return 'http://gopher.floodgap.com/gopher/gw?gopher://%s' % uri
        return reverse('gopher_selector', args=[value['selector']])
    
    @register.filter
    def icon(value, static_url):
        try:
            return mark_safe('<img src="%sdjgopher/%s"/>' % (static_url, ICON_TYPES[value]))
        except KeyError:
            return ''
    

    The first dictionary you see there is to make selecting the icon like a case-select, which is much more efficient than a ton of if statements. The first filter here, selector is used to resolve a menu item into a URL that WWW browsers can understand. Here, we check various attributes based on the Gopher standard and write an HTTP compatible URL link the user can click on. Anything from the supported Gopher server is handled internally by our selector view. Anything that has an external host is handled according to it's type. This makes it so that all Gopher menu items work regardless if the local client supports remote servers. Finally, we have the simple icon filter that does nothing more by write out a proper IMG tag with the icon for the menu item.

    And there you have it, a fully functional Gopher client all done from within a Django app! So now you have no excuse not to run your own Gopher server. For some interesting places to visit on Gopher, check out my Bookmarks.

    As mentioned in my previous post about Gopher, I will soon be making all my Python Diary content available in Gopherspace, it will contain both full HTML versions and text versions. So, if you ever wanted to try browsing a less distracting side of the Internet where advertisers and trackers can't find you, now is a great time! Over time, I am planning on developing many gateway applications for the Internets most popular online locations, such as Twitter, and various news outlets. I think I'd personally like to add RSS feeds to my Gopherspace to enable the easy reading of many popular Python blogs from Planet Python. If your on Android, be sure to check out the updated Gopher client, which can really save you on your mobile bandwidth!

  • Developing an XBMC add-on in Python

    Posted on Thursday, December 11th, 2014

    XBMC is one awesome piece of media center software that runs on all popular platforms. What's more, is that it's entirely programmable via Python. I have released a couple XBMC related Python tools and videos earlier this year, and now I wanted to dive into add-on development. The add-on I will show you today takes advantage of my gopherlib.py module. This add-on will soon be submitted to the Official XBMC Addon repository in the coming weeks for all to enjoy. Here is how the add-on looks when it's running in XBMC:

    As I mentioned in my last post about Gopher, XBMC follows the Gopher protocol pretty closely, so it was pretty easy to create an XBMC Gopher client. Here is the code which I will explain afterwords:

    import sys, xbmc, xbmcgui, xbmcplugin, urlparse, os
    from gopherlib import Gopher, GopherMenu, TextFile
    
    BASE_URL = sys.argv[0]
    addon_handle = int(sys.argv[1])
    
    xbmc.log('HANDLE: %s' % addon_handle, xbmc.LOGNOTICE)
    
    def add_directory(title, url):
        li = xbmcgui.ListItem(title, iconImage='DefaultFolder.png')
        xbmcplugin.addDirectoryItem(handle=addon_handle, url=url,
                                    listitem=li, isFolder=True)
    
    def add_text_file(title, url):
        li = xbmcgui.ListItem(title, iconImage='DefaultVideo.png')
        xbmcplugin.addDirectoryItem(handle=addon_handle, url=url,
                                    listitem=li)
    
    def add_image_file(title, url):
        li = xbmcgui.ListItem(title, iconImage='DefaultPicture.png', path=url)
        #li.setProperty('isPlayable', 'true')
        xbmcplugin.addDirectoryItem(handle=addon_handle, url=url,
                                    listitem=li)
    
    def add_info(info):
        li = xbmcgui.ListItem(info, iconImage='DefaultVideo.png')
        xbmcplugin.addDirectoryItem(handle=addon_handle, url='/', listitem=li)
    
    def text_viewer(title, text):
        WINDOW = 10147 # Text Viewer window in XBMC
        TITLEBAR = 1
        CONTENTS = 5
        xbmc.executebuiltin("ActivateWindow(%d)" % WINDOW)
        w = xbmcgui.Window(WINDOW)
        xbmc.sleep(500)
        w.getControl(TITLEBAR).setLabel(title)
        w.getControl(CONTENTS).setText(text)
    
    def image_viewer(img):
        w = xbmcgui.Window()
        w.addControl(xbmcgui.ControlImage(0,0,w.getWidth(), w.getHeight(), img, 2))
        w.doModal()
        del w
    
    xbmc.log('BASE URL: %s' % BASE_URL, xbmc.LOGNOTICE)
    xbmc.log('Arguments: %s' % sys.argv[2], xbmc.LOGNOTICE)
    
    scheme, remainder = BASE_URL.split(':', 1)
    parts = urlparse.urlparse(remainder)
    netloc, path, query = (parts[1], parts[2], parts[4])
    args = urlparse.parse_qs(query)
    
    query = args.get('query', None)
    
    xbmc.log('Path: %s' % path, xbmc.LOGNOTICE)
    
    if path == '/':
        path = '/gopher.floodgap.com:70/1'
    
    hostname, selector = path[1:].split('/', 1)
    host, port = hostname.split(':')
    
    g = Gopher(host, int(port))
    
    if selector[0] == '7' and query is None:
        keyboard = xbmc.Keyboard('', 'Input Required', False)
        keyboard.doModal()
        if keyboard.isConfirmed():
            query = keyboard.getText()
    
    data = g.get_selector(selector+sys.argv[2], query)
    
    if isinstance(data, GopherMenu):
        for item in data.get_data():
            url = 'plugin://plugin.picture.gopher/%s:%s/%s' % (item['host'], item['port'], item['selector'])
            if item['type'] == '0':
                add_text_file(item['name'], url)
            elif item['type'] == '1':
                add_directory(item['name'], url)
            elif item['type'] == 'i':
                add_info(item['name'])
            elif item['type'] == '7':
                add_directory(item['name'], url)
            elif item['type'] == 'I':
                add_image_file(item['name'], url)
        xbmcplugin.endOfDirectory(addon_handle)
    elif isinstance(data, TextFile):
        text_viewer(selector, '%s' % data)
    elif selector[0] == 'I':
        filename = selector.split('/')[-1]
        temp = xbmc.translatePath('special://temp/')
        open(temp+filename,'w').write(data)
        image_viewer(temp+filename)
        os.unlink(temp+filename)
    

    This code should be fairly easy to follow, as it basically goes from top to bottom, with a function call here and there. The first 4 functions are for generating the correct directory entries within the XBMC GUI. After, you will see 2 specialized functions used to display textual data, and another to display an image. I was unable to find a more elegant way to display resolved image URLs in XBMC. I tried using many different XBMC functions, but xbmcplugin.setResolvedUrl just wouldn't work with image files... When I implement video and audio support in the future, I will try it again. I may need to see if I can create a new VFS module for XBMC, so that the gopher:// protocol is natively supported for streaming.

    The next section here takes the arguments from XBMC and turns them into a valid Gopher URL which we can then send to my gopherlib module. Once that is out of the way, we can instantiate our Gopher class. It is here that we check if the selector is a search selector and bring up the on-screen keyboard for the user if needed. Finally we get our selector and the resulting data object.

    The final if structure here is used to either display an XBMC directory, a text file, or an image to the user.

    You can view many more screenshots and even download the prepackaged add-on in ZIP format from my Gopher server or use the Gopher proxy.

    On a side note, I also updated the Android Gopher client, since it's licensed under the BSD license, the source was available. The update now brings a more modern look and feel on newer phones and tablets, and now included a working address bar. You can grab the APK from my Gopher or Gopher proxy and side load today!

    One added benefit coming soon to Gopher users, will be the ability to browse and search all of the content on Python Diary from the comfort of Gopher. I will be developing a basic Gopher application which will be-able to access the current Python Diary database and query it. So, Python Diary will soon be available on XBMC I guess. So keep a look out for that update when it arrives.

  • Remember the Gopher protocol?

    Posted on Wednesday, December 10th, 2014

    Depending on how old you are and if you were on the Internet before the World Wide Web became the norm, you may not have heard of a rather awesome Internet protocol called Gopher. I, unfortunately never had the chance to experience Gopher in the day it ruled the information highway(at this point in time the highway wasn't super). Late last year I finally managed to into gopher and experience it for what it is. It is a rather awesome system, very easy to navigate, and is basically pick-up and play. It works using a very simple menu system concept, and every gopherhole as they are called, works the exact same. It is sort of similar to say the FTP protocol, but more descriptive and can actually run server-side scripts to perform database look-ups and such. In fact, the main search engine, which is basically the Google of Gopherspace called Veronica is still active and can search of the current Gopher spaces and any archived servers from days past.

    The reason I am bringing this up on my Python Blog, is that there are some new Gopher servers actually written in Python, and Python can obviously be used to create server-side scripts. However, this isn't the main reason why I bring this up. I bring this up on my Python blog because of a little program called XBMC. If anybody reading this recalls Gopher or has used gopher recently, you may notice a similarity to XBMC apps, where all XBMC apps are all consistent and look and function the exact same way. Each XBMC app has a root menu, and then many virtual sub directories generated by the XBMC VFS API in the apps code itself. Then, depending on what type of app you are using let it be a picture app, video app, or music app, the final content type to load will be of that type. This last tidbit is the least similar to Gopher, as gopher supports mixed content, where you can have a directory with images, music, and video files. I believe that XBMC would be a great place to have a gopher client on, since a gopher server can also act as a proxy to WWW content, it would be perfect. If you have ever seen Gopherpedia, then you will understand what I mean.

    Python used to support gopher in the standard library, but support has since been removed. I found the old standard library module and completely updated it to be more modern and along the lines of how current Python standard library modules function, and it's now class based. If your interested in checking out this module, you can obtain a copy on my Gopherhole, if your browser no longer has native gopher support you can use a gopher proxy server or configure Squid to proxy it for you. Here are both the Gopher and Gopher Proxy links. Originally, I was going to use these links to develop a more take on a Gopher client, but now I am thinking of creating an XBMC Gopher client, as I think that would be a really fun project. If you want to seamlessly explore Gopherspace, you should install the Firefox add-on or use Lynx. I once had a Lynx proxy running on my server, where you would be-able to use Lynx directly from within any HTML5 compatible web browser. I am either thinking of bringing this back or running a basic Gopher-WWW proxy, although Floodgap does have a really nice Gopher proxy server. If you want to check out my Gopherhole and see what content I have there, it can be accessed using Gopher or the Gopher Proxy. I have plenty of interesting and fun content available there.

  • Building a CARDIAC Assembler in Python

    Posted on Sunday, October 19th, 2014

    In my last article I gave a fully working example of a CARDIAC CPU simulator in Python. This article will continue down the path of creating a build toolchain for the Cardiac. Due to how the Cardiac accepts data from the outside world, namely in the form of a deck of cards or plain numbers, developing an assembler was a bit more difficult than a traditional bytecode assembler. At least when I build a binary assembler, I assemble the bytecodes in-memory. This ensures that all memory pointers are correctly pointing to the proper memory addresses, and also eliminates errors due to how information is stored. This is similar to how the MS-DOS DEBUG.EXE worked to build .COM files. It was simple and yet very effective. This tends to be the model I normally follow, as it makes debugging the virtual machine very easy. This is how I built up my simple-cpu project, and now I am in the process of enabling dynamic memory relocation to better enable the loading binary files. I also recently added a full memory map interface to it as well, so now memory mapped I/O is possible, and I plan on using that to allow my binaries to directly draw onto an SDL surface. Anyways, onward to the Cardiac Assembler, as that is the real reason your here today!

    As of this post, you can view all the source code for this post and the last one on my Python Experiments Bitbucket repo.

    The assembler is going to use just the standard Python library modules, so no fancy GNU Bison or YACC here. First, let's begin by defining what our language will look like, it's overall syntax. Here is deck1.txt turned into an Assembly program:

    # This deck1.txt converted over to ASM format.
    # Lines which start with a hash are comments.
    
    # Here we declare our variables in memory for use.
    var count = 0
    var i = 9
    
    # This is needed to add the special Cardiac bootstrap code.
    bootstrap
    
    # Program begins here.
    CLA
    STO $count
    label loop
    CLA $i
    TAC $endloop
    OUT $count
    CLA $count
    ADD
    STO $count
    CLA $i
    SUB
    STO $i
    JMP $loop
    label endloop
    HRS
    
    # End of program.
    end
    

    To make developing programs for the CARDIAC easier, the assembler is going to introduce the concept of variable storage, and labels. The first few lines here that begin with var are basically setting aside memory addresses to store these variables in question. The bootstrap statement is more of a Macro as it just writes out a basic bootstrap loader for use on CARDIAC. There is also an alternative bootloader, which uses a second stage boot program to load in your code. This has the advantage of being able to load in very large programs without needing a very large deck. I'd recommend playing around with both options when assembling your programs. The bootloader was custom built code made by me, here is an assembly listing of it:

    INP 2
    JMP 0
    
    INP 89
    INP 1
    INP 90
    CLA 89
    INP 91
    ADD
    INP 92
    STO 89
    INP 93
    CLA 98
    INP 94
    SUB
    INP 95
    STO 98
    INP 96
    TAC 3
    INP 97
    JMP 89
    INP 98
    INP 13
    INP 2
    JMP 89
    

    The bootloader is passed in the size of the deck to load, and loops over each card to load in your program, this means that your input program will no longer need to have lots of address markings. It can also be used to load in other data into a specific memory address. I am planning on updating the bootloader code to also act like a subroutine, so you could call it to load additional data into memory. Okay, so onto the actual Assembler now.

    from cmd import Cmd
    from cStringIO import StringIO
    import shlex, sys
    
    class AsmError(Exception):
        pass
    
    class Assembler(Cmd):
        """
        I am sure this could be done better with say GNU Bison or Yacc,
        but that's more complicated than needed for a simple assembler.
        """
        op_map = {
            'inp': 0,
            'cla': 1,
            'add': 2,
            'tac': 3,
            'sft': 4,
            'out': 5,
            'sto': 6,
            'sub': 7,
            'jmp': 8,
            'hrs': 9,
        }
        padding = '00'
        def configure(self):
            self.start = self.addr = None
            self.pc = 0 #: Allows us to keep track of the program pointer.
            self.var_map = {} #: This is used to keep track of variables.
            self.labels = {} #: Stores the label names, and where they point to.
            self.buffer = StringIO() #: This is our buffer where we will store the CARDIAC deck
            self.size = 0
        def emptyline(self):
            """ This is requried due to how the Python Cmd module works... """
            pass
        def unknown_command(self, line):
            self.stdout.write('*** Unknown syntax: %s\n'%line)
        @property
        def ptr(self):
            """ This will always give the proper pointer in the deck. """
            if self.addr:
                return self.addr
            return self.pc
        def write_cards(self, *card_list):
            """ Helper fuction to make life easier. """
            for card in card_list:
                self.buffer.write('%s\n' % card)
            self.pc += len(card_list) #: Increment the program pointer.
        def write_addr(self):
            """ This method will only write out the address if we're in that mode. """
            if self.addr:
                self.write_cards('0%s' % self.addr)
                self.addr +=1
            self.size +=1
    

    The first thing to notice here, is that the assembler is a subclass of Cmd, which is a command line parser. I choose to use a command-line parser and it makes things easier and can also be used interactively if I need to debug or later want to enable that feature. The first thing we declare in this class, is a dictionary called op_map, which maps our op codes to actual byte codes. This makes it efficient to write the output and also add new opcodes in the future. The first method here, configure() could have well been an override of __init__(), but I choose to just have it a separate method that I can call manually. The next 2 methods are specific to the command parser, and tell it how to handle empty lines and invalid commands. The ptr property added here is to make it transparent to the code the proper location in the deck at runtime. This is required due to how the Cardiac bootstraps code. You cannot just load in RAW code, you need to bootstrap your deck of cards, and the deck itself needs to keep track at which memory locations each of it's operations are being placed in. Since I offer 2 modes of assembly here, and also support RAW assembly, there are different ways the exact memory pointer is obtained for runtime variables and jumps. The next 2 methods are for writing specific card types, one being a normal list of cards, while the other being an address card. The address cards are only used if you assemble using bootstrap, the original Cardiac bootstrap program.

    Since this is a command processor Python subclass, next we are going to put in some commands that our source code can use to perform specific tasks:

        def do_exit(self, args):
            return True
        def do_bootstrap(self, args):
            """ Places some basic bootstrap code in. """
            self.addr = 10
            self.start = self.addr #: Updates all required address variables.
            self.write_cards('002', '800')
            self.pc = self.start
        def do_bootloader(self, args):
            """ Places a Cardiac compatible bootloader in. """
            self.write_cards('002', '800') 
            addr = 89 #: This is the start address of the bootloader code.
            for card in ('001', '189', '200', '689', '198', '700', '698', '301', '889', 'SIZE'):
                self.write_cards('0%s' % addr, card)
                addr+=1
            self.write_cards('002', '889')
            self.pc = 1
        def do_var(self, args):
            """ Creates a named variable reference in memory, a simple pointer. """
            s = shlex.split(args)
            if len(s) != 3 or s[1] != '=':
                raise AsmError('Incorrect format of the "var" statement.')
            if s[0] in self.var_map:
                raise AsmError('Variable has been declared twice!')
            value = int(s[2])
            self.var_map[s[0]] = '000'+str(value)
        def do_label(self, args):
            """ Creates a named label reference in memory, a simple pointer. """
            if args == '':
                raise AsmError('Incorrect format of the "label" statement.')
            if args in self.labels:
                raise AsmError('Label has been declared twice!')
            ptr = '00'+str(self.ptr)
            self.labels[args] = ptr[-2:]
    

    Okay, there's a fair amount to explain with this code. Firstly, the two first methods here, bootstrap and bootloader are obviously Cardiac specific, and it's doubtful that a homemade Assembler will even need such things. The bootstrap() method sets up some variables required for that mode of assembly, namely the address variables used to keep track which memory location we are at. We also write 2 basic cards, the bootstrap that Cardiac needs to load your program in a real or simulated Cardiac. The next method, bootloader is a bit more complex, it writes the same initial bootstrap required by all cardiac programs, but also writes out a second stage bootloader program which is responsible for loading in your program. Once I get to the guide about creating a compiler, the compiler will automatically choose which mode of assembly will be used based on the program being compiled.

    The next two very important methods, do_var, and do_label are responsible for controlling the variable and label system of the assembler. They make it easy to set a side a memory address for use as a variable, and to place in a label within your code so that you may JMP to it. If these didn't exist, you would be forced to keep track of memory addresses yourself, and if you update your program, you will also need to update the memory addresses... So, variables and labels keep track of all that for you, and the memory addresses are generated automatically during assembly time. Both of these methods should be fairly easy to understand what they are doing. They are storing the variable and label name into a hash along with the needed metadata.

        def default(self, line):
            """ This method is the actual lexical parser for the assembly. """
            if line.startswith('#'):
                return
            s = shlex.split(line)
            op, arg = s[0].lower(), '00'
            if len(s) == 2:
                if s[1].startswith('$'):
                    arg = '*%s' % s[1][1:]
                else:
                    arg = self.padding+s[1]
                    arg = arg[-2:]
            if op in self.op_map:
                self.write_addr()
                self.write_cards('%s%s' % (self.op_map[op], arg))
            else:
                self.unknown_command(line)
        def do_end(self, args):
            """ Finalizes your code. """
            for var in self.var_map:
                ptr = self.padding+str(self.ptr)
                self.write_addr()
                self.write_cards(self.var_map[var][-3:])
                self.labels[var] = ptr[-2:]
            if self.start:
                self.write_cards('002', '8%s' % self.start)
            self.buffer.seek(0)
            buf = StringIO()
            for card in self.buffer.readlines():
                if card[1] == '*': #: We have a label.
                    card = '%s%s\n' % (card[0], self.labels[card[2:-1]])
                elif card[:4] == 'SIZE':
                    card = '000'+str(self.size-1)
                    card = '%s\n' % card[-3:]
                buf.write(card)
            buf.seek(0)
            print ''.join(buf.readlines())
            return True
    

    Now onto the bread and butter of the assembler, these two methods here do most of the heavy lifting during the assembly process. First lets talk about default(), this method is called by the command-line parser when it cannot find the command the end-user attempted to call. Makes sense? So, here is where we place the code to check if any of those lines are assembly instructions and process them accordingly. First we filter out any comments, any line that starts with a hash is a commented line, simple as that. Next, we use the help of shlex module to parse the actual line. This module splits all the tokens into separate array elements, it even works with quotation marks, so take this line for example: echo "Example echo command" will result in only 2 array elements, the first being echo, and the second being the string. In the next line we set some needed defaults for Cardiac cards, so if there is a missing argument it will still work fine on the Cardiac. It is here that we also check the arguments to see if we are using any variables or labels and mark them accordingly in the buffer for the second pass.

    The last and final method here do_end is when you place end in your code at the very end. This method is essentially the last pass, and which generates the final assembled code. It gathers together all the variables, and generates a table of pointers to them. Then we write out the required cards to run the program the Cardiac just loaded, if we are using the original bootstrap. The bootloader doesn't use this, the bootloader manages that during runtime. Then we seek the buffer to the beginning and being our last pass through the assembled data. We check for any memory pointer references and fill in the holes as needed in the output code. If using the bootloader, it needs to be passed the size of the program code to be loaded, and this is where we fill that in so that the bootloader can locate the variable. Then finally, we display the assembled Cardiac code to standard output.

    You can find all the source for both the simulator and this assembler on my Python Experiments BitBucket page, along with an example deck that counts to 10 in both Assembly and Cardiac ready format. The code assembled by this assembler should run without error on either a real cardiac or a simulated one. You can find a simulator made in JavaScript by a Drexel University professor by following the links through this blog page here: Developing Upwards: CARDIAC: The Cardboard Computer.

    Hopefully this guide/article was helpful and inspiring to you, and I hope to have the final part of this guide/toolchain ready for consumption soon, the guide on how to build a Compiler for the Cardiac computer.

  • Building a CPU simulator in Python

    Posted on Wednesday, October 15th, 2014

    You can find the entire source listing from this article, plus more interesting code on my Python Experiments BitBucket page. Plus, the source code there will evolve and grow.

    Early this morning in my Planet Python feed, I read this really interesting article called Developing Upwards: CARDIAC: The Cardboard Computer, which is about this cardboard computer called the Cardiac. As some of my followers and readers might know, I have another project that I have been working on and evolving for the past months called simple-cpu, which I have released the source code for. I really should give that project a proper license so that others might feel interested in using it in their own projects... Anyways, hopefully I'll have that done by the time this publishes.

    After reading that article and the page it links to, I felt a bit inspired and decided to write my own simulator for it since I already have experience in writing bytecode engines. I plan on following this article up with another on how to write a Assembler for it, then shortly after how to write a compiler. So basically, throughout these articles, you will learn how to create a build toolchain for the Cardiac in Python. The simple-cpu I wrote already has a fully working Assembler, and in the game I am putting it in, already has the first steps of a working compiler. I choose to also use Cardiac as a demonstration machine due to it's pure simplicity. There are no really complex mnemonics here, each opcode only accepts a single parameter, so it's perfect as a learning tool. Furthermore, all data parameters are known to be the same, there is no need to detect if the program wants a register, a literal, or a memory address. In fact, there is really only one register, the accumulator. So lets get started. We will be making this class-based, so that the scope is contained and so that you can easily sub-class it to say add new opcodes if you want to feel adventurous. First, we will focus on the Initialization routines. This CPU is very simple, so we only need to initialize the following: CPU registers, opcodes, memory space, card reader/input, and the printer/tty/output.

    class Cardiac(object):
        """
        This class is the cardiac "CPU".
        """
        def __init__(self):
            self.init_cpu()
            self.reset()
            self.init_mem()
            self.init_reader()
            self.init_output()
        def reset(self):
            """
            This method resets the CPU's registers to their defaults.
            """
            self.pc = 0 #: Program Counter
            self.ir = 0 #: Instruction Register
            self.acc = 0 #: Accumulator
            self.running = False #: Are we running?
        def init_cpu(self):
            """
            This fancy method will automatically build a list of our opcodes into a hash.
            This enables us to build a typical case/select system in Python and also keeps
            things more DRY.  We could have also used the getattr during the process()
            method before, and wrapped it around a try/except block, but that looks
            a bit messy.  This keeps things clean and simple with a nice one-to-one
            call-map. 
            """
            self.__opcodes = {}
            classes = [self.__class__] #: This holds all the classes and base classes.
            while classes:
                cls = classes.pop() # Pop the classes stack and being
                if cls.__bases__: # Does this class have any base classes?
                    classes = classes + list(cls.__bases__)
                for name in dir(cls): # Lets iterate through the names.
                    if name[:7] == 'opcode_': # We only want opcodes here.
                        try:
                            opcode = int(name[7:])
                        except ValueError:
                            raise NameError('Opcodes must be numeric, invalid opcode: %s' % name[7:])
                        self.__opcodes.update({opcode:getattr(self, 'opcode_%s' % opcode)})
        def init_mem(self):
            """
            This method resets the Cardiac's memory space to all blank strings, as per Cardiac specs.
            """
            self.mem = ['' for i in range(0,100)]
            self.mem[0] = '001' #: The Cardiac bootstrap operation.
        def init_reader(self):
            """
            This method initializes the input reader.
            """
            self.reader = [] #: This variable can be accessed after initializing the class to provide input data.
        def init_output(self):
            """
            This method initializes the output deck/paper/printer/teletype/etc...
            """
            self.output = []
    

    Hopefully I left enough comments in the code above for you to understand exactly what's going on here. You might notice here a key difference from my simple-cpu project, is the method which opcodes are handled. I actually plan on incorporating this new way of detecting opcodes into that project in the near future. This method makes it easier for other developers of the library to extend it for their own requirements. As mentioned before, that project has been evolving as I started to understand more about how things should be done. In fact, taking on a CPU simulator project is a really incredible learning experience all on it's own. If you are truly a computer scientist, you should understand the under workings of a CPU, and how each opcode is processed at it's lowest level. Plus, developing and seeing a custom built CPU simulator made by your own imagination is a very gratifying experience. It's almost like giving birth, as this machine was entirely built by your mind alone, and seeing it work, is something magical.

    In the next part of this class, we will focus on the utility functions that we may need to call and use multiple times, these methods can also be overridden in subclasses to alter how the CPU actually functions:

        def read_deck(self, fname):
            """
            This method will read a list of instructions into the reader.
            """
            self.reader = [s.rstrip('\n') for s in open(fname, 'r').readlines()]
            self.reader.reverse()
        def fetch(self):
            """
            This method retrieves an instruction from memory address pointed to by the program pointer.
            Then we increment the program pointer.
            """
            self.ir = int(self.mem[self.pc])
            self.pc +=1
        def get_memint(self, data):
            """
            Since our memory storage is *string* based, like the real Cardiac, we need
            a reusable function to grab a integer from memory.  This method could be
            overridden if say a new memory type was implemented, say an mmap one.
            """
            return int(self.mem[data])
        def pad(self, data, length=3):
            """
            This function pads either an integer or a number in string format with
            zeros.  This is needed to replicate the exact behavior of the Cardiac.
            """
            orig = int(data)
            padding = '0'*length
            data = '%s%s' % (padding, abs(data))
            if orig < 0:
                return '-'+data[-length:]
            return data[-length:]
    

    These are the various utility functions, all of which might be overridden in a subclass. Later in this article, I will also provide an alternate source output which displays how this simulator can be built using Python Mixin classes, to make things even more pluggable. Finally, we find ourselves at the final part of code to get this simulator running, the actual processing methods:

        def process(self):
            """
            Process a single opcode from the current program counter.  This is
            normally called from the running loop, but can also be called
            manually to provide a "step-by-step" debugging interface, or
            to slow down execution using time.sleep().  This is the method
            that will also need to used if you build a TK/GTK/Qt/curses frontend
            to control execution in another thread of operation.
            """
            self.fetch()
            opcode, data = int(math.floor(self.ir / 100)), self.ir % 100
            self.__opcodes[opcode](data)
        def opcode_0(self, data):
            """ INPUT Operation """
            self.mem[data] = self.reader.pop()
        def opcode_1(self, data):
            """ Clear and Add Operation """
            self.acc = self.get_memint(data)
        def opcode_2(self, data):
            """ Add Operation """
            self.acc += self.get_memint(data)
        def opcode_3(self, data):
            """ Test Accumulator contents Operation """
            if self.acc < 0:
                self.pc = data
        def opcode_4(self, data):
            """ Shift operation """
            x,y = int(math.floor(data / 10)), int(data % 10)
            for i in range(0,x):
                self.acc = (self.acc * 10) % 10000
            for i in range(0,y):
                self.acc = int(math.floor(self.acc / 10))
        def opcode_5(self, data):
            """ Output operation """
            self.output.append(self.mem[data])
        def opcode_6(self, data):
            """ Store operation """
            self.mem[data] = self.pad(self.acc)
        def opcode_7(self, data):
            """ Subtract Operation """
            self.acc -= self.get_memint(data)
        def opcode_8(self, data):
            """ Unconditional Jump operation """
            self.pc = data
        def opcode_9(self, data):
            """ Halt and Reset operation """
            self.reset()
        def run(self, pc=None):
            """ Runs code in memory until halt/reset opcode. """
            if pc:
                self.pc = pc
            self.running = True
            while self.running:
                self.process()
            print "Output:\n%s" % '\n'.join(self.output)
            self.init_output()
    
    if __name__ == '__main__':
        c = Cardiac()
        c.read_deck('deck1.txt')
        try:
            c.run()
        except:
            print "IR: %s\nPC: %s\nOutput: %s\n" % (c.ir, c.pc, '\n'.join(c.output))
            raise
    

    As mentioned above, I will now refactor the code into Mixin classes and provide a full source output of that:

    class Memory(object):
        """
        This class controls the virtual memory space of the simulator.
        """
        def init_mem(self):
            """
            This method resets the Cardiac's memory space to all blank strings, as per Cardiac specs.
            """
            self.mem = ['' for i in range(0,100)]
            self.mem[0] = '001' #: The Cardiac bootstrap operation.
        def get_memint(self, data):
            """
            Since our memory storage is *string* based, like the real Cardiac, we need
            a reusable function to grab a integer from memory.  This method could be
            overridden if say a new memory type was implemented, say an mmap one.
            """
            return int(self.mem[data])
        def pad(self, data, length=3):
            """
            This function pads either an integer or a number in string format with
            zeros.  This is needed to replicate the exact behavior of the Cardiac.
            """
            orig = int(data)
            padding = '0'*length
            data = '%s%s' % (padding, abs(data))
            if orig < 0:
                return '-'+data[-length:]
            return data[-length:]
    
    class IO(object):
        """
        This class controls the virtual I/O of the simulator.
        To enable alternate methods of input and output, swap this.
        """
        def init_reader(self):
            """
            This method initializes the input reader.
            """
            self.reader = [] #: This variable can be accessed after initializing the class to provide input data.
        def init_output(self):
            """
            This method initializes the output deck/paper/printer/teletype/etc...
            """
            self.output = []
        def read_deck(self, fname):
            """
            This method will read a list of instructions into the reader.
            """
            self.reader = [s.rstrip('\n') for s in open(fname, 'r').readlines()]
            self.reader.reverse()
        def format_output(self):
            """
            This method is to format the output of this virtual IO device.
            """
            return '\n'.join(self.output)
        def get_input(self):
            """
            This method is used to get input from this IO device, this could say
            be replaced with raw_input() to manually enter in data.
            """
            try:
                return self.reader.pop()
            except IndexError:
                # Fall back to raw_input() in the case of EOF on the reader.
                return raw_input('INP: ')[:3]
        def stdout(self, data):
            self.output.append(data)
    
    class CPU(object):
        """
        This class is the cardiac "CPU".
        """
        def __init__(self):
            self.init_cpu()
            self.reset()
            try:
                self.init_mem()
            except AttributeError:
                raise NotImplementedError('You need to Mixin a memory-enabled class.')
            try:
                self.init_reader()
                self.init_output()
            except AttributeError:
                raise NotImplementedError('You need to Mixin a IO-enabled class.')
        def reset(self):
            """
            This method resets the CPU's registers to their defaults.
            """
            self.pc = 0 #: Program Counter
            self.ir = 0 #: Instruction Register
            self.acc = 0 #: Accumulator
            self.running = False #: Are we running?
        def init_cpu(self):
            """
            This fancy method will automatically build a list of our opcodes into a hash.
            This enables us to build a typical case/select system in Python and also keeps
            things more DRY.  We could have also used the getattr during the process()
            method before, and wrapped it around a try/except block, but that looks
            a bit messy.  This keeps things clean and simple with a nice one-to-one
            call-map. 
            """
            self.__opcodes = {}
            classes = [self.__class__] #: This holds all the classes and base classes.
            while classes:
                cls = classes.pop() # Pop the classes stack and being
                if cls.__bases__: # Does this class have any base classes?
                    classes = classes + list(cls.__bases__)
                for name in dir(cls): # Lets iterate through the names.
                    if name[:7] == 'opcode_': # We only want opcodes here.
                        try:
                            opcode = int(name[7:])
                        except ValueError:
                            raise NameError('Opcodes must be numeric, invalid opcode: %s' % name[7:])
                        self.__opcodes.update({opcode:getattr(self, 'opcode_%s' % opcode)})
        def fetch(self):
            """
            This method retrieves an instruction from memory address pointed to by the program pointer.
            Then we increment the program pointer.
            """
            self.ir = self.get_memint(self.pc)
            self.pc +=1
        def process(self):
            """
            Process a single opcode from the current program counter.  This is
            normally called from the running loop, but can also be called
            manually to provide a "step-by-step" debugging interface, or
            to slow down execution using time.sleep().  This is the method
            that will also need to used if you build a TK/GTK/Qt/curses frontend
            to control execution in another thread of operation.
            """
            self.fetch()
            opcode, data = int(math.floor(self.ir / 100)), self.ir % 100
            self.__opcodes[opcode](data)
        def opcode_0(self, data):
            """ INPUT Operation """
            self.mem[data] = self.get_input()
        def opcode_1(self, data):
            """ Clear and Add Operation """
            self.acc = self.get_memint(data)
        def opcode_2(self, data):
            """ Add Operation """
            self.acc += self.get_memint(data)
        def opcode_3(self, data):
            """ Test Accumulator contents Operation """
            if self.acc < 0:
                self.pc = data
        def opcode_4(self, data):
            """ Shift operation """
            x,y = int(math.floor(data / 10)), int(data % 10)
            for i in range(0,x):
                self.acc = (self.acc * 10) % 10000
            for i in range(0,y):
                self.acc = int(math.floor(self.acc / 10))
        def opcode_5(self, data):
            """ Output operation """
            self.stdout(self.mem[data])
        def opcode_6(self, data):
            """ Store operation """
            self.mem[data] = self.pad(self.acc)
        def opcode_7(self, data):
            """ Subtract Operation """
            self.acc -= self.get_memint(data)
        def opcode_8(self, data):
            """ Unconditional Jump operation """
            self.pc = data
        def opcode_9(self, data):
            """ Halt and Reset operation """
            self.reset()
        def run(self, pc=None):
            """ Runs code in memory until halt/reset opcode. """
            if pc:
                self.pc = pc
            self.running = True
            while self.running:
                self.process()
            print "Output:\n%s" % self.format_output()
            self.init_output()
    
    class Cardiac(CPU, Memory, IO):
        pass
    
    if __name__ == '__main__':
        c = Cardiac()
        c.read_deck('deck1.txt')
        try:
            c.run()
        except:
            print "IR: %s\nPC: %s\nOutput: %s\n" % (c.ir, c.pc, c.format_output())
            raise
    

    You can find the code for deck1.txt from the Developing Upwards: CARDIAC: The Cardboard Computer article, I used the counting to 10 example.

    Hopefully this article was inspiring to you and gives you a brief understanding on how to built class-bases modular and pluggable code in Python, and also gave you a nice introduction to developing a CPU simulator. In the next article which I hope to publish soon, will guide you through making a basic assembler for this CPU, so that you can easily and effortlessly create decks to play around with in the simulator.

  • Why Python?

    Posted on Wednesday, September 17th, 2014

    Here is an interesting subject which I would love to bring up and share with everyone. This article is not meant to compare Python with any other language, and in fact I will not be mentioning any language in this article but Python. As Everyone reading this might know by now, Python is a very diverse language. It can be used for tasks such as web development to the development of a Geographical Information System, to even computational biology and bioinformatics. Python has a vast array of fully supported GUI and web frameworks. This makes it a snap to apply the Model-View-Controller programming methodology to an entire application's ecosystem of software. The same code that is used to generate data for a GTK or Windows-based GUI application can also be applied to displaying the same data in a web application on the Internet. It is absolutely astonishing on what Python is really capable of when it comes to full-stack development. A developer really only need to know Python, and then the possibilities of what she or he can do is limitless.

    That is only the introduction to this article, and by now, regardless if you know or use Python or not, you should be at least somewhat impressed on it's capabilities. Python is so much more than a diverse language with limitless possibilities. Python is a vast community of developers from many backgrounds, ranging from computer science, to biology, to mathematicians, to even engineering. Python for many in the science community is the go-to language because of packages such as SciPy and Biopython.

    Python is normally installed by default on most Linux distributions, if not all. Python is also pre-installed on Mac OS X, making it a very universal language in terms of install base. This makes it easy to share your programs with other developers and colleagues. Due to this type of default install base on these platforms, some of the operating system specific scripts are also made in Python, and thus there are more bindings to operating system specific services that work right out of the box. This makes Python ideal for system automation and a general shell scripting replacement.

    If a developer knows Python, they are able to develop a wide array of applications, ranging from system automation, GUI applications, embedded development, web development, to even 3D modeling with Blender, all with a single language. This enables a developer to be extremely proficient in one language alone, as the developer doesn't need to worry about learning any additional languages to take on additional development tasks for their project. I personally distribute my Python knowledge throughout the Python ecosystem, I know how to build GUI applications, console application, system automation applications, network servers, and even web development. With Python, I am hardly limited to a single task, as just like Linux, Python is a general purpose. As with Linux, you aren't limited to just servers, you can deploy Linux on workstations, and embedded devices to even smartphones, as seen with Android. Knowing how to operate Linux in this day and age is starting to become critical due to the wide array of embedded devices and servers on the market today. Linux is only growing, and I believe it will play a vital role in the future of computing as we gravitate away from traditional PCs into the more mobile and embedded world. If Google and Apple have their way, PCs of the future will only be used for software development and PC gaming. I don't PC gaming dying, at least anytime soon.

    Python in the future will have a similar role to that of Linux, it will be that universal language that every computer science major will know and understand. With the Raspberry Pi, and children learning Python at a young age because of it, Python will be everywhere in our next generation of offspring. When I was a kid, BASIC was all the rage, where every home computer manufacturer had to ship with BASIC in ROM or the unit wouldn't sell as good. Python is like the new BASIC. When I grew into my teens and early adulthood, guess what was one of the more popular and widespread development kits for desktop development? If you said VisualBasic, then you would be correct. VisualBasic was huge in the late 90s and early 2000s, it was everywhere in the media and lots of organizations used it for development of internal software tools. As a result, ASP/ASP.NET used VisualBasic at the primary language at first, and is still used in .NET programming today.

    I believe Python is very critical for any developer to know and understand, or they will risk being left behind like FORTH and COBOL developers of the generation before BASIC. There are still some careers out there that require FORTH and COBOL, but for very old companies which have been around since the 70s and find it difficult to migrate away from their current solutions. If you understand how technology trends and have done research of how technology has trended since the 70s, it is difficult to not determine that Python is turning into the next hugely trending language and is only still growing. It is the language which our current offspring is learning that will be the next COBOL or BASIC in our technological world. Depending where some children grow up, you can also agree that Linux is also trending, especially in some European countries, and especially in communist countries where monopoly rule is banned. With the advent of Android, Ubuntu, Raspberry Pi, and the One-Laptop-per-Child project, children are seeing and using UNIX-like operating systems at a very young age. Also, the language shipped with the OLPC project is Python, with the entire Sugar desktop environment being made in Python. Another popular software product that teens and young adults are getting their hands on which uses Python is the Blender 3D modeling and animation software. One of my cousins has been learning Blender for her college degree as a 3D designer. In order to automate aspects of the 3D rendering software, and to enable game development, one can use Python. Python is literally everywhere, and in heavy use on popular software projects for the aspect of scripting and automation.

    In conclusion, to why to use Python should be a simple answer for any developer now. It enables a single language and code-base to be used across multiple projects easily, and it might very well be the next big language of our technological future. It is always best to be ahead of the curve and to not be left behind.

  • A JSON version of the standard Python xmlrpclib

    Posted on Thursday, August 14th, 2014

    I have always loved how easy Python made it to create simple RPC services using XML-RPC. I like how dead simple the API for both an XML-RPC server and client are. Here's an example XML-RPC Server in Python:

    from SimpleXMLRPCServer import SimpleXMLRPCServer
    
    # Create server
    server = SimpleXMLRPCServer(("localhost", 8000))
    
    # Register a function
    @server.register_function
    def hello(name='World'):
        return 'Hello %s!' % name
    
    # Run the server's main loop
    server.serve_forever()
    

    Accessing this service via Python couldn't be simpler:

    import xmlrpclib
    
    s = xmlrpclib.ServerProxy('http://localhost:8000')
    
    print s.hello()  # Outputs "Hello World!"
    print s.hello('XML-RPC') # Outputs "Hello XML-RPC!"
    

    I have been using XML-RPC a lot as a result, as I build mostly everything in Python. However, the overhead of XML parsing can be daunting for larger applications that are expected to receive many requests per minute. One technology that Python seems to be lacking in it's standard library is a simple service called JSON-RPC. Luckily I recently wrote a client library that currently only works on pure socket connections, here is the source code:

    #
    # JSON-RPC CLIENT LIBRARY inspired by the standard library xmlrpclib
    #
    # an JSON-RPC client interface for Python.
    #
    import urllib, json, socket
    
    class _Method(object):
        # some magic to bind an JSON-RPC method to an RPC server.
        # supports "nested" methods (e.g. examples.getStateName)
        def __init__(self, send, name):
            self.__send = send
            self.__name = name
        def __getattr__(self, name):
            return _Method(self.__send, "%s.%s" % (self.__name, name))
        def __call__(self, **kwargs):
            return self.__send(self.__name, kwargs)
    
    class ServerProxy(object):
        def __init__(self, uri):
            protocol, uri = urllib.splittype(uri)
            if protocol not in ('socket',):
                raise IOError, 'unsupported JSON-RPC protocol'
            self.__host, self.__handler = urllib.splithost(uri)
            self.__id = 0
        def __request(self, method, params):
            s = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)
            s.connect(urllib.splitnport(self.__host, 80))
            self.__id +=1
            data = {"jsonrpc": "2.0", "method": method, "id":self.__id}
            if params != {}:
                data.update({'params':params})
            s.send(json.dumps(data))
            resp = json.loads(s.recv(512))
            s.close()
            return resp['result']
        def __repr__(self):
            return '<ServerProxy for %s%s>' % (self.__host, self.__handler)
        __str__ = __repr__
        def __getattr__(self, name):
            return _Method(self.__request, name)
    

    This library works in the exact same manner as the xmlrpclib shown above, expect it connects to JSON-RPC servers, such as the XBMC Media Center software Here is how you would interface with XBMC:

    #
    # A super simple XBMC JSON-RPC Interface for Python
    # Control your XBMC in a Pythonic way!
    #
    import jsonrpclib, time
    
    class XBMC(jsonrpclib.ServerProxy):
        def __init__(self, host, port=9090):
            super(XBMC, self).__init__('socket://%s:%s/' % (host, int(port)))
        def notification(self, title, message):
            return self.GUI.ShowNotification(title=title, message=message)
        def global_search(self, query):
            self.home()
            self.Input.Up()
            self.Input.Select()
            time.sleep(2) # Wait for keyboard on slower boxes.
            return self.Input.SendText(text=query)
        def weather(self):
            return self.GUI.ActivateWindow(window='weather')
        def home(self):
            return self.GUI.ActivateWindow(window='home')
        def favourites(self):
            return self.GUI.ActivateWindow(window='favourites')
        favorites = favourites
    

    All of this code, plus many more, are now available on my newest bitbucket repository aptly called Python Experiments.

  • Two-factor laptop decryption using a smartphone

    Posted on Thursday, August 7th, 2014

    This is a pet project I showed to a select few at PyCon 2014 in Montreal. I have not publicly disclosed this project to anybody else in the tech community until now. I will not be releasing the source code for this project, so do not ask. I am either planning on obtaining a patent for this, or starting a Kickstarter project to obtain the funding required to develop this into a full fledged product for multiple platforms. Currently it only works with Linux, as it was much easier to prototype using such an open and hackable operating system. However, given enough interest, I would love to port this product over to Mac OS X and Microsoft Windows platforms. I am currently looking for any feedback on this, and to see how much interest there might be for such as product.

    This product has three separate parts, which enable what I am calling Chained passphrase decryption. The third part of the chain is not mentioned in this video, nor will I talk about it at all. Basically, all parts of the chain are required to successfully construct the final encryption key that will be used to decrypt the device in question. My software does not perform any actual encryption, my software only constructs the key which is then passed on to the encryption software. On the Linux platform seen in the video, the encryption software used is LUKS, which is standard in every Linux distribution for block level encryption. This software should theoretically be usable with any software that can accept a passphrase.

    The main communication link as seen in this video is a standard authenticated bluetooth connection, from the smartphone to the laptop. The connection is only active for as long as it takes to obtain that part of the chain to construct the final passphrase. In most cases, the software is manually launched by the consumer when he or she needs to decrypt his or her device. On most secure smartphones, the consumer first needs to unlock their device with a password, PIN, or fingerprint in order to launch an application on said device. Most devices also support full encryption, securing this solution even further. With only these two chains being mentioned, there is a lot of work a hacker will need to do in order to obtain the encryption key to unlock the device. Let's review what a hacker will need to do:

    • The hacker will need physical access to both the laptop and smartphone
    • If the hacker only has the laptop, he or she can try to bruteforce the encryption, however since the actual key is a binary format, and can be very long. This might take the hacker a very long time to crack. This key isn't your everyday ASCII passphrase, which are relatively easy to crack since the human alphabet is limited.
    • If the hacker has both the laptop and smartphone, they need to unlock that smartphone using the smartphones authentication. The only way by this is to remove the NAND chip(and hope that it's not encrypted), or find a vulnerability in the smartphone's lock screen.
    • If the hacker finally has access to the devices data and can run the application which creates the bluetooth link. The hacker then is required to enter in the correct PIN on the device. This can be brute forced, but hopefully the hacker never gets to this point.
    • Finally, there is one last part of the chain, the third part, which can break the chain of trust entirely, preventing that smartphone and laptop pair from even working. I will leave this part to the imagination of the reader.

    As you can see, much thought has been put into this product to make a more secure laptop for consumers and corporate workers. Unlike existing solutions which rely on local-only measures to enforce encryption, this method takes it a step further to authenticate the end-user in question using their most personal device, their smartphone. In this day and age, everybody has a smartphone, or if you work for any large company and have a company laptop, you most likely carry a company smartphone with you at all times. So naturally, the next step of laptop encryption should use this pairing to harden laptop encryption.

    UPDATE 08/08/2014: I have been receiving various comments from the different sources which I posted a link to this. I would like to clarify that this project may or may not be open source, this variable has yet to be determined, thus is why I am not releasing the source code at this time. Some comments are pointing on the fact that bluetooth is insecure, well this might be true, so is plain text SMTP, HTTP, etc... That is why you place an encryption layer over top. For email, this is normally PGP encryption, and for HTTP, this is SSL. This bluetooth link is also encrypted, currently using AES, but this is subject to change. AES was mentioned in the original video. Hopefully that clarifies some additional left out details.

About Me

My Photo
Names Kevin, hugely into UNIX technologies, not just Linux. I've dabbled with the demons, played with the Sun, and now with the Penguins.




Kevin Veroneau Consulting Services
Do you require the services of a Django contractor? Do you need both a website and hosting services? Perhaps I can help.

This Month

If you like what you read, please consider donating to help with hosting costs, and to fund future books to review.

Python Powered | © 2012-2014 Kevin Veroneau