What if Keyboards NEVER existed? Part 1 of Many.

Wow! I have been busy, although it may not seem so, seeing as how I haven't updated this in forever, but really, I assure you, have been quite busy working away. So I love retro things, whether it is an arcade, comic, or even computers. I've spent countless hours working on low level Linux kernel development, hacking android, working on the x86 port of it and tuio drivers / different multi touch implementations on Linux, and worst of all, I've spent most of my time trying to write new or rework existing window managers and UI toolkits for multi touch.

Obviously we have some great multi touch frameworks out there. The big problem though, is that none of it is native. I could go on and on forever about everything there is out there but they all have one commonality, and one ultimate downfall: they are all based and running on top of a keyboard and mouse system.

By the way I just want to throw in there that I absolutely love where android and the tablet scene is headed, unfortunately, that and almost all tablet technologies are based around single pointer emulation, i.e. your finger is the mouse click.

Now some of you may be saying, this guy has a really disorganized thought process and this article is all over the place, others will have noticed I’ve hit three main points so far. Let’s recap:

1. I love old things.

2. Everything is native keyboard / mouse (or mouse emulation) based

3. I love where tablet computing is headed

It is now time to tie those three things together.

What if the keyboard never existed? What if the mouse never happened? I got to thinking, if we had to start all over, how would we do it now? How would the operating system and how we interact with computers have developed? Not only that, what if they developed around the notion of directly interacting with the information and screen from day one? So that is what I have been doing. Some of you are going to be all like, way cool, and others like, that’s way dumb. I personally think it is great. So what am I doing exactly? Well, I’ve built a nice little 16 bit computer with LCD and touchscreen overlay, but I’m going to take it one step back further. I’m going to build an 8 bit native multi touch computer using the good ol’ z80. I’m actually not sure how well this will work, and if it will even have enough processing power to be able to drive the screen and do the gesture work in a time responsive fashion, but that is part of the fun and challenge of the 8 bit world, and even if it doesn’t succeed, at the very least it is going to force me to optimize and trim down much of the code.

On to the next part, I’m done with UI tool kits, specifically the realm of widgets, buttons etc. Done. That is the bane of modern computing and the biggest thing holding back multi touch application development as far as I am concerned. If you think about it, even the concept of the widget, the button, text box etc., is based around the concept of a single mouse click and the keyboard. So how much sense does it make that we are trying to do that with multi touch frameworks, that is make standard widget toolkits. It doesn’t make sense at all; they are two different realms of operations. Think of it like this. The current desktop is like dry land and the mode of transportation is a car with wheels, and the car is the widgets. The other realm is the tablet / multi touch scene and that is like water, and the mode of transportation ought to be a boat, but we are pulling in land technology, we are trying to use a car, trying to use widget based UI to navigate! This is obviously a no go. We need a boat.

Think about it like this as well: There is an infinite amount of information out there, and an infinite amount of ways to display it, and just as many ways to interact with it. So why are we going to limit ourselves to working with the data and displaying it with a single set of simple widget tools? What needs to happen instead is two parts; one part has already taken shape somewhat in the tablet scene. Apps. Apps? Yes, Apps, apps in the sense no two apps look or interact the same. That many apps are based around that specific data / function set, so in many ways, apps have already achieved widget independence for the most part, but not on the desktop, I’m talking desktop power, commercial grade application, power user apps. Not novelty checkers and Sudoku / tide table apps. The second part being that create an easy way for a person to PAINT, or render whatever kind of UI, basically custom widget for that application in that context, whatever works for the BEST interaction with the data. Think early WPF, but don’t think of all the problems with it. If programmers can easily and quickly render whatever they want and interact with what they renders, how they want, then you don’t need a widget framework / toolkit, and aren’t bound by its ridiculous constraints.

Anyways, I’d like to think some of you can see what I’m trying to get at and where this post is headed. This is just one of many posts in a series I will be doing on this over the couple weeks. Don’t expect to see me posting much code or hardware specs because, one it isn’t done, and two 99% of people would find it useless and or confusing and it would be defeating the purpose of this multi-part series, which is for me to explain my idea and thought process behind this course of development.

As always, I love feedback and constructive criticism etc. and most kinds of reader input.

Expanded Thought Map

Here is an expanded thought map, as you can see I've added some pages of documentation to a few nodes, I'm going to put up the interactive map today when I get off work, which will let you explore it more.

Why a hardware standard?

I had a comment and a couple emails asking as to what the point of standards are, especially the point of a hardware standard, let us take a short moment to explore this, hopefully after you read this post the entirety of reason will become self evident.

If you take the time to look, everything has standards, the lcd screen in front of you, the hdmi cable connecting your xbox and tv, the cd you put in your drive, everything has a standard, standards promote quality and insure interoperability between components. Lets take a look at how standards between the three branches feed themselves:

Software Standard:

The idea is to creat the worlds first native multitouch operating system based on the linux kernel. The operating system will run and work properly on all certfied hardware.

App Dev Platform:

The most successful and capable software is properitary, adobe, autocad, maya, etc. This is because sales gives the funding to develop them. The applications in turn cause the operating system to be successful. A successful operating system promotes the following of a hardware standard.

Hardware Standard:

The idea is that in the future there will not just be one manufacturer of multitouch tables / boxes, just as there is not one manufacture of mobile devices / touch screens / pc's. By creating a set of standards, if manufacters comply with them then they will be gaurnteed to work with the software. As hardware is purchased it provides funds to further develop the technology and promte growth.

As you can see standards are a good thing, each branch feeds back into the next fostering growth and development.

Thought Map

Here is my thought map for Multitouch Specification Project. I will add to this and continue to develop it. This is pretty much the bare, bare bones skeleton of it. As you can see it it is split up into the 3 main branches I discussed in my last post plus a new administrative branch I will develop to see oversight of all three primary branches.

I used vym to make this, you should be able to find it in most distro's software repositories.

If you want to add onto the thought map or suggest changes let me know, better yet, download it from this link, and email me a copy of your changes.

Update? A BIG UPDATE.

Alright so I have not been up keeping with this blog as well as I would have liked. I am glad though that it seems to have served many people very well! I have received a couple comments and many private messages requesting the live CD's. Here is what is going on with that, and how it ties into everything else:

1. By the time I finalized a live CD that I determined was bug free and ran on all the hardware / virtual hardware that I could get my hands on Ubuntu 10.10 came out.

2. Ubuntu 10.10 broke a few dependencies and other things and is largely untested.

3. Ubuntu itself is a problem in regards to the long term efforts of my project.

4. What is my project?

5. Linux in general on the desktop.

Alright, you can read lots of articles about this, Linux for the most part is awesome when it comes to the server environment, and even with modern desktop distros such as 10.10 Ubuntu, which in my opinion right out of the box is very comparable to even a mac, the desktop Linux has largely failed due to its lack of applications. As good and amazing as some open source software is the simple fact is nothing besides GIMP has come even close to its commercial counterpart, and debatable GIMP still can't touch a well trained Photoshop user. I realize I will probably get a lot of hate mail for this, but it is the truth. Unless you have the millions of dollars , time, and resources to develop something like auto-cad inventor, or Maya, it will never compare.

Back to the point, even with amazing as Ubuntu 10.10 is out of the box, it will and the rest of desktop Linux is failing because of application support. This got me thinking, open source is a great thing for an Operating System but is for the most part counterproductive to software. Also open source doesn't necessarily have to be free of charge either.

Where am I going with this? I think it is fairly self evident that eventually the keyboard and mouse will disappear. The obvious replacement is touch screen which has in the land of mobile computing, largely if not entirely replaced the keypad, just look at iPhone's and Android's. Eventually touch, and naturally multi touch will take over the desktop interface.

As we all know there is really no standard what so ever when it comes to multi touch. There is not even a hardware standard. There is also no real application base out there either. Linux actually has a chance for a massive market gain here. To be the pioneer where others are not yet reaching. All of the building blocks are here they just need to be put together. One must choose an operating system to build these future applications on. I'm not going to get into the details of this one but Linux is the obvious choice here. As far as the hardware is concerned, if you are reading this I am sure you are all to familiar with LLP, FITR, DSI, and every other derivative out there.

I feel like I am getting lost in the sauce here and just rambling, okay to collect all of this into a sensible post:

Here is what I want to do. I want to develop an open hardware solution, a Linux headless TUIO server which will pretty much be an embedded device with the camera attached directly to it; and a Linux distribution with an OpenGL interface with standard application development platform, think Visual Studio and WPF for all Windows developers reading this.

I want to build a team of people that will work on all three of these aspects.

The reason for the open hardware solution, there needs to be a standard for multi touch hardware, so pretty much we will have to pick a technology and stick with it. If you don't understand why there needs to be a standard, I don't even know what to say about that...

The distribution, this is kinda a toughie. If you think about it no current operating system in existence is really suitable for multi touch development. All current operating systems are based around keyboard and mouse input. If Linux were to drop out to the command line and not start the x server on a multi touch table that has no keyboard and mouse it would be pretty hard to recover the system. And no, we are not going to making a new operating system. There are ways for us to work around this problem. As far as basing this new distribution off of a current one, there is not much point, the only reason for that would be package, dependency and application support, but seeing as how all these applications and even the GUI is based around a mouse and keyboard input it is completely useless the multi touch community. All we really need is a bare bones Linux distro with XServer on it and OpenGL support. Remember that a large portion of desktop users spend their time on the internet so the distribution will have to pull all the required pieces to support things such as flash player etc. Other than that, it will be pretty much a blank slate for us to roll with. We need the WPF / Visual Studio style standard because the new technology needs an amazing development platform if companies will ever develop applications for it.

So hardware is never free, but if we get the standard down, we can make a table / slate style design that can be mass manufactured, something that people can pick up at a best buy.

Linux based and open source operating system, free of charge installed on the hardware.

Standardized development platform for companies to develop software for. Software that people will pay for.

I have seen a lot of something in the multi touch community that I really do not like. So much time and argument is being put into creating a gui that is best for or which one is the most correct for multi touch. Something that I have learned from my time in the military is that a timely decision is better than a decision that took a very long time but is more "correct". In the military it costs people their lives. The same principle applies here. A timely decision must be made. That's why who ever joins me on this project, we will come up with a set of standards and stick with it. People will use whatever you give them. They will make it work, and things evolve towards the better over time. Look at windows 95 compared to windows 7. I imagine our first distribution will be more like windows 95 than windows 7. If we keep debating and "researching" critical time will be lost and some one else will come in and take the multi touch reigns.

As many of you can see I am proposing a sort of hybrid thing here. Open standards and an open source operating system, but something that will have the support and backing of privatization.

If you want to join this team, and project send me a comment, email or message, point is contact me some how and let me know. I look forward to hearing from you all some more. and hope you enjoyed my latest post.

Live CD's

Good news everyone, I know I promised live cd's, and I am running a bit behind, but as I mentioned earlier I can't find a cost effective solution for hosting multi-gig iso files. As a result I will be offering live cd's via the mail, I plan on being able to offer them hopefully within the next week.

Another thing I would like to mention, I turned ccv into a background process and turned the gui into a webpage. This would allow you to run ccv on a bare-bones pc and simply connect it to the multitouch pc via Ethernet. You can then control the bare-bones via the web page dished out by apache. I will upload screen shots around the same time I offer the live cd's. It is still a little buggy, especially in the regards of streaming the video, but I really like the idea of a independent tuio server that is platform independent. This tuio server is currently the main focus of my project and I will be putting off the open gl multitouch gui tutorial off for at least another month as there are a few more bugs I need to work out before I offer that as a tutorial.

Back Again!

All right so I have been away for a couple weeks now but I am back!

I see a couple of you have had success and some are having problems. I am going to look at the problems and see if I can't resolve them.

Also here is what is coming up soon:

1. I am going to remaster this into a live-cd so that you can just download the iso and have a working fresh install of ubuntu 10.04 plus CCV.

2. The next tutorial will follow my development of an openGL based windowing system.

This windowing system will incorporate a tuio listner so that everything will be multitouch, even the login after you press the power button on your computer. :)