I’ve been pretty quiet recently. Not a lot has happened except I’ve been working a fair bit recently, along with trying to pull myself out of various bits of mess I landed myself back at the end of spring. Today I’ve run into a nasty bug with jQuery and Java Applets.
Due to the nature of jQuery it appears that it likes to transverse through the DOM and do various alterations to it. This is all fine, until it comes to a pesky Java Applet. Java Applets however modify the behaviour of the host EMBED/OBJECT tag, probably because the Applet’s public methods are exposed as an external interface so that the applet can be controlled by JavaScript.
currently there is a bug open for this on the dev site
My way round this was to block jQuery from attempting to attach events to the JavaApplet. The implementation is a little dirty (see link above about issues with OBJECT being used in XHTML 2.x to replace IMG tags).
@elem.nodeType == 3 || elem.nodeType == 8… with …
!jQuery.isHtmlElement(elem)… throughout the rest of the jquery.js file.
Hopefully this will save a few other people going through the torture I had to go through, particularly as Internet Explorer also throw up some rather vague errors in the process of testing.
While I prefer and tend to use Ubuntu for most of my work, I occasionally have to do bits in OS X and Windows. Windows is a nightmare of an operating system at the best of times, but what often get caught out by is the fact that Windows line endings are CR LF based, which really upsets shell scripts in Linux/Unix. Whats more is that the errors you tend to get are not very descriptive, whether it be a script run from the console, or through web gateway interface (i.e. public/dispatch.*).
Another thing I often find that gets lost in translation is the executable permission bit if things get archived, or put on FAT/NTFS drives (i.e. flash USB disks). Subversion has an explicit svn:executable property, however git will just inherit the permission from the underlying operating system. Both have systems have advantages and disadvantages to their approach.
In case you need to reset these for some odd reason here is a rake task I hacked up that deals with this. You will want to make sure you have dos2unix installed. Normally it is in a package called tofrodos in Debian systems, and can be installed with sudo apt-get install tofordos
In this post I want to cover a number of very Rails specific ways to make code more robust. Many of these are small defensive coding techniques, but some also help tie down security issues.
Bang methods are methods that end with an exclamation mark. Their root is from impure functional languages such as LISP where this was used to denote side effects (mutability, IO etc). In ActiveRecord this is used to force exceptions to be raised if an action fails to act on the database due to validations, or failing callbacks.
I always had a tendency to avoid using these methods initially when I started off programming in Rails. Probably because I have a bit of a hatred for exceptions. This probably stems from the annoyance they cause when I was forced to do some university work in Java (I hate the fact you have to bloat up trivial code with catch clauses, and putting throws in method prototypes). Likewise in C++ exceptions are considered a bit of an evil because they don’t play nicely with raw pointers.
However in recent Rails projects I found that the except based Active Record methods can be a great aid during development. One area particular is during migrations. During development I have a tendency to break migrations into the following three main initial files:
During the data loading, I find it handy to use code to create basic model instances, such as initial users etc. By forcing exceptions you can be assured that all of the actions you execute will happen and there will be no cases where data will silently fail to save back to the database.
Another case where these methods become useful is during transactions, as exceptions will force transactions to rollback. It is often a good idea to use transactions in databases to ensure actions are atomic, not to mention in some databases the use of transactions can aid the speed up operations by reducing the communication between the DB server and client libraries.
Due to the event driven model of Rails programming, there are many methods such as before_filter, after_filter, before_save, after_validation, etc that can be defined either by overriding the function, referencing a function through a symbol, or passing a block code.
Unless there is a critical requirement for speed (in which case you are probably using the wrong language) then one should always use symbols or ‘proc’ objects. The issue when you override methods can make it very easy if you are using inheritance to block calls to callbacks defined in the parent class. This can be an issue when developing projects in teams as it means other developers will need to check parent classes when ever they want to use a filter.
I would guess most people are educated enough to know about SQL injection security issues. However it is very easy to overlook sensitive fields like password hashes, salts, and other credential-orientated. Without attr_protected it is very easy to exploit mass assignment in Rails for one who knows or has a good idea how the database schema works.
Again, this is a common mistake, and I often make this mistake. While pre-mature optimisation is considered to be an evil in programming, indexes are a very passive / non-intrusive form of optimisation. Typically one should look at placing indexes on any fields that are heavily used in comparison criteria. While ad-hoc guesses at optimisation can be bad, this will generally avoid your site from suddenly crawling in production mode due to drive seeks and heavy memory usage.
Clearly ad-hoc optimisation should not be substitute to a proper database tune, but it will quite often your intuition will be helpful. Likewise one should take time to clearly understand how your database uses indexes.
MySQL and PostgreSQL have very different techniques and features. For instance MyISAM tables have great text indexing support, but lack any form of atomic safety for transactions. Clever DBA’s have exploited this by using InnoDB for most data, and have used replication to mirror text-search content to a slave database that uses MyISAM to provide fast text searches.
MySQL has some very basic spatial data type support. PostgreSQL has even better support. Spatial data such as locations can not use your typical linear B+ tree based index, because the data works in several dimensions. Typically most DB systems will use bounding box based systems using structures such as R-Trees. While it’s not important knowing exactly how these algorithms use, a good programmer / DBA needs to be aware how to make the best use of them.
Active Record is a very powerful abstraction for SQL. You rarely need to use SQL directly if you use it properly. However, I still see a number of Rails projects with significant chunks of SQL code.
Even in cases where queries can get potentially very complex, such as search systems with large permutations of criteria, the use of scopes, and/or external libraries such as ThoughtBot’s Squirrel can still allow you to avoid this.
In cases where you do find you need to use direct SQL, it should generally only be when you need to exploit DB specific features or speed. In the case of speed, one should look at alternative indexing strategies first. Failing that, I have found it helps to hide each vendor’s SQL specific code behind modules that can be included to Active Record on the fly on start up. This way all vendor specific code it clearly isolated.
Surely you would have seen the rant on MySQL I made. MySQL is like VB for databases. It encourages some really poor practices. If you use PostgreSQL for development and SQLite (where possible) during development, you can pretty much guarantee it your code will work on MySQL in production (and you can always run tests using MySQL occasionally to ensure this is the case). Many apps coded with MySQL often have silly syntax issues when switching to another database system. To be honest, in most cases PostgreSQL and SQLite are far superior alternatives for a fair amount of projects.
I think this wraps up some fairly trivial, yet handy ways to write more robust Rails code. It always helps to keep an eye on blogs such as “The Rails Way” and the “Ryan Baits” blogs. “The Rails Way” promotes a number of ways to improve code using existing projects as examples. Ryan Baits will keep you updated with all the progress in edge Rails which often contains improvements to help DRY up code.
Excuse the AvenueQ reference here, basically horrible as it sounds, this is going to be one of my programming rants. I have a couple of pet programming rants I tend to find myself repeating, this is one of them and in due course I may even put the other rants up here. That way I can just point people to this blog instead of repeat myself yet again.
MySQL is one of the many popular Open Source systems that I hate because it gets way too much credit than it really deserves, not to mention the biased option that people tend to have. On top of this, much like PHP (yet another rant, for another day!), MySQL seems to have a somewhat deluded and brainwashed user base. While this post will come across as somewhat grumpy and scathing towards MySQL, the main reason I wanted to write this was to clear some of the common myths / misconceptions about MySQL that seem to have spread round the internet like some virus to developers.
All in all, I will bold enough to say that MySQL is as evil to SQL as VB is to programming in general.
Firstly, the first thing that annoys me no end about MySQL is while it is generally compatible with ANSI SQL 92 standards, it seems to encourage users to write poor SQL code that is not portable. It would not be a surprise to me if this was some cunning design plan my MySQL’s management as a poor means to building some form of vendor locking through developer ignorance.
For example, the MySQL often encourages developers to use non standard characters such as backticks instead of double quotes for specifying structure names such as field and table names. The other common trick is to use C style string escaping that uses the backslash instead of the using two double quotes like the ANSI SQL standards specify. Unless you manually tell MySQL to use ANSI compatible SQL, it will generate it’s quirky SQL commands when creating dumps, which I guess has encouraged many developers who started databases with MySQL to follow this ill-fated convention.
As a result, there are so many Open Source projects where you run into issues if you want to use SQLite or PostgreSQL because the SQL code they give you for basic DML statements uses the MySQL specific quirks.
MySQL only dominates the Open Source DB market because it’s had corporate marketing, which has lead people to believe absolute crap. The reality is that it’s a known fact that PostgreSQL is a far more stable and feature rich RDBMS, and has been for many years compared to MySQL. Yet if you enter put ‘MySQL vs PostgreSQL’ into Google, you’ll be inundated with shoddy reviews which all claim that MySQL is X number of times faster for practically everything.
From experience, having run into major performance issues with MySQL before, I looked at PostgreSQL. I found the issue with MySQL was that it would only use one index per a table / aliased table, which was making it slow. There was no way to get round this in a graceful manner using MySQL, and due to the requirements of the application I was coding, I couldn’t really adapt it to work round this rather undocumented design flaw of MySQL. Basically, what I needed was MySQL to be able to utilise several indexes per a table because the nature of the query was one where any number of field criteria where being applied in a flexible way. For instance, sometimes I needed just to search by a given location, sometimes it was for a given price range, sometimes it for was for entries modified in a given date range, but on top that any permutation of those criteria could be combined. For instance someone might want to search by location, price range and date range. However MySQL was too dumb to be able to use each of the indexes separately, combine the results using set logic and then get rows based on the result of the set operations. As a result, MySQL was SLOW AS TURD.
PostgreSQL did not have this annoying restriction and thus was much faster. The bottom line is that MySQL is fast for very trivial queries, however for more complex applications you can soon run into serious issues due to it’s design flaws. This annoys the hell out of me because people are dumb enough to assume the trivial benchmark results for simple queries still hold up for their application in the real world, despite the context being completely different. I only came to realise that the indexing system in MySQL was my problem because I quickly picked up how the query optimiser was utilising indexes in both MySQL and PostgreSQL from commands such as EXPLAIN SELECT. Thankfully, I was able to see past the brainwashing that existed on many comparison benchmark pages.
OK this is not such an issue as MySQL has been playing catch up as both versions 4, and 5 have managed to start implementing things like transactions, joins, etc. However back in the day of MySQL 3.x, it was pure hell trying to do anything a real database was supposed to do. MySQL was really nothing more than a fancy file storage interface. Even so, MySQL is still playing catch up heavily to PostgreSQL in terms of typical database features that a quality database system should be capable of doing. GIS support is just one area where I realised this.
This isn’t a major issue and many other places mention it, but people often forget that MySQL is not as free as it sometimes appears to be. It licensing is far muddier than the plain and simple BSD license that PostgreSQL offers.
While this doesn’t seem the most productive of blog posts, my main point was to highlight how MySQL has got developers into bad habits, and clear up the generally deluded reviews of MySQL that exist all over the internet, rather than sound like a grumpy old sod. If you are still using MySQL, seriously look at PostgreSQL or SQLite as alternatives. While developers are responsible for researching what RDBMS is suitable for their projects, I have to say from personal experience PostgreSQL generally tends to be far better system for larger projects than MySQL. For small projects where you don’t need to scale up in a distributed manner SQLite is great because it’s lightweight and hassle free. As a rule of thumb I try hard to ensure my SQL code is portable and that I minimise using vendor specific elements in order to keep my options open. When I do need to fall back on vendor specific code, I try hard to ensure it’s clearly documented and abstracted away.
In my previous post I covered in bit of depth some of the issues relating to the Rails fixtures system. In this post I’ll cover a couple more bad coding smells that make me cringe, along with simple defensive fixes that will hopefully protect you (especially for those who are still novice Ruby/Rails programmers) from hidden bugs arising later in your development cycle.
In this post I will probably follow on where I left off with issues relating to the issues caused by global data. I am going to home in on a couple of very simple gotchas that I don’t often see mentioned very often in the Rails ‘blogsphere’ that can lead to some very hard to trace bugs that only show up much later in the development cycle (or in some cases not until production deployment).
Code such as:
This was a major gotcha for me when I started using Rails. Many Rails programmers will use class and module variables without giving much thought. Because the variables exist within the scope of the related module or class it is easy overlook the fact this data is effect global.
What’s more is that issues created from this sort of code virtually never manifest themselves in development. The Rails development environment large chunks of code on request, and further more generally runs as a single web server instance.
Firstly, the biggest problem relates to what I outlined above, is due to the way the development environment shielding you away from the side effects this will cause. As a result you will end up with bugs in production that magically disappear when you try and debug them when go into development mode. This is a classic example of a ‘Heisenbug’ in the making!
A typically example where this will manifest is when you have multiple Rails’ instances running, each request is for a given user can be sent to any of the Rails instances running. However each Rails instance is not going to have it’s own class and module level variables. That is if Joe is served by instance A, which sets a class level variable during the request. When Joe makes another request, it might end up being sent to instance B, because instance A is busy. However because each instance has it’s own class level variables, it can not expect the variable it set in instance A to exist in instance B. That is Rail’s should really be treated as a shared nothing architecture.
In many cases Rails sites often utilise the ability for user’s to login. A common pitfall here is that user-sensitive data is held in any sort of class level variable, you are going to run the risk of leaking user data between different users. This is not nice given the fact that this issue probably will not expose its self even if you are using some sort of functional tests within your project.
Another overlooked issue is the the possibility for memory to ‘leak’. Well, the memory doesn’t as such leak, but any variables referenced from a class level variable will not get picked up by the garbage collector. This applies to all variable data with it’s referenced directly, or indirectly through a chain of several variables. If you have to store data in class or module variables, it’s generally best to try and stick to simple data types, such as strings and numbers. It is always good when deploying an application to use something like monit to keep a watch on memory consumption, not just on Rails app, but also any part of the Rails application that might run in a persistent / daemonised manner (such as a rake task that remains running in the background until explicitly terminated). I have been caught out a couple of times by unchecked memory consumption myself (Mainly because RMagick needs you to call GC.start after doing image related tasks to clean up C-bound API resources)
Many people don’t seem to understand the constant semantics fully in Ruby. It is very easy when dealing with constants that are not simple data types to suffer from the same issues as I outlined as above.
false,
:suspended => false,
# More criteria here
}
named_scope :some_search, :conditions => SEARCH_DEFAULTS
end
class UsersController < ApplicationController
def search
# Ignore the fact we our not using our named scope here
@users = User.find(:all, :conditions => User::SEARCH_DEFAULTS.update(params))
end
end
]]>
Despite the fact User::SEARCH_DEFAULTS should be constant, it does not mean the contents of the User::SEARCH_DEFAULTS hash table is constant. This is because Ruby’s constants act much like the const type qualifier in C++. That is firstly if you want, you can explicitly override constants in a similar way that C++ can let you use const_cast to remove the ‘const’ from a variable (Interestingly, Rails exploits the undef_const in it’s Dependency / class reloading system). Secondly, and most importantly in this case, is that only the reference to the object is constant, not the object it’s self. That is you can’t make SEARCH_DEFAULTS reference another object. However you can modify the object SEARCH_DEFAULTS points to. This will most likely trip you up when you are not using a simple data type such as a number, symbol, or maybe string. In the above example each search request will modify the User::SEARCH_DEFAULTS hash table. You need to be particularly careful here when using collection data types.
There is a very simple fix to this. It’s called the freeze method. All we need to do is call freeze after defining the hash table. For example:
false,
:suspended => false,
# More criteria here
}.freeze
]]>
This will ensure the hash table object is not mutable (modifiable). This therefore means that we need to update the UsersController#search method to call .dup after referencing User::SEARCH_DEFAULTS because the Hash#update method will modify the the receiver.
I hope this post will help others. As a seasoned Rails developer, I have to admit that even I got caught out a few months ago by one of the issues I outlined in this post. It goes to show how easy it is to overlook something when you are caught off-guard and are not coding in a defensive manor!
In my next post, I will probably cover some smaller more Rails centric issues that have managed to bite me when getting dirty with other peoples code when haven’t coded defensively.
I know so many people go on about Rails based coding practises. However, it seems despite all this harping that other people do, there are a large numbers that are out of tune with this. It’s some what has got to me as I really hate getting involved in projects when there are no clear guidelines on coding style or practises, because as soon as several programmers with varying styles get involved things get unconformable for everyone.
The concept of ‘Best Practises’ is certainly a very subjective issue. Certainly I will agree when it comes to basic code layout, there is no right or wrong way, likewise some of the points I will put here are going to be subjective, but this is what I think are some good starting points.
In this post, and probably some subsequent posts, I want to cover some basic principles that I personally think should be applied to all Rails projects as a form of defensive programming.
I know some will think this is somewhat odd, but I hate fixtures with a passion. My problem with fixtures is that they generally tend to work on a global basis, and thus can manifest issues that are typical of programs that misuse global variables and other forms of undesired side effect created by a shared state.
A typical example that illustrates this is that you need to add or remove data to a set of fixtures in order to reflect a specific new feature you’ve just added. Because of the global nature of fixtures, many other tests that use that set of fixtures are at risk if breaking because you are sharing that data. These other tests are completely unrelated to changes you’ve made, and therefore are ‘leaky’ in nature because they shouldn’t be failing as they are totally unrelated.
Another issue I have with fixtures is that they are not expressive. They seems to carry a lot of baggage that is not needed. For instance if I’m writing a test that relates to the billing of a property, I only am wanting to focus on fields such the price of the property and maybe some date specific stuff if the system is subscription based. Yet with fixtures you end up carrying a whole load of extra baggage for all the other fields.
What would be far better is to have some sort of base shared state, which basically is just used to set up the bare minimum defaults required for the given model. This would mainly be there to ensure things such as validations are kept happy. Each test can then then simply modify / update attributes to this base data. Personally, I think this is far more expressive. For example:
'My test property',
:description => 'Please buy me!',
:price => 450_000.usd,
:market => 'Sale',
:classification => 'Residential',
}
end
class User
EXAMPLE_DEFAULTS = {
:login => 'jase',
:password => 'dont ask me',
:credit_balance => 0.usd,
}
end
# ... etc for each model
# Now in your test cases you generate data on the fly rather than pull data from a fixture.
# This way each test is only having to deal with data specific to the it's test case.
describe Property, 'billing' do
attr_reader :property, :user
before :each do
# Note how we create each instance we need, overriding field relevant to what our
# test is going to cover
@user = User.example :credit_balance => 50.usd
@property = Property.example
end
it "should cost $10 to list a property" do
property.active?.should be_false
property.activate_listing!
property.active?.should be_true
user.credit_balance.should == 40.usd
end
# Put other related scenario examples here obviously
end
]]>
Using the technique illustrated above we manage to DRY up our test data in a similar way that fixtures try to keep things DRY up common data between tests. This also gets round the issues highlighted in the ‘shared state scenario’. That is if we decide to rename/delete a field such as User#credit_balance, we only need to change code in test that use this field. In the case of modifying a field like Property#summary, which is shared (because it is required through the use of a validation) everywhere, we only need to change it in one place, thus keeping our code DRY, and avoiding the ‘leaky failing case’ issue.
Not only does the above concept DRY up code and make it more robust, but we also improve our code coverage (If you haven’t heard of it, look at RCov). This is because we are using code to create our models code such as callbacks (i.e. before_create, after_create, etc) and validations will get called during our tests. Thus this helps reduce the flakiness of tests with very little extra effort.
Hopefully, I’ve illustrated the case why fixtures create brittle tests, and looked at a possible alternative style that addresses the major pitfalls incurred by using fixtures.
I’ll be planning to create a whole series of posts on this topic, so stay tuned.
RDoc is a really nice code-comment documentation generator to be honest, and compared to my experiences with other systems where you have to prefix ugly ‘@’ symbols all over the place, along with do other nasty things like give type hints (in the cast of messy PHP which doesn’t seem to know if it should use C style weak typing or Java’s strong typing). However, I do think that nearly all doc-comment systems seem to have awful CSS styling by default.
Below is a cut and paste CSS style sheet you can use to replace the default rdoc-styles.css file. You also might want to replace the index.html file with the version I provide here if you want a left aligned frame-set like the Ruby on Rails API Docs
You probably want to take a peek to see what all the rage is about. Well I’m currently using these styles for the TypeFace Documentation Site
Just copy and paste this, and use freely in your projects. Treat loosely it like Creative Commons Share Alike license. If you can give me a link back to here, to spread the word, I’d be grateful, but it’s not required if you don’t want to. If you do use these, it would be nice to know that others have found my CSS useful, if want to drop me an email I’d be grateful, let me know the URL of your project if it’s on-line.
Rails Framework Documentation
]]>
a {
background: transparent;
color: #039;
text-decoration: none;
}
/* and inside a section title */
.section-title > a {
background: transparent;
color: #eee;
text-decoration: none;
}
td {
font-size: 11px;
}
/* === Structural elements =================================== */
div#index {
margin: 0 -16px;
padding: 0 0.7em;
font-size: 0.9em;
line-height: 1.4em;
}
div#index a {
white-space: nowrap;
}
div#index a:hover {
font-weight: bold;
}
div#index .section-bar {
padding: 0.3em 0.7em;
background: #ccc;
font-size: 1.3em;
margin: 0 -0.7em 0.5em -0.7em;
}
div#classHeader, div#fileHeader {
width: auto;
color: white;
padding: 0.5em 1.5em 0.5em 1.5em;
margin: 0 -16px;
border-bottom: 3px solid #006;
}
div#classHeader a, div#fileHeader a {
background: inherit;
color: white;
}
div#classHeader td, div#fileHeader td {
background: inherit;
color: white;
}
div#fileHeader {
background: #057;
}
div#classHeader {
background: #048;
}
.header-table td {
vertical-align: baseline;
border: 0 none;
}
.class-name-in-header {
font-size: 2.3em;
font-weight: normal;
font-weight: bold;
font-family: "Arial Unicode MS", Verdana, Arial, Helvetica, sans-serif;
}
div#bodyContent {
padding: 0;
margin: 0;
}
div#description {
padding: 1em 1em 0.5em 1em;
background: #f8f8f8;
border: 1px solid #ddd;
margin: 2em 0 0 0;
}
div#description h1,h2,h3,h4,h5,h6 {
color: #125;
background: transparent;
}
div#description h1 {
color: #004663;
border-bottom: 1px solid #ddd;
padding-bottom: 0.25em;
font-weight: normal
}
div#validator-badges {
text-align: center;
}
div#validator-badges img { border: 0; }
div#copyright {
color: #333;
background: #efefef;
font: 0.75em sans-serif;
margin-top: 5em;
margin-bottom: 0;
padding: 0.5em 2em;
}
/* === Classes =================================== */
table.header-table {
color: white;
font-size: 0.8em;
}
.type-note {
font-size: 0.8em;
color: #DEDEDE;
}
.xxsection-bar {
background: #eee;
color: #333;
padding: 3px;
}
.section-bar {
color: #333;
border-bottom: 1px solid #999;
}
.section-title {
background: #79a;
color: #eee;
padding: 3px;
margin-top: 2em;
margin-left: -30px;
border: 1px solid #999;
}
table {
border-collapse: collapse;
}
td {
padding: 4px 20px 4px 0;
border-bottom: 1px solid #eee;
}
.top-aligned-row { vertical-align: top }
.bottom-aligned-row { vertical-align: bottom }
/* --- Context section classes ----------------------- */
.context-row { }
.context-item-name { font-family: monospace; font-weight: bold; color: black; }
.context-item-value { color: #448; }
.context-item-desc { color: #333; }
/* --- Method classes -------------------------- */
.method-detail {
background: #f8f8f8;
padding: 0;
margin-top: 0.5em;
margin-bottom: 1em;
border: 1px solid #ddd;
}
.method-heading {
color: black;
background: #ddd;
border-bottom: 1px solid #ccc;
padding: 0.2em 0.5em 0 0.5em;
}
.method-signature { color: black; background: inherit; }
.method-name { font-weight: bold; }
.method-args { font-style: italic; }
.method-description { padding: 0 0.5em 0 0.5em; }
/* --- Source code sections -------------------- */
a.source-toggle { font-size: 0.9em; }
div.method-source-code {
background: black;
color: #ffdead;
margin: 1em 0;
padding: 0 1em;
border: 1px solid #999;
overflow: hidden;
}
div.method-source-code pre {
color: #aaa;
overflow: hidden;
font-family: "Lucida Console", "Bitstream Vera Sans Mono", monospace;
}
/* --- Ruby keyword styles --------------------- */
.standalone-code { background: #221111; color: #ffdead; overflow: hidden; }
.ruby-constant { color: #d00; background: transparent; font-weight: bold; }
.ruby-keyword { color: #f60; background: transparent; font-weight: bold; }
.ruby-ivar { color: white; background: transparent; }
.ruby-operator { color: #00ffea; background: transparent; }
.ruby-identifier { color: white; background: transparent; }
.ruby-node { color: #fe6e34; background: transparent; }
.ruby-comment { color: #0e7e92; font-weight: bold; background: transparent; }
.ruby-regexp { color: #ffa07a; background: transparent; }
.ruby-value { color: #05dfdc; background: transparent; }
]]>
In recent months there seems to be a bit of attention building up with a project called Adhearsion which provides an excellent Ruby interface to the Asterisk VoIP open source server. I discovered these projects last week while researching into a VoIP based project that might end up doing.
Adhearsion provides a very simple way to control login in telephony applications that are powered by Asterisk and it’s AGI API. This allows developers to create bespoke switchboards systems very easily and quickly. While phone switchboards are seen as an annoyance, you can do some very nice things, and the examples provided show how one can integrate this with external API’s, such as weather and stock quotes very quickly.
To add to the existing Adhearsion / Asterisk bindings, I was interested in the possibility of looking at text-to-speech synthesis, because by default Adhearsion can only play back pre-recorded sounds such as GSM files. Likewise, to complement this, I also wanted to see how viable it was to use basic speech-recognition. I’ve not looked deeply into this, but this is something I plan to look at and maybe do follow up post / presentation about at some point in the future.
Festival TTS, which can be easily installed with most Redhat/Debian based systems provides a simple means for providing text-to-speech. There are also some basic Ruby Bindings which simply wrap round the command line festival command.
Speech recognition is not so easy. Phones generally only have a 8Khz signal, which means letters such as F and S can easily get mixed up due to the loss of sound quality. For accurate speech-to-text recognition, you really need 16Khz signals. However using a very limited vocabulary (utterance is the technical term), one can do some privative recognition via CMU Sphinx.
Again CMU Sphinx 2.x can be installed easily on Debian/Redhat based distros. Sphinx 4 however is Java based and needs be installed manually from the binary packages provided on the CMU Sphinx website. If you are using Debian based distro’s you should be warned that it typically ships with the CGJ flavoured virtual machine which is much less efficient than the original Sun JVM, so you will want to do something like sudo apt-get instal sun-java-bin to resolve this.
Sphinx takes a while to load, so it would probably be best to setup some sort of daemon system, of which I have seen some Perl scripts for that might work.
While I’ve not covered anything in detail, if anyone is having to do a similar project, it might help point them in the right place. If there are people who want to know more (or know more and would like to collaborate with me on a possible tutorial) please give me a shout at jason at hybd dot net. If a number of people are interested, it would certainly motivate me to do a detailed post regarding the topic.
As you may gather, I’ve been abusing RedCloth a lot recently, which is also reflected by the fact I’ve submitted a number of bug reports before the official release and a couple of small patches.
Like the old 3.x branch, you can create custom block commands. However due to the C/Ragel based parser the implementation details are incompatible. I’m going to illustrate how we can use Eric Meyer’s S5 tool for making XHTML based presentations and combine that with the new RedCloth 4.x formatter system in order to extend on the existing Textile mark-up system.
RedCloth 4.x can support custom block items thanks to a patch added by Tim Pease. Basically what we are going to do is make a new module that will provide two extra commands, slide and annotation to the existing Textile system when rendering to S5 format. This is done by having methods called slide and annotation in our custom formatter module. All block commands accept a hash table instance which contains any optional attributes and the block text (with in-line Textile commands pre processed). The S5 formatter is basically an extension of HTML formatting and you will see how we inherit the same formatting support though the include directive.
#{h opts[:text]}
' + opts[:text].to_s
end
def slide_end
'In order to allow for customisation of the main S5 mark-up, I will use ERB (as it’s comes with Ruby) to do this. When rendering to S5 format using RedCloth#to_s5 instead of RedCloth#to_html we can provide additional details such as the author, template file, a custom footer, etc.
As you will see most of the code is in a single module which as a few ERB helper methods, we include ERB::Util too so our template can use methods such as h and u.
You may see that I’m doing something particularly odd with the after_transform call back method. Basically the processed text is passed to it as a parameter and needs to be treated with reference semantics, despite the fact strings normally use value semantics by default in most scripting languages such as Ruby. If your from a C/C++ background, this will be a fairly normal idiom. Basically, you need to modify actual the contents of text parameter, rather than return a modified version. This is most probably because it was easier from an implementation point of view, and is marginally faster I’m guessing.
Typically, you should be able to do something like:
'S5 on Red Cloth Example',
:author => 'Jason Earl',
:footer => 'Custom footer content can go here'
]]>
By default the RedCloth#to_s5 method will look for a default.html.erb file in the current folder. You might want to change the location of this. You will probably want to save the result of the RedCloth#to_s5 method somewhere. This should have a sub folder containing the ui/default files from the original S5 package, as this contains CSS and JavaScript files that control the presentation system.
You can use this as a starting point / default template for S5 presentations. This needs to be in the same folder as the S5 formatter module.
<%= h settings[:title] %>
<% if settings[:author] %>
<% end %>
<% if settings[:company] %>
<% end %>
<%= content %>
]]>
Hopefully someone has found this a handy concept, and maybe has given others some idea how they can make their own extensions to RedCloth 4.x through the new formatter system. One of the nice things is that if you use an other formatter, the annotation and slide commands will simply be ignored and be rendered though the paragraph fallback, thus the case of generic XHTML formatting, will appear as a p tag. Obviously, one can use textile class names and custom CSS to control how these get displayed in other formats.
Between mocking this up and writing this post, I realised that there is a slideshow gem on RubyForge that works in a similar way. Slideshow offers some SVG graphics support which would be handy given the dynamic nature of screen resolutions.
Writing unit tests is a great way to ensure your code works and ensure that other programmers don’t break your code by accident. Not only that the whole testing philosophy seems to encourage you to think about making your code as modular as possible in order to be make it testable, thus making it more robust architecturally.
However, difficulties often seem to arise with when testing pieces of code that are strongly associated to any sort of I/O with external systems, or anything else that is tied to some sort of global side effect, such systems need to use randomisation or has time dependencies. Over the years most test frameworks have managed to over come these issues through the use of Mock Objects.
One issue though seems to haunt me a lot is dealing with systems which trigger events based on the system time. For instance credit based billing systems where the amount you pay for a service is based on the amount of time you actively use it. A good example would be property website where users will get billed a fixed amount for each active property that have listed on a monthly basis. In a test scenario we need to be able create a property, and simulate going into future by a set period of time and ensure that our system is able to pick up and bill our user for the time period we’ve moved forward by.
Thankfully, Ruby’s dynamic ability to override core features allows us to rewrite how the Time class behaves in a test environment. This can easily be done by making the Time.now method return a different time if we have decided to manually override the time. A piece of sample code you could use for in your Test/Spec helpers might be:
This code should be fairly simple to understand for those who know Ruby. Basically we use a class variable to hold the overridden time value. If this is not set it will get it from the system clock via the Time.new method which does the same thing as the old Time.now method. While we can override the time using the class writer for Time.active=, it is preferable that we use Time.is as this protects us from possible control flow issues that arise from not resetting the time back if pre maturely exit our code, either by accident through programmer error (i.e. returning from a function), or because of an Exception / failed test case. The use of blocks + ensure handles this in a similar way to RAII in C++.
As stated above, we should protect ourselves by using the Time.is method in critical areas, or at least use Time.active = nil in the tear down process of our test, otherwise we run the risk of affecting other our test cases incorrectly.
Below shows a basic pseudo code example for the property scenario I outlined earlier:
Due to the various issues with the old RedCloth 3.x branch I was one of the early adopters of RedCloth 4. RedCloth is a Ruby implementation of the Textile mark up / micro language for text formatting. For those who don’t know, Textile is a lightweight alternative to XHTML which is often used on the web in place of WYSIWYG HTML editors, mainly because it is simple and makes it easy to filter the text so there is no harmful (i.e. libale to XSS attack through injecting SCRIPT tags) or bad (i.e. Anything pasted from Micro$oft Office).
RedCloth 4 should be a lot faster, while eliminating most of the bugs known in On top of this it has very basic support for LaTeX and a simple system for support custom formatters. This proves to be very handy for me as I can use Textile both for web pages and PDF documents in a seamless way.
Here’s the nitty gritty details for those wanting to render PDF’s in Textile, here’s a quick summary how to do it.
sudo apt-get install texlivegem install RedCloth. You will need a C compiler installed for this, and note the capitalisation of the package name.Using the following code as a starting point / example, you should be get a basic PDF rendered:
As you can gather we need to generate a bit of extra meta data after call the to_latex method to let pdflatex know how to format things on the page. The resulting PDF will be in your home folder, again this will probably need to change to somewhere different, however this puts across the main points across for getting PDF documents out of RedCloth using Textile.
Also note that images are not fully supported at the moment.
While I’ve never really had issues with Subversion, I’ve always used it in a very closed environment where the centralised work flow never was an issue. However with the release of TypeFace, and the general raving in the Rails and Linux Kernel communities regarding Git, it’s hard to ignore it.
Having watched an excellent tongue ’n cheek presentation by Linus, I have to say I very quickly sold by it considering Linus is no marketing guy!
One of the interesting things about Git is that it completely changes your work flow and you suddenly find your self with a lot more freedom. Even if you are a solo coder, or still choose to flow the centralised model you were using under Subversion, you find that the ability to commit locally means that you commit more often because you don’t have to worry about making sure your code is ‘stable’ or ‘passes all the unit tests’ until you push it to the centralised system or someone else needs to pull from you. The increased commit rate makes a big allows everyone to have more up to date copies, which in turn avoids conflicts and saves merging. Certainly for OSS applications this a huge improvement as it encourages people to fork code and experiment now that merging is not such a chore.
Anyone using Subversion should seriously consider the switch. While there seem to be a number of nice distributed version control systems out there such as Darcs, and Bzr, it does seem like Git is showing it’s self to be one of the most superior. I prefer using .gitinore files to control ignored files rather than running svn propedit svn:ingore {folder} all over the place with new projects. The .gitignore file can easily be copied from project to project in the case of Rails projects.
There are tonnes of other cool features I haven’t discovered or reported on here; such as the ability to track code throughout the project is great. For instance git can tell if you move a function from one file to another because it ‘sees content, not files’ in the words of Linus. Likewise easy email patch generation, and the lack of .svn files being littered in each folder are a couple of others that spring to mind. (This is so handy as I often use grep to search through code, and hated all the annoying contents of .svn being reported by default).
If you use Windows, you probably need to use Cygwin for now until native binaries emerge. Also, the only thing I have found frustrating is that files that get end up getting the odd DOS style CRLF line ending in them later seem to make git moan about patching or something. I found the easy way round this was to use git config --global core.autocrlf=input (only needs to be one once on your computer) and turn off precommit hooks using git commit --no-verify when git complains. This global CRLF setting will ensure DOS line endings are only converted on input. Ideally if you use Windows, you should be using an editor like UltraEdit which allows you to edit in UNIX mode by default.
At long last I’ve finally got round to releasing TypeFace. While this initial release is still pretty rough on the edges, any reasonable Rails developer who is prepared to pick about a bit with the source should be able to do something with it.
Creating documentation and general tidying up has take a lot longer than expected. Documentation is still not great either. I plan to commit a few hours most days towards the project in order to make it more polished over time.
Expect some more regular posts on here relating to project in due course! You can find out more from by clicking here to go to the project page
For those who haven’t seen in my portfolio, TypeFace is a publishing system I wrote which I use to power a few websites. It’s ideally suited for static websites where the client needs a nice web user interface for updating their website.
I will warn the first release will be very rough. There will be bugs, and documentation will be poor. However, anyone with reasonable Rails skills should be able to make use of it. I would really like feedback from others as I will do my best to maintain it. All I need to do is get some time to create some documentation, create an example ‘site’ for people to get their hands dirty and clean up a few things in the interface.
As stated on the portfolio, the main advantage with TypeFace over most of the open source CMS systems out there is that it doesn’t require you to install it on the site’s web server as it generated content. This means you don’t have the issue of having to suddenly migrate the rest of your site round the architecture of the CMS like you do with other popular systems like Drupal, XOOPS, etc which require extra functionality to be implemented using their plug-in systems.
This will be my first serious contribution to back to the Open Source model. As strong supporter in the Open Source model, I think there is real value in programmers managing their own OSS projects. Not just for the OSS community, but it has many personal plus points:
The new TypeFace site will be going live over the next few days over here
Despite being involved in web development on-and-off in the last 4 years my life, I have managed to somehow shy away from setting up a blog of any sort. I did have a few very mediocre attempts, which never managed to get past my development environment.
Anyhow, I will do my best to keep this updated with the various thoughts, ideas and events relating to computing in my life. It will probably end up being very much a mash-up between web and games development articles.
The blog system I use at the moment is a very basic home-brew thing which still needs comments support added to it. Maybe in another 4 years I get round to implementing that!