Quantcast
Channel: the evolving ultrasaurus » code
Viewing all 27 articles
Browse latest View live

markdown to textile with vim regex

$
0
0

So, I needed to change markdown to textile and google didn’t yield any handy scripts, so I sharpened my vim fu with Rubular, my favorite regular expression tester and came up with a few substitutions that took care of everything but lists and code blocks.

In vi, type ESC to go into command mode, then :%s/one/two/g will find every instance of “one” and replace it with “two”

First the easy stuff, headers. ^ finds the beginning of the line.

:%s/^# /h1. /g
:%s/^## /h2. /g
:%s/^### /h3. /g

To replace images, I needed to replace ![alt-text](link) with !link! so I needed to capture text. I suppose I didn’t really need the first capture, but I was working on the replace expression for a regular link when I realized it would be easier to do the images first. To understand the expression below, you need to know that \(stuff\) captures some text which can be inserted in the replacement text with \1 and \2, etc. So to get everything between square brackets, I use [\(.*\)]

:%s/!\[\(.*\)](\(.*\))/!\2!/g

All of my images appeared on a single line, so I didn’t catch a potential issue in the above expression until I got to replacing text links. I needed to use a non “greedy” capture so that I wouldn’t pull in text after the link that happened to include a parenthetical comment. Normally, in reg ex I would use (.*?) but in vim I needed to write \(.\{-}\) …wtf?

:%s/\[\(.*\)](\(.\{-}\))/"\1":\2/g

Special thanks to Adam Wolf’s tip via ShareGrove which helped me document these steps.

you can put VIM in a mode where the command history is just like another buffer. Not in insert mode, try q:

You should get a new buffer that you can edit with the command history in it, so “*yy would yank the current line into the system clipboard, etc.


rails security review checklist

$
0
0

I’m reviewing the security of a web app built with Ruby on Rails, so I put together a checklist for a security audit. This isn’t a bank or high security situation, but there were a number of engineers and quite a bit of open source code, so I thought a few checks were in order.

Here’s the list I came up with that I thought other folks might appreciate as a starting point (special thanks to the sfruby list, Mike Gunderloy, and Scott Bronson for feedback):

0) Make sure your Rails and gems are up to date for latest security patches (see rails security mailing list for recent advisory notes)

1) Active Record audit:
  A) SQL injection:
    (i) whole word search for “find”, “first”, and “all” then visually inspect all instances of ActiveRecord find calls for potential SQL injection vulnerability (also search for “sql” not whole work search to find find_by_sql and “execute” to find cases where raw sql is executed.
    (ii) search your models for “named_scope” and check :conditions
  B) check for mass assignment Either disable mass assignment as Eric suggests in his article, or audit its use. If doing an audit, check every model to make sure it declares which attributes are settable with attr_accessible. (While attr_protected may technically work, a white list approach is recommended by security experts and the rails security advisory on this topic)

2) Scripting attack: search all eRB files for <%= and ensure that if dynamically generated text was originally entered by the user, it is HTML escaped. Consider rails_xss

3) Secure Access: If some of the site does not have public access, check controllers and ensure that public actions are specifically allowed and that protected access is the default

4) search for “eval” (whole word) and verify that usages are safe (assume javascript eval is ok)

5) search for “forgery” (not whole word), make sure that
config.action_controller.allow_forgery_protection = false
is only disabled in test config
protect_from_forgery should be in the ApplicationController, unless there is a good reason for it not to be

6) check user auth and review that controller actions are limited to expected use

7) passwords: not saved as clear-text in the db, not logged

8) check that private data is not stored in cookies

rails 3 vs. rails 2 validation errors

$
0
0

Not sure if this is a bug or a feature. I’d guess it is here for a reason, and maybe I’m late for noticing, but Rails 3 errors now provides an array for each attribute, whereas in Rails 2.3 it was just a string.  Here’s the output from two almost identical applications…

Loading development environment (Rails 2.3.8)
>> person = Person.new
=> #<Person id: nil, first_name: nil, last_name: nil, present: nil…
>> person.valid?
=> false
>> person.errors
=> #<ActiveRecord::Errors:0×1034d8f10 @errors=#<OrderedHash …
>> person.errors[:first_name]
=> “can’t be blank”

Loading development environment (Rails 3.0.0)
>> person = Person.new
=> #<Person id: nil, first_name: nil, last_name: nil, present: nil…
>> person.valid?
=> false
>> person.errors
=> {:first_name=>["can't be blank"]}
>> person.errors.class
=> ActiveModel::Errors
>> person.errors[:first_name]
=> ["can't be blank"]

I didn’t see that in the release notes, but it failed my tests for ActiveRecord class. Someone else must have a list of these details, yes?

repl rspec mocks

$
0
0

REPL (Read-Eval-Print-Loop) is a great way to learn. With Ruby, the experience is enabled with irb. Sometimes, to do this we need to peek into the innards of things, which I find to be an extremely effective way to explain mocks and stubs. It’s a regular part of my Ruby curriculum, even though I have needed to figure out the syntax three times in the last couple of years. (Many thanks to Jen-Mei Wu for the most recent iteration.) I still think it is worth it, even though it seems to change with crazy frequency.

Just in case anyone else ever wants to do this with current or previous versions of RSpec, I thought I would write it down before old versions become lost in the mists of time:

RSpec 2.9

[update for repl mocks for RSpec 2.9 by Curtis Schofield]

>> require 'rspec/mocks/standalone'
>>Time.stub(:now).and_return(10,20)
>>Time.now
10
>>Time.now
20
>>Time.now
20
 

RSpec 2.5

>> require 'rspec/mocks'
>> include RSpec::Mocks::Methods
>>Time.stub(:now).and_return(10,20)
>>Time.now
10
>>Time.now
20
>>Time.now
20

RSpec 2.0

>> require 'rspec/mocks'
>> require 'rspec/mocks/extensions/object'
>>Time.stub(:now).and_return(10,20)
>>Time.now
10
>>Time.now
20
>>Time.now
20

RSpec 1.3

>> require 'spec'
>> require 'spec/mocks'
>>Time.stub(:now).and_return(10,20)
>>Time.now
10
>>Time.now
20
>>Time.now
20

what exactly does rake spec do?

$
0
0

$ rake spec
(in /Users/sarah/src/../my_app)
You have 1 pending migrations:
20110416135407 CreateCourses

The rake spec command reminds us that we need to run our migration before running
our tests. In fact, it does a whole lot more than that. There are a whole bunch of best practices rolled in that one
command. To see exactly what is going on, we can run rake spec with the –trace
option:


$ rake spec --trace
(in /Users/sarah/src/tfr/svn/Book/code/class_app_new_source)
** Invoke spec (first_time)
** Invoke db:test:prepare (first_time)
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:abort_if_pending_migrations
** Execute db:test:prepare
** Invoke db:test:load (first_time)
** Invoke db:test:purge (first_time)
** Invoke environment
** Execute db:test:purge
** Execute db:test:load
** Invoke db:schema:load (first_time)
** Invoke environment
** Execute db:schema:load
** Execute spec

When it says invoke it is calling a particular rake task, but then it will call its dependencies. To really see what is happening in what order, check out the execute commands. The commands db:test:prepare and db:test:load don’t do much themselves, aside from setting up the environment and executing another task or two. We can see from the output that rake is actually executing the following steps:

  1. Don’t run the specs if there are pending migrations in the development database. (db:abort_if_pending_migrations)

  2. Drop the test database (db:test:purge)

  3. Load the schema into the test database (db:schema:load in environment “test”)

These steps make sure that we are always testing in a clean environment, so we know exactly what we’re testing when we run our specs.

The code that makes this happen in Rails 3, can now be found in railties. (Thanks to @pixeltrix for pointing me to it.)/62206174505873408

rails 3.0 and rake 0.9.2

$
0
0

I really want to upgrade a Rails 3.0 project to Rails 3.1, but I’ve done a few spikes and it lacks test coverage, so I decided to pull in cucumber and write some features before moving forward.

I added cucumber-rails to my gemfile, and ran “bundle” and got this error:

/Users/sarah/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:289:in `load': uninitialized constant Psych::Syck (NameError)

What I really needed was to update my Ruby Gems (bundle update –system) but before I discovered that I did “bundle update” which moved me forward to rake 0.9.2, so I started getting these warnings:

/Users/sarah/.rvm/gems/ruby-1.9.2-p290@pie-bakery/gems/psych-1.2.1/lib/psych.rb:93: warning: already initialized constant VERSION
/Users/sarah/.rvm/gems/ruby-1.9.2-p290@pie-bakery/gems/psych-1.2.1/lib/psych.rb:96: warning: already initialized constant LIBYAML_VERSION
WARNING: Global access to Rake DSL methods is deprecated.  Please include
...  Rake::DSL into classes and modules which use the Rake DSL methods.
WARNING: DSL method Bakery::Application#task called at /Users/sarah/.rvm/gems/ruby-1.9.2-p290@pie-bakery/gems/railties-3.0.0/lib/rails/application.rb:214:in `initialize_tasks'

So, I’ve learned from google, stackoverflow, various blogs and my twitter friend @excid3 that I need to update my Rakefile to include:

require 'rake/dsl_definition'
require 'rake'
include Rake::DSL

That lets me use rake (yay!). I still have the following two warnings:

/Users/sarah/.rvm/gems/ruby-1.9.2-p290@pie-bakery/gems/psych-1.2.1/lib/psych.rb:93: warning: already initialized constant VERSION
/Users/sarah/.rvm/gems/ruby-1.9.2-p290@pie-bakery/gems/psych-1.2.1/lib/psych.rb:96: warning: already initialized constant LIBYAML_VERSION

which I’m hoping will go away with the Rails 3.1 upgrade, but I thought I would write up the rest of it in case it helps other wayward souls on their journey.

ffmpeg on osx lion

$
0
0

I found that I needed to convert an m4a audio file (which is what QuickTime saves when I record audio) to a wav file, so I decided to use my favorite “can opener.” The versatile open source ffmpeg tool has always seemed to be able to convert anything to anything in audio-video formats.

I decided to pull the source from git:

$ git clone git://source.ffmpeg.org/ffmpeg.git
$ cd ffmpeg/

Stable versions are tagged (which I could see with “git tag -l”). I don’t need to live on the edge right now, so I switched to the tag “n0.9.1″ which I assume is for the latest stable build “harmony” 0.9.1 and made a local branch based on that.

$ git co n0.9.1
$ git checkout -b n0.9.1

Instructions for building ffmpeg are in the “INSTALL” file. I discovered I needed yasm, which I could install with brew. Here’s what I did:

$ brew install yasm
$ ./configure
$ make
CC libavdevice/alldevices.o
CC libavdevice/avdevice.o
CC libavdevice/lavfi.o
AR libavdevice/libavdevice.a
CC libavfilter/af_aconvert.o
libavfilter/af_aconvert.c:53: warning: function declaration isn’t a prototype
libavfilter/af_aconvert.c:105: warning: function declaration isn’t a prototype
CC libavfilter/af_aformat.o
CC libavfilter/af_anull.o
CC libavfilter/af_aresample.o
:
:
ffserver.c: In function ‘parse_ffconfig’:
ffserver.c:4236: warning: ‘avcodec_get_context_defaults2’ is deprecated (declared at ./libavcodec/avcodec.h:3948)
ffserver.c:4237: warning: ‘avcodec_get_context_defaults2’ is deprecated (declared at ./libavcodec/avcodec.h:3948)
LD ffserver_g
CP ffserver
STRIP ffserver

I saw a lot of warnings, but they didn’t seem to negatively affect what I was trying to do. I found a nice blog post from catswhocode to remind me of the usage, and was able to use this simple command:


$ ./ffmpeg -i frog.m4a frog.wav
ffmpeg version 0.9.1, Copyright (c) 2000-2012 the FFmpeg developers
built on Jan 7 2012 21:19:08 with llvm_gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)
configuration:
libavutil 51. 32. 0 / 51. 32. 0
libavcodec 53. 42. 4 / 53. 42. 4
libavformat 53. 24. 2 / 53. 24. 2
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 53. 0 / 2. 53. 0
libswscale 2. 1. 0 / 2. 1. 0
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'frog.m4a':
Metadata:
major_brand : M4A
minor_version : 0
compatible_brands: M4V M4A mp42isom
creation_time : 2012-01-08 05:09:05
Duration: 00:00:07.22, start: 0.000000, bitrate: 206 kb/s
Stream #0:0(und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, s16, 201 kb/s
Metadata:
creation_time : 2012-01-08 05:09:05
handler_name :
Output #0, wav, to 'frog.wav':
Metadata:
major_brand : M4A
minor_version : 0
compatible_brands: M4V M4A mp42isom
creation_time : 2012-01-08 05:09:05
encoder : Lavf53.24.2
Stream #0:0(und): Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Metadata:
creation_time : 2012-01-08 05:09:05
handler_name :
Stream mapping:
Stream #0:0 -> #0:0 (aac -> pcm_s16le)
Press [q] to stop, [?] for help
size= 1244kB time=00:00:07.22 bitrate=1411.3kbits/s
video:0kB audio:1244kB global headers:0kB muxing overhead 0.003611%

$ ls
frog.m4a frog.wav

Success!!

cucumber and custom rspec matchers with rails 3.1

$
0
0

I’m working my way through an epic Rails 3.1 upgrade and some of my cucumber features were failing because I was using a custom RSpec matcher and the method wasn’t found.

My custom matcher looks something like this:

module CustomMatchers

  class XmlSubsetMatcher
      :
  end

  def be_xml_subset_of(expected)
    XmlSubsetMatcher.new(expected)
  end

and when I ran my feature I was getting this failure:

undefined method `xml_subset_of?' for #
0x007f9839d30378>



’’’


fixing brew install opencv on osx

$
0
0

This is more about fixing my brew install, than about opencv. As with many install issues the root cause was actually pretty simple, but finding it was challenging. Along the way, I fixed a number of issues which took a bit of digging to find, so I’m leaving a little trail on the web in case other people run into the same things — or in case some inspired open source citizen has time to add better solution messages to brew. The first step of any solution, is, of course, understanding the problem.

$ brew install opencv
==> Installing opencv dependency: cmake
==> Downloading https://downloads.sf.net/project/machomebrew/Bottles/cmake-2.8.7-bottle.tar.gz
######################################################################## 100.0%
Error: SHA1 mismatch
Expected: f218ed64ce6e7a5d3670acdd6a18e5ed95421d1f
Got: 3a57f6f44186e0dba34ef8b8fb4a9047e9e5d8a3

solution:
$ brew update
:
:
Error: Failed executing: make install (libtiff.rb:18)
If `brew doctor’ does not help diagnose the issue, please report the bug:
https://github.com/mxcl/homebrew/wiki/reporting-bugs

tl;dr;
install command-line tools from developer.apple.com

before I figured that out I fixed all of the issues found with ‘brew doctor’

$ brew doctor

Warning: Some directories in /usr/local/share/man aren’t writable.
This can happen if you “sudo make install” software that isn’t managed
by Homebrew. If a brew tries to add locale information to one of these
directories, then the install will fail during the link step.
You should probably `chown` them:

/usr/local/share/man/de
/usr/local/share/man/de/man1

solution:
$ sudo chown sarah /usr/local/share/man/de/*
$ sudo chown sarah /usr/local/share/man/*

Warning: “config” scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and what additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following “config” scripts:

/Library/Frameworks/Python.framework/Versions/2.7/bin/python-config
/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-config

solution:
Uninstalled python, which I don’t use much — I figure I can install later with brew
$ sudo rm -rf /Library/Frameworks/Python.framework/Versions/2.7
$ sudo rm -rf /Library/Frameworks/Python.framework/Versions/2.7
$ sudo rm -rf “/Applications/Python 2.7″
$ sudo rm /usr/local/bin/py*

Warning: You have unlinked kegs in your Cellar
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built.

coreutils
geoip

solution:
$ brew link coreutils
Linking /usr/local/Cellar/coreutils/8.12… 0 symlinks created
$ brew link geoip
Linking /usr/local/Cellar/geoip/1.4.6… 2 symlinks created

Warning: You have uncommitted modifications to Homebrew’s core.
Unless you know what you are doing, you should run:
cd /usr/local && git reset –hard

tried this:
$ cd /usr/local && git reset –hard
HEAD is now at ffb9aa5 Remove “__brew_ps1″ function from completion
–> didn’t work

solution:
$ pushd /usr/local
$ git status
–> lots of untracked files, no idea how I got into that state
$ git add .
$ git reset HEAD –hard
$ popd

Warning: Your Xcode is configured with an invalid path.
You should change it to the correct path. Please note that there is no correct
path at this time if you have *only* installed the Command Line Tools for Xcode.
If your Xcode is pre-4.3 or you installed the whole of Xcode 4.3 then one of
these is (probably) what you want:

sudo xcode-select -switch /Developer
sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer

DO NOT SET / OR EVERYTHING BREAKS!

I don’t have anything at /Developer, so I did this:
$ sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer

$ brew doctor
Your system is raring to brew.

Of course, it wasn’t, the key clue for me was finding this in the long stream of installation output:
tiffgt.c:35:11: fatal error: ‘OpenGL/gl.h’ file not found

which convinced me that I was missing some fundamentals. Searching on the text of the error led me to:
https://github.com/mxcl/homebrew/issues/11088

Ideally ‘brew doctor’ would have caught that I was missing the command-line tools that don’t get installed automatically with XCode 4.3. I installed those and all was well.

d3.js experiments in the console

$
0
0

d3 (aka Data-Driven Documents) is a great little JavaScript framework for data visualization. It’s got a nice declarative syntax for DOM manipulation that’s quite readable, but takes a bit of effort to understand exactly what it’s doing.

Favorite links:

  • UPDATE: Dashing D3.js is an amazing series of tutorials with great conceptual grounding
  • d3 tutorials provide a great conceptual foundation
  • Thinking with Joins by d3 creator, Mike Bostick, helps explain the syntax for chaining methods
  • Scott Murray’s d3 tutorial offers a very nice step-by-step, covering a lot of the same ground as my little tutorial below with excellent discussions of the fundamentals.

I like to understand stuff by playing with it interactively, so I created a skeleton index.html which just includes d3.js and a style a div where I’ll display some data.

UPDATE: blank file below posted here

<html>
  <head>
    <title>d3 experiment</title>
    <script type="text/javascript"
            src="https://raw.github.com/mbostock/d3/master/d3.js">
    </script>
    <style type="text/css">
      .box {
        background-color: skyblue;
        width: 24px;
        height: 18px;
        padding: 4px;
        margin: 1px;
      }
    </style>
  </head>
  <body>
  </body>
</html>



Then in the FireBug console, we can interact with d3, the top-level object that allows us to access all of d3’s goodness.

>>> d3
Object { version="3.0.1", random={...}, ns={...}, more...}
>>>  body = d3.select("body")
[[body]]

Like jQuery, d3 let’s us “select” one or more DOM elements to operate on them. I only have one body tag, so I just get one element in an array — not yet sure why it needs a nested array. Now I can manipulate the DOM:

>>>  body.append('p').text('Hello d3!')
[[p]]

and “Hello d3!” appears at the top of my page. Yay! Of course that could have been written in a single line like:

d3.select("body").append('p').text('Hello d3!')

and if I want to change the text, I can use a regular old css selector to grab the paragraph element I just created:

d3.select("body p").text("Welcome to d3")

or, using the reference to the ‘body’ variable I created above:

body.select("p").text("d3 is cool")

Data-driven Boxes

Ok, now that we understand the basics, let’s put some boxes on the page:

body.append('div').attr('class','box')

and let’s add a couple with text in them:

body.append('div').attr('class','box').text('hi')
body.append('div').attr('class','box').text('foo')

With my set of boxes, I can select one or all of them:

>>> d3.select('.box')
[[div.box]]
>>> d3.selectAll('.box')
[[div.box, div.box, div.box]]

Then I can specify data to bind to each box and display it. I’ve read that d3 can deal with all sorts of data (like json, csv, etc.) but we’ll start with an array of numbers.

>>> my_data = [20, 7, 32]
[20, 7, 32]
>>> d3.selectAll('.box').data(my_data).text( function(d) { return d } )
[[div.box, div.box, div.box]]


We can see that our data is associated with the DOM element and we can get at it via JavaScript in the console. (Of course, we should only do that for debugging. I would guess that __data__ is the private implementation of d3’s data binding.)

>>> d3.select('.box')[0][0].__data__
20

We can change the data like this:

>>> new_data = [10, 50, 25]
[10, 50, 25]
>>> d3.selectAll('.box').data(new_data)

You’ll see that the page doesn’t change visually, but in the console, you can see that the data does:

We need to explicitly tell d3 to do something with the data like this:

d3.selectAll('.box').text( function(d) { return d } )

We can also use this handy shortcut:

d3.selectAll('.box').text( String )

getting started with minecraft modding

$
0
0

I started learning about creating a Minecraft mod today, using this excellent tutorial (thanks @0×17h and @adudney).

Background: Minecraft is a very popular world-building game. They announced last November that they will release an API, but they have a friendly attitude toward users who have reverse-engineered how to create “mods” (extensions to the game and changes to the behavior of objects in the world). Minecraft Forge is the de-facto standard API toolkit for making mods.

We found that the easiest way to run a server is to run it locally on an old MacBook, allowing external connections to connect to a noip domain and tunnel into our home network.

Prerequisites:We’re running Mac OSX SnowLeopard. We need Java and the JDK, 1.6 or better.

Check that java is installed

$ java -version
java version "1.6.0_37"

Check that the JDK is installed

$ javac -version
javac 1.6.0_37

We installed 64-bit version of Eclipse.

Other than those platform-specific details that I had to look up, we had no problem following the basic modding tutorial. The mod doesn’t do anything yet, but we can run minecraft and see it load:

ffmpeg for ogg and webm

$
0
0

I wanted to experiment with Ogg and WebM and the <video> tag, and I thought the easiest way would be with my favorite command-line transcoder, ffmpeg. It’s fabulous support for just about every audio and video file format makes it fabulous for hacking.

Sadly “brew install ffmpeg” doesn’t support the new file formats and codecs, so I had to compile it myself, as well as the codec libraries. Here’s what worked (on MacOSX Mountain Lion):

0. Clone from git and use a stable branch
Check the bottom of the download page for descriptions of latest stable versons.

git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

Look for the version tag you want with: git tag -l
I picked 1.2.1 “Magic”

git checkout -b n1.2.1

1. Install Xcode, Homebrew, then and Remaining Dependendencies
following MacOSXCompilationGuide instructions.

2. Compile libvpx (needed for WebM)
I took these instructions from the UbuntuCompilationGuide, and swapped out the prefix to be what the Mac instructions told me to do:

git clone --depth 1 http://git.chromium.org/webm/libvpx.git
cd libvpx
./configure --prefix=/usr/local --disable-examples
make
sudo make install
make clean
cd ..

3. Configure & build ffmpeg
I picked just the libraries I needed plus whatever didn’t require extra dependencies:

./configure --enable-gpl --enable-version3 --enable-nonfree --enable-postproc --enable-libaacplus --enable-libcelt --enable-libfaac --enable-libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-openssl --enable-libopus --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvo-aacenc --en
make && make install

Ready to Make Movies!

Ogg Theora/Vorbis

I just followed the instructions on the ffmpeg wiki I used lower video and audio quality since the wiki recommendations ended up creating a huge file.

ffmpeg -i mymovie.mov -codec:v libtheora -qscale:v 3 -codec:a libvorbis -qscale:a 3 mymovie.ogv

WebM

ffmpeg -i mymovie.mov -vcodec libvpx -acodec libvorbis mymovie.webm

size chart

 2788953  pool-480p.mov
19459157  pool.mov
 2870873  pool.mp4
  833597  pool.webm
 3022359  pool3v3a.ogv
 4398308  pool4v3a.ogv
 5755066  pool5v5a.ogv
 7767888  pool6v5a.ogv
11424378  pool7v5a.ogv

getting started with drupal 7 and openshift

$
0
0

When learning a new web technology, I believe it is important to have hands-on experience from development to deployment. I chose Open Shift, one of the recommended cloud deployment options for Drupal 7. It’s free for 3 small instances, which seems perfect for experimentation.

I used the very simple web UI to create a Drupal 7 instance:

Note: Just like Heroku, you can’t write to the file system with open shift, so by default the Drupal install is “not scalable.” However, there’s documentation about how to configure it so that it would be scalable if you don’t need file upload — not sure why they wouldn’t just do that by default.

Next there’s a screen that provide credentials for the app, and a link to my new instance:

Woo hoo! I can log in with the default credentials (admin / openshift_changeme ) and see the default Drupal admin UI:

If you go to “People,” you can find your admin user and change the password. Of course, I’ve done this on my instance, so you can’t log in — you should go make your own!

Dev Setup?

I already had an account that I had experimented with for Ruby, but it was some time ago, so I needed to reinstall the local tools and setup:

1. requires git and ruby (since its “rhc” tool is a gem) — I’m using rvm with ruby 1.9.2
2. to setup client tools, as well as ssh key and a token locally for automagic auth:

rhc setup

3. clone the repo

git clone ssh://51d..183@drupal7-ultrasaurus.rhcloud.com/~/git/drupal7.git/ openshift-drupal7
cd openshift-drupal7

Not clear what to do from here. The OpenShift forums provides the hint that I can take a snapshot of files with

rhc snapshot save -a {appname}

but that doesn’t hint at a dev workflow.

I do want to figure out a nice workflow to develop locally and deploy on a free cloud. I’ve played with Acquia Desktop. MAMP has been recommended. I’d be more comfortable with “brew install …”

I will post this for now, in case any of you readers can clue me in or point me to some good install docs. I’ll update this post once I figure out something I like. Thanks in advance.

easy drupal install on mac osx lion

$
0
0

Lion comes with Apache and PHP installed. We just need to enable PHP, install MySql and create a database, then download drupal.

Enable PHP

My new laptop which came with Lion has PHP 5.3.15, using the test page we’ll build below you can verify you’ve got a good version of PHP to go with the Drupal you need:

  • Drupal 6: PHP 4.4.0 or higher (5.2 recommended).
  • Drupal 7: PHP 5.2.5 or higher (5.3 recommended).
  • Drupal 8: PHP 5.3.10 or higher.

Commands below use vi (of course, you should use your editor of choice). (thank you stackoverflow)

You’ll need sudo to edit the Apache config file:

sudo vi /etc/apache2/httpd.conf

uncomment the following line:

LoadModule php5_module libexec/apache2/libphp5.so

On the following line change _www to the username you log in with:

User _www

By default, Apache only loads index.html, so update that line to:

DirectoryIndex index.html index.php index.htm

Restart apache on the command line

sudo apachectl restart

Now, let’s make a test file:

cd ~/Sites
mkdir php-test
cd php-test/
vi index.php

Then paste in this test file and browse to http://localhost/~yourusername/php-test/

Install MySql

I like to install everything with homebrew which is my favorite package manager for osx. It keeps everything in /usr/local/ by default, and then I don’t have to remember where GUI installers put everything.

brew install mysql

I like to have MySql always start up, so I use a Launch Agent, which homebrew describes how to do after the install:

To have launchd start mysql at login:
    ln -sfv /usr/local/opt/mysql/*.plist ~/Library/LaunchAgents
Then to load mysql now:
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist
Or, if you don't want/need launchctl, you can just run:
    mysql.server start

Now make a database:

mysqladmin -uroot create drupal_dev

The default way this is set up is for a dev environment, so there’s no password and root has full access. You can see this in the mysql console:

mysql -uroot
show databases;
show grants;

Now we need to fix the PHP configuration to refer to mysql correctly so we don’t get PDO errors (thanks Hiraash):

sudo cp /etc/php.ini.default /etc/php.ini
vi /etc/php.ini

replace all instances of /var/mysql/mysql.sock with /tmp/mysql.sock
and restart apache

sudo apachectl restart

Download Drupal

I followed the drupal.org instructions, but you need to go find the right version. As of this writing, 7.22 is the recommended v7 release, which is what I’ll be working with.

cd ~/Sites
wget http://ftp.drupal.org/files/projects/drupal-7.22.tar.gz
tar -xzvf drupal-7.22.tar.gz
rm drupal-7.22.tar.gz

Now, I’ll just rename it to be the same name as the database we set up:

mv drupal-7.22 drupal_dev

browse to http://localhost/~yourusername/drupal-dev and proceed with the install wizard

step through the wizard and drupal sets itself up.

tesseract html page with text overlay

$
0
0

I’m learning a bit about OCR, and wanted to get some hands on experience using the open source Tesseract to get a feel for how it works. I’m a long way from any reasonable visual or interaction design, but the result of today’s exploration is an html page where the original image is overlaid with machine generated text in roughly the right location. This page looks like crap, but it’s a neat first step (click on the image below to see the full html page):


Tesseract

Tesseract is an open source OCR tool (Apache 2.0 license) that produces fairly accurate output (relative to its open source peers) for scanned, type-written documents in English and many other languages.

On the Mac, we can easily install it with homebrew:

brew install tesseract

The latest version supports lots of different input image types, via Leptonica, an open source C library for efficient image processing and image analysis operations.

If you just want to get text from an image, check the ReadMe file. I wanted to display the generated text over the image, which Tesseract supports via the HOCR format.

Each word, line, and block of text, is annotated with an HTML tag. I look at just the word element, which is generated as a span tag with the attribute title:

<span class="ocrx_word" id="word_14" title="bbox 398 506 471 527" >WHOM</span>

I wrote a little javascript to create a style tag from the bbox attribute:

Manuscript.bboxToStyle = function(bbox_str) {
  arr = bbox_str.split(" ");
  left_pos = "left:"+arr[1]+"px; ";
  top_pos = "top:"+arr[2]+"px; ";
  right_pos = "right:"+arr[3]+"px; ";
  bottom_pos = "bottom:"+arr[4]+"px; ";
  return left_pos + top_pos + right_pos + bottom_pos;
};

Then used jQuery to apply that to every word element:

$(document).ready(function() {
    $(".ocrx_word").attr('style', function() {
        return Manuscript.bboxToStyle(this.title);
        });
});

Resulting in word elements that are positioned roughly where they appear in the image:

<span class="ocrx_word" id="word_14" title="bbox 398 506 471 527"
style="left:398px; top:506px; right:471px; bottom:527px; ">WHOM</span>

I tried to experiment with the background-color of the words, but that’s not working for some reason. Complete source on GitHub.

Would love to hear about anyone else creating HTML UI for OCR results, either with Tesseract or other open source tools.


building unix package with autotools

$
0
0

ever wonder what all those random files are at the root of some package source that you are playing with? and how exactly does the mystical configure command actually do?

Alexandre Duret-Lutz has created a fabulous Autotools Overview & Tutorial — well worth flipping through the first 19 slides (38 pages of the PDF since each there are “builds” of individual slides exported as multiple pages).

Later, at page 97, he starts a tutorial section, which I replicated in this git repo — the README has a little cheat sheet of step to set up and build the package.

building tesseract from source

$
0
0

Thanks to the prior work of Matt Christy at eMOP, I got started building Tesseract from source (on Mac OSX 10.8.4).

Here’s my slightly modified workflow:

svn checkout http://tesseract-ocr.googlecode.com/svn/trunk/ tesseract-ocr
cd tesseract-ocr
./autogen.sh
mkdir build
cd build
../configure
make
make install

recently a makefile changed, and I need to regenerate them, starting at the source code root:

autoreconf --force --install
cd build
../configure
make
make install

Making a “build” directory, makes it easier to keep track of source code changes with svn. I set up my global ignores to ignore the interim files and directories.

vi ~/.subversion/config

then I uncommented this line and added everything after .DS_Store

global-ignores = *.o *.lo *.la *.al .libs *.so *.so.[0-9]* *.a *.pyc *.pyo
   *.rej *~ #*# .#* .*.swp .DS_Store *.in build config configure *.cache
   aclocal.m4 m4

So then I only see source code files that are added or modified when I check

svn status

getting started with drupal

$
0
0

Emeline Glynn and Anthony Glynn gave a helpful talk called “How to teach anyone Drupal in 7 months.” The timeline was based on their experience where Anthony, an experience Drupal developer, taught Emeline remotely over a period of 7 months, to the point that she is now working professionally as a Drupal developer. Slides posted here.

Emeline notes that you need a passionate & dedicated student. It can be very frustrating, especially as someone new to development, especially learning command line tools, git and debugging, but she was excited about it and found it very rewarding even when she was very new. She usually spent 4-6 hours a day. Anthony noted that if someone has experience with another framework, you can expect the timeline to be 2-3x faster.

I liked this breakdown of stages in learning:

  1. Learn the Language
  2. Get the Skills
  3. Cross the Chasm
  4. Leave Footprints

For me, as a developer with 20+ years of experience, I pick this stuff up fairly quickly.  I spent a few solid days on getting my dev env setup, understanding the major components and making small changes to a module.  As important as the code, is understanding its patterns and jargon.  The immersive experience of CapitalCamp with its enthusiastic community and these references to key learning resources has significantly accelerated the learning curve.

This learning path focuses on the non-programmer; however, Anthony suggested (and I agree) that the experienced developer would take a similar path.

  1. Register at drupal.org
    drupal.org/planet should be your home page, with news feeds of all the best blogs online
  2. drupal.org/security – the Drupal security team meets once a week and issues security advisories less frequently — they also have a mailing list.
  3. Meetups — Anthony learned by reading Pro Drupal Development & going to meetups. (He noted that the community is a bit less friendly on line.) Meetups provide inspiration and the landscape of what to learn.
  4. Setup your dev env
    Recommended: Linux, git, Drush, PhPMyAdmin, Firebug or Chrome Developer Tools
    (Linux much easier than Windows)
    On Mac OSX  it is pretty easy too, but finding good setup instructions was hard.  I posted the steps I the I use and have since added apache vhost config so I can run multiple drupal sites easily.
  5. Embrace the command line! students may be freaked out at first, but it makes you very productive. If the student is non-technical, there are a lot of skills that are not Drupal that you still need to learn (e.g. command line, git)

Consider starting with a distribution profile

  • Drupal Commons: if you want to manage communities
  • Drupal Commerce: e-commerce
  • OpenPublic: government site

Terms & Landmarks

  • API api.drupal.org
  • node: a piece of content
  • module: a Drupal extension written in PHP

Helpful modules

  • administration menu: allows you to hover on Drupal menu items and shows submenus (drill down)
  • coffee: type in the page you want to go to and link pops up (also for the admin UI)
  • module filter: allows your module page to look clearer
  • devel: key helper module, allows you to access those variables,
  • develthemer – helps you develop themes
  • views – pretty much on every site
  • features – export your config (config + content is in the database)
  • panels & context: each puts you don’t a certain path
  • context – if you are in this part of the site, show this
  • panels – let you pick

Learn about themes – Omega, Adaptive theme and something else.  Also, subthemes.

Build

Debugging tips

  • “Calm down and clear the cache”
  • become fmiliar with the apache logs and drupal watchdog logs

Make students feel adventurous! Db backup + git makes it safe

More tips

  • codecademy – good for PHP
  • make sure you understand the hook system
  • look & use other people’s code
  • write patches
  • write your own blog
  • PHP function: debug_backtrace

litmus test / goal is to write your own modules

empowering content creators with drupal

$
0
0

Bryan Braun (@bryanebraun) gave a refreshingly opinionated talk Empowering Content Creators with Drupal.  Coming directly from the Ruby and Rails communities where a core value is to articulate best practices, it is great to see this kind of guidance from a member of the Drupal community.  (slides here)

Bryan referred to a blog post by Dilbert creator Scott Adams on the Botched Interface Market, where sites like Orbit and Travelocity had such poor user experiences that they inadvertently created a market opportunity for new sites like Hipmunk.  Bryan’s goal is that Drupal should not become a botched interface market.

He organized his advice into two categories:

  1. More Control: Power Tools, Reducing Risk
  2. Less Confusion: Better Interface, Streamline Workflow

Bryan highlights borrowed techniqes from other platforms (mostly WordPress), and points out lots of “low-hanging fruit” techniques.

Example: the default layout for a node has a lot of labels and whitespace which doesn’t contribute to understanding the page.  The default UI for a file attachment is not concise.  Compare Gmail’s message compose interface vs. what it would be in Drupal (~1/3 of your screen vs. a page that is two screens high!)  Having a very long page can contribute to confusion, since you have to scroll to get to some of the functionality.

Larry Garfiled’s blog post Drupal is not a CMS points out that Drupal is something you use to build a CMS.  Perhaps we should think of it as a CMF, a Content Management Framework.  In this case, we are designing the workflow and user experience of a new custom CMS that we build with Drupal.

Myth: I am not a designer
Fact: um.. you actually are. Whether you identify as a designer or not. The decision you make when you are building a site has an effect on the end user — these are design decisions!

Myth: it will take a long time and effort
Fact: it could but it doesn’t have to

More Control

We can make our content creators more productive by giving them “power tools” while at the same time, reducing risk that they will make mistakes will give them confidence to move faster.

WYSIWYG

It’s not really an option anymore to tell people to edit HTML. [My personal perspective is that even though I am perfectly capable of HTML markup, why should you make me type extra characters? why should I have to learn the markup that works with your stylesheet? Though I do like the option of editing HTML when the rich text editors fail me.]

The following modules are good for WYSIWYG and File Upload:

  • wysiwyg or ckeditor which both appear to be rich text, WYSIWYG editors
  • media or imce for uploading and managing file (also seems to be an either/or choice

Node Editing Power

Always check “Add to Menu” on the Node Edit page — for most content creators, if a menu isn’t there, the feature doesn’t exist.

With context, use a module with Context Node which can pull out just a few context options and put it on the Node Edit page.

As a themer, you can make a bunch of templates and the content creator can pick with template -picker module.

Use Short Codes

Short code are a best practice from WordPress and can accelerate content creation:

[video:url]
[[nid:123]]

Drupal Modules:

  • Caption Filter
  • Video Filter works with the wysiwyg module or with TinyMCE or CKEditor (not sure why those are in a different category).
  • Insert View lets content editors add views without editing PHP
  • Make your own

In Drupal we tend to expose this kind of functionality as a block, but that gives the power to the site builder, rather than the content creator. In Drupal, the modules that do this tend to be called filters.

Reduce Risk

Always put description text when you are creating new fields (you probably won’t come back later). If you are going to come back later, you can improve it then — at least put something in.

  • Autosave
  • Revisions: just enable it by default
  • Preview — poor experiences with the preview button, not always what you expext
  • Turn off preview button on content types page, use use view_unpublished module instead

Granting Appropriate Access is not an easy fix.  You need to understand how your organization works, know the users, watch them work, etc. Once you do know those things, you can set up good workflows for them with clear options for the different roles in the organization.


Less Confusion

Admin Themes can help

  • Rubik is less cluttered: 2-column node-edit page, description text on
  • Ember is responsive

Logical Grouping

Group things according to how your content creators think about them, not how you built them. Consider grouping admin menu items into safe vs. risky like WordPress does. Bury advanced and less-frequently used functionality.

  • Fields into fieldsets & field collections
  • Admin Menus for content creators
  • Permissions into Roles
  • WYSIWYG Icons

Default to expanded for highly recommended options, collapse for optional, and hide anything that is unnecessary.

  • Use conditional fields (never show someone an option that does nothing
  • Simplify hides fields
  • Jammer hide many other things)
  • Hide Submit Button avoids double submissions and just an importantly communicates to the creator or editor that the site is actually doing something
  • Preview Button (see above)

Streamline Workflows

  • Use Chosen instead of the default multi-select
  • Add Another is great for repetitive node creation.

Dashboard

There are lots of great dashboard modules.  Consider what your creator or editor wants to see the most — make that their default homepage.

“the best tool of them all… …is the one that works best for your users”

It looks like I’ll still have to do quite a bit of experimentation, since Bryan often points to multiple modules to solve the same challenge — still a great reference to address common concerns and highlight best practices.

rails 4 with mongodb on osx

$
0
0

This post covers getting started with Rails 4 and Mongodb, using the Mongoid gem. As part of getting up to speed, I enjoyed reading Rails + Mongo take on the object-relational mismatch by Emily Stolfo from MongoDB.

For starters, here’s a how to create a super simple toy app. I assume Ruby is installed via rvm, but you are new to Mongo.

Install Mongodb on Mac OSX

brew update
brew install mongodb

To have launchd start mongodb at login:
ln -sfv /usr/local/opt/mongodb/*.plist ~/Library/LaunchAgents
Then to load mongodb now:
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.mongodb.plist
Or, if you don’t want/need launchctl, you can just run:
mongod

Rails 4 with Mongoid

I chose Mongoid over MongoMapper (see quora, stackoverflow)
I used MRI 1.9.3 (at this writing, Mongoid supports HEAD but not 2.0)
rvmrc:

rvm use ruby-1.9.3-p429@rails4 --create

added to Gemfile:

gem "mongoid", git: 'git://github.com/mongoid/mongoid.git'

on the command-line:

rails new mongo-people --skip-active-record
rails generate mongoid:config
rails generate scaffold person name street city state
rails s

Woo hoo! We’ve got an app — looks like a vanilla Rails app from the outside, but it is different on the inside:

class Person
  include Mongoid::Document
  field :name, type: String
  field :street, type: String
  field :city, type: String
  field :state, type: String
end

No database migrations needed. If we want a new field, we could just declare one in the model and add it to our views. I assume migrations could be used for data migration, but that would be a subject of another post.

References

Rails 4 with MongoDB Tutorial

Viewing all 27 articles
Browse latest View live




Latest Images