Moved blog

I got interested in the jekyll project for generating static websites. As a test I have migrated my blog here.

Code paste

Use this service for pasting code with syntax highlighting and even test execution:
http://codepad.org

How to make web2py faster

Avoid regenerating tables:

db = DAL('sqlite://storage.sqlite', migrate=runonce) 

Create indexes:

db.executesql('CREATE INDEX IF NOT EXISTS {table}_index ON {table} (id);'.format(table=table))

Compile byte code in admin

Serve static files direct from server

Move as much logic as possible from models into controllers and modules

Disable session if possible

Cache database queries and controller functions:
db().select(db.log.ALL, cache=(cache.ram, 60))


@cache(request.env.path_info, time_expire=5, cache_model=cache.ram)
def index():
pass

Let webfaction serve your static content

Create a static symbolic link application:
Name: web2py_static
App category: Symbolic link
App type: Symbolic link to static-only app
App doc: This creates a symlink from ~/webapps/ to the absolute path specified in the extra info field. 

Then in your website mount this at /static


Now requests to static files will be handled directly by webfaction and never reach your app.

web2py with uwsgi

I found Apache used too much memory so changed to uwsgi:

# build uwsgi
hg clone http://projects.unbit.it/hg/uwsgi
cd uwsgi
make -f Makefile.Py27
cd ..

# get web2py
wget http://www.web2py.com/examples/static/web2py_src.zip
unzip web2py_src.zip
cd web2py
# run web2py rocket server to setup password
python2.7 web2py.py
# set your password then kill the server
cp parameters_8080.py parameters_PORT.py
cd ..

# start uwsgi with 32 MB of memory, 24 threads, 1 process

./uwsgi/uwsgi --http=127.0.0.1:PORT --pythonpath /path/web2py --module wsgihandler -d /path/uwsgi.log -t 20 --async 24 --ugreen --limit-as 32 -r --no-orphans -M -p 1 --touch-reload /path/uwsgireload.txt --reload-on-rss 50

# reload server

touch /path/uwsgireload.txt

How to determine what Linux is installed

Kernel version:

$ uname -r
2.6.18-238.12.1.el5PAE


Distribution name:

$ cat /etc/issue
CentOS release 5.7 (Final)


32 or 64 bit:

$ getconf LONG_BIT
32

How to create an image mosaic

For a birthday present I generated an image mosaic using the metapixel command.
Firstly get atleast 1000 images in [base_image_folder] and sub-directories. Then process the data with:  


metapixel-prepare --width=64 --height=96 -r [base_image_folder] [tmp_thumbnails_dir]


This processing will take a while. Then generate the mosaic with:


metapixel -w 48 -h 64 -s 8 --metapixel [image_pattern_for_mosaic] [output_image_file] --library [tmp_thumbnails_dir]


This should only take a few minutes. For different resolutions change the size flag.

Automatically fix rotation of images

My camera stores the orientation of the images in the header, but this is not recognized by most programs.
To automatically fix the image rotations I use:

jhead -autorot

And to recursively rotate all images:

find . -name "*.JPG" -exec jhead -autorot -exonly {} \;

GoDaddy domains with webfaction hosting

In your GoDaddy account select Set Nameservers and then I have specific nameservers for my domains (Documented here)


Enter webfaction's domain name servers (Documented here):
ns1.webfaction.com
ns2.webfaction.com
ns3.webfaction.com
ns4.webfaction.com 


Wait an hour

Downloading data from GAE

As documented here you can download your data into a blob:

appcfg.py download_data --application=[app-id] --filename=[output dump file] --url=http://[app-id].appspot.com/_ah/remote_api

And then upload to another app:
appcfg.py upload_data --application=[app-id] --kind=[kind] --filename=[input dump file]


Or to work with structured data.

First generate the bulkloader file:

appcfg.py create_bulkloader_config --filename=bulkloader.yaml --url=http://[app-id].appspot.com/_ah/remote_api

Set the connectors to CSV

Download the data:

appcfg.py download_data --application=[app-id] --config_file=bulkloader.yaml --filename=[csv file] --db_filename=progress.sql --kind=[which entity]
--db_filename is important for large uploads to avoid re-uploading the same data

Deploying code with rsync

I use rsync to deploy code to my web server:

rsync --recursive --compress \
        --verbose --progress --stats \
        --times --perms --delete \
        --exclude "*~" --exclude "*.pyc"  \
        . login@server:path

This command will transfer the current directory and all sub-directories and files to my server, but only transfer modified files to save time and bandwidth.
verbose, progress, and stats display useful output
times / perms ensure the file timestamps / permissions are the same on my server
delete will delete remote files when deleted locally
exclude temporary files

Convert DVD to AVI

I tried a few programs to burn a DVD and found acidrip works best.

Remember:

• set size to around 1000 for each hour
• turn scale off
• detect the correct crop size
• preview results

Code syntax highlighting

I use this when I need to syntax highlight code snippets:
http://tohtml.com

Writing in Korean

Install ibus, ibus-m17n, ibus-hangul

Restart - I couldn't use ibus in Chrome or choose the Romaji keyboard option until restarting

Ctrl-Space to activate language. Change keyboard to Romaji for English keyboard.

Scim also has a Korean package but I couldn't get it to activate reliably.

Set linux date

Use this graphical tool to set the timezone:

sudo dpkg-reconfigure tzdata

Or set the time directly with:

sudo date MMDDhhmmYYYY

Left hand mouse

One of the buttons on my laptop broke so to avoid that key I switched to using a left hand mouse:

xmodmap -pp
There are 12 pointer buttons defined:

    Physical        Button

     Button          Code

        1              1

        2              2

        3              3

        4              4

        5              5

        6              6

        7              7

        8              8

        9              9

       10             10

       11             11

       12             12

 xmodmap -e "pointer = 3 2 1 4 5 6 7 8 9 11 12 13"

chrome user agent

Chrome doesn't seem to currently have a setting or extension to set the user-agent.

You can however define it with a command line argument:

  chromium-browser --user-agent="richard"

Working with large CSV files

When working with large CSV files I sometimes get this:

Traceback (most recent call last):
  File "stats.py", line 12, in
    for i, row in enumerate(csv.reader(open(filename))):
_csv.Error: field larger than field limit (131072)

This means the CSV file contains more content between delimiters that allowed by default. To raise this limit use:

import sys
import csv
csv.field_size_limit(sys.maxint)

Change keyboards

Change keyboard to DVORAK:

setxkbmap dvorak

And back to QWERTY again:

setxkbmap us

My vimrc

set encoding=utf8

set textwidth=0
set tabstop=4       " Number of spaces to expand tab (messy)
set softtabstop=4   " Number of spaces for insert for tab
set shiftwidth=4    " Number of spaces for << or >>
set expandtab

set ruler
set autoindent
set nocompatible        " use Vim defaults (much better!)
set bs=2                " allow backspacing over everything in insert mode
set ai                  " always set autoindenting on
set vb          " turn off bell
set guifont=Monospace\ 12   " determined with 'set gfn'
syntax on               " switch syntax highlighting on

" searching options
set nowrapscan
set ignorecase

" use ALT key to treat wrapped text as new line when traversing
map gj
map gk
imap gki
imap gji

" disable indentation for pasting with f2
nnoremap :set invpaste paste?
set pastetoggle=
set showmode

" for working with Windows line endings
autocmd BufReadPost * nested
      \ if !exists('b:reload_dos') && !&binary && &ff=='unix' && (0 < search('\r$', 'nc')) |
      \   let b:reload_dos = 1 |
      \   e ++ff=dos |
      \ endif

" Protect large files from sourcing and other overhead.
if !exists("my_auto_commands_loaded")
  let my_auto_commands_loaded = 1
  " Large files are > 10M
  " Set options:
  " eventignore+=FileType (no syntax highlighting etc
  " assumes FileType always on)
  " noswapfile (save copy of file)
  " bufhidden=unload (save memory when other file is viewed)
  " undolevels=-1 (no undo possible)
  let g:LargeFile = 1024 * 1024 * 10
  augroup LargeFile
    autocmd BufReadPre * let f=expand("") | if getfsize(f) > g:LargeFile | set eventignore+=FileType | setlocal noswapfile bufhidden=unload undolevels=-1 | else | set eventignore-=FileType | endif
    augroup END
  endif