Archive for the ‘ development ’ Category

haystack, xapian and virutalenv

Ok, so I finally added the use of virtualenv to my python development process.  The whole isolationist sandbox approach gives me the warm fuzzes.  And now that I’m happy with the approach, I want to extend it into use into production environments I’m deploying.

In the current django project I’m working to deploy, I’m running into some interesting little hoops I have to jump through in order to effectively get the project sandboxed.  One such case is making xapian available to the sandboxed django (through haystack).

Now to give you some background on my production environment, I’m running with a default install of ubuntu, which includes xapian and the bindings specific to the system version of python.  But because I’ve add “–no-site-packages” to my virtualenv, I can’t actually make use of it.  With a little bit of google fu, I was actually able to find two completely separate solutions on the first search page.

With the first solution, I was given a bit of hope that it would be simple.  This mail thread explains that you could just symbolically link the library into your virtualenv.  Ok, simple enough but I just couldn’t get it to work.  No worries, there was another solution to try.

The second solution really is just a script posted to gist.  Simply put, it installs xapian and the bindings into the sandboxed directory for the virtualenv.

Now in my development environment on my laptop, I run xapian (and practically everything else I use for development) like this.  In a production environment though, I try to not install development tools (compilers and such). Following the steps in this script (or running the script itself) means I need to install those tools.  Not quite what I wanted but c’est la vie.

After some modification to the script, I was able to get things up and running.  Django is happy and able to utilize xapian via haystack.

Here’s the modified script I used for review:


export VENV=$VIRTUAL_ENV
mkdir $VENV/packages && cd $VENV/packages
wget http://oligarchy.co.uk/xapian/1.2.7/xapian-core-1.2.7.tar.gz
wget http://oligarchy.co.uk/xapian/1.2.7/xapian-bindings-1.2.7.tar.gz
tar xzvf xapian-core-1.2.7.tar.gz
tar xzvf xapian-bindings-1.2.7.tar.gz
cd $VENV/packages/xapian-core-1.2.7
./configure --prefix=$VENV && make && make install
export LD_LIBRARY_PATH=$VENV/lib
cd $VENV/packages/xapian-bindings-1.2.7
./configure --prefix=$VENV --with-python --without-ruby --without-perl --without-php --without-tcl
make && make install
python -c "import xapian"

2011
11/21
CATEGORY
deployment
development
TAGS




Leave a comment

Rugged Software

Ton of things going on and I’m still trying to get together a couple of Post-Defcon posts on impressions and the app, but I wanted to start touching something that I plan to talk about at length.

Rugged software is on the rise as hot topic.  As a manifesto, it makes some bold statements.  In practical terms, it’s not seen any traction yet.

Right after DC18, I did some digging to find out more and if you look at the Rugged site, there’s not much. Two interesting pieces do come up and I wanted to make sure they get out to the world.  This first is balanced overview of both sides of the Rugged discussion, as of June 2010:

http://www.infoq.com/news/2010/06/rugged-software-manifesto

The second is a recent piece by Mike Dahn on slogans vs practicality:

http://chaordicmind.com/blog/2010/08/07/empty-slogans-and-their-effectiveness-examples-and-statistics/

My initial impression is nice idea, but needs to be applied before we can understand whether this type of approach is effective.

2010
08/10
CATEGORY
development
security
TAGS
Leave a comment

DC18 Update, Data and Bugs

Good News Everyone!

So the app was approved yesterday afternoon and was available in the store by 5pm pst.  So, rock on!

If you’re just downloading the app, then awesome.  You’ll get the updated version (1.1) for dc18.

If you have the dc17 version of the app (1.0 or 1.0.1), then you can just go into the app store and check for updates.

All Your Data…

Make sure to have good network connectivity and do an update of the conference data.  To do this, you need to do the following:

  1. From the defcon 18 menu, navigate to either the Talks (knowledge) or Events lists.
  2. From the top of the list, Pull to Update (like the tweetie app feature).

Give it a minute to actually do the update, since this is a new feature and we’ve not worked out the performance issues yet.

IF YOU FIND A BUG IN THE DATA OR KNOW OF AN UPDATE, direct message @dtjedi or send mail to feedback [at] group6 [dot] net.

Bugs (and Workarounds)

Well, it was bound to happen that some bugs would ship, but I’m a little embarrassed by the number of bugs that got through the development process.  Better planning and more eyes on the app before shipping is still the best bet for finding this stuff early.  I plan to write a post mortem like I did last year, but for now here are the known bugs and their workarounds.

Update Failures

With 1.1, the ability to update data was added to the app.  In the talks or events listings, you can navigate all the way to the top of the list and use “pull to update” functionality to get the latest data.  There are two known issues:

Update Screen Failed Message

If you use the “pull to update” functionality, have network connectivity and the update fails, you will see an update failed screen.  If you navigate away from this screen where the talk or event list should be and then come back, the update failed screen is still there.

Workaround: You need to kill the app and restart.  Bad, but needs to happen until the next update.  NOTE: iOS 4.0 users need to actually kill the app in the multitasking menu.  Otherwise, the app will return to it’s previous state when you launch it.

Update Failure When Offline

If you use the “pull to update” functionality and have no network connectivity (ie, airport mode), the interface will lock up with the list pulled down and the update status message spinning.  The app is locked up and you can’t actually do anything else.

Workaround: You need to kill the app and restart.  NOTE: iOS 4.0 users need to actually kill the app in the multitasking menu.  Otherwise, the app will return to it’s previous state when you launch it.

Cosmetic

If you navigate to a talk detail screen from any list besides the talk list screen, it will display the speakers heading as “organizers”.  No workaround, but will be fixed after con.

 

2010
07/30
CATEGORY
cons
development
iphone
TAGS


Leave a comment

Top Ideas

I’m not typically a fan of Paul Graham.  I think that his writing is extremely inspirational and yet a bit narrow in his views of the startup world.  Occasionally though, I do find that I enjoy his essays.

This one on top ideas hit it out of the park for me.  Having had a couple of stellar startup ideas that never came to fruition, I can totally vouch that these ideas are usually the one that stays on top of everything else in my brain.  But I’ve also fallen victim to the circumstances that prevent them from actually being developed into real projects.

2010
07/26
CATEGORY
development
meta
TAGS

Leave a comment

Catchup….

Going to do some quick catchup here.  Originally, I was going to post on each of these subjects.  But since most of the links are a week old and I want to get them off my plate, it’s going to be a friday cleanup.

Diaspora

There’s been a lot made of the project called Diaspora.  I think between the good marketing and the general distaste for facebook right now, these guys are capitalizing on the right thing at the right time

However, here’s the thing about the project: it’s niche at best.  Facebook may be the biggest privacy disaster we’ve seen, but only a fraction of it’s user base really care about that.  It’s an excellent goal to try and change the way people use social networks (and by extension, the web).  It’s another great goal to help people try and focus enough to care about privacy  The wrong way to do it though is to offer folks an alternative that increases complexity of use.

I know a lot of folks in the security community are excited by the possibility, but most of them have the skill set to tackle the complexity of usage.  The typical facebook user doesn’t.  BTW, I’m not the only one saying this.

Facebook/iPhone Integration

Patently Apple posted the following:

Perhaps “Apple’s new hooks into Facebook,” reported Frommer “will make this sync process part of the phone’s operating system and not just a feature of the Facebook app.” Coincidentally, a recently published Apple patent sheds light on this very subject under the scope of a fuller social networking application concerning various workflows including an “Add Contact” workflow and a “Social Networking” workflow which specifically highlights an exemplary Facebook example.

As a developer, this offers me some interesting possibilities for future apps.  But as a iPhone user, this scares the hell out of me.  With the privacy problems facebook is already dealing with, I’m more than a little concerned with tighter integration between my personal data and my social “network”.

One of my biggest concerns about the android platform is the tight integration with google accounts.  I would never chose to buy an android device AND have it tied to my google account.  It’s hard enough that I’ve made a concession to let them host some of my personal data.  To let facebook onto my iPhone may be a deal breaker for my continued use of it as my personal device.

HTML 5

On a lighter note, check out this very cool use of html 5’s canvas element.  It even loads up on an iPad, though it’s not playable due to gestures not being implemented.

2010
05/21
CATEGORY
development
iphone
privacy
security
TAGS




Leave a comment

fabric and help

After discovering how cool Capistrano is for use with Rails, I went looking for a python equivalent for django.  While it’s not specifically focusing on django, that equivalent is fabric.  It’s decent though it a bit rudimentary in comparison to Capistrano.  Things like only having a single namespace and how you pass arguments to verbs are my biggest complaints.

One of the neat things about python is it’s use of doc strings for generating documentation.  Fabric makes some use of docs strings for outputting help but it again feels a bit clunky.  For example, when you use the -l option on fabric, it lists the command verbs (implemented as functions) along with the first line of the doc string associated with a command.

So:

[host:~/Projects/server] user% fab -l

Available commands:
    clean_db                 Deletes existing db, runs syncdb, adds user. NO...
    clean_up                 Clean up byte compiled files and emacs backups.
    dump_data                Dumps the existing data to a file.

And if you wanted more detail, you’d need to add the -d option for a specific command:

 

[host:~/Projects/server] user% fab -d clean_db
    Displaying detailed information for command 'clean_db':
    Deletes existing db, runs syncdb, adds user. NOTE: DATA LOSS

Call me weird, but I want use the verb help when I’m asking about more details for a command.  But because of the way fabric implements commands, you can’t really just add a help verb to the mix.
So I created the following function (command verb) to get the interface/results I was looking for:
def help(command):
    from fabric.main import display_command

    display_command(command)

Which allows me to do this and get the same results by asking for help:

[host:~/Projects/server] user% fab help:clean_db
    Displaying detailed information for command 'clean_db':
    Deletes existing db, runs syncdb, adds user. NOTE: DATA LOSS

While still a little clunky, it feels more natural to just ask fabric for help this way.

So to sum up, users interfaces are extremely important.  I think we all need to remember that when writing software.  Verbs feel easier to use than command line flags.  Granted, it’s nice for me to be able to hack my own interface for use the way I like, but it’s always just simpler to lead the user down the right path.

 

 

2010
05/20
CATEGORY
development
TAGS




1 comment

More on Migrations

So wanted to post a follow up on the situation I was discussing yesterday with migrations.  I’ve found a simple verbose workaround of sorts.

I’m taking one app in a django project and pushing different areas of responsibility into different apps.  For example, in khanfu people and events are being managed under the same app.  I’m pushing people data models into their own separate app and need to transfer existing data from the one app tables to the other.

To do this, first I copied over the relevant model classes into the new app and ran a schema migration using south.  In order to get the data from the big app to the people app, I decided to write a data migration vs some raw sql.  After some twists and turns around foreign keys and many relation managers, I’ve gotten the data into the new tables. Then I made the change to the old app’s models such that those old tables would be dropped in the migration.

After running through the migration correctly once, I dumped the new state of the db and attempted to refresh a new copy of the db with the data.  By running the migrate command from south, it choked because the model file for the old app could not be use.  Originally, I expected south to just silently fail here but that didn’t happen.

After much meandering and tom foolery, I figured out a work around for the moment to allow me to check in the data migration file and make sure it when migrations are run that it doesn’t break my workflow.  I’ve written a script (after a fashion, really it’s a fabfile) to run the migrations manually in a particular order, including running the data migration as a fake.  This gives me two benefits:

  • Allows me to check in the data migration, meaning I get history for work that was done on the model file.
  • Makes sure I don’t break the workflow of folks I’m working with.  The only caveat here being that I need to make sure to communicate the workflow to the team.

I’m still digging for a good workflow alternative that doesn’t require me to manually manage the migrations (via a script), but nothing’s come up yet.

2010
05/11
CATEGORY
development
TAGS


Leave a comment

The Trials & Tribulations of Migrations

So I’m working on some updates to khanfu.  Initially, it involves a lot of refactoring for the existing project layout in the codebase (a single app for the majority of functionality in the django project).  Given that this will involve a number of database changes, I figured it was time to start using a migration framework for django.

If you check out the django page on schema evolution, you’ll find that there are lot of projects around.  A couple in continued development but most have been abandoned at this point.  After review of the documentation for the active projects, I decided to go with South since it’s the most “Active-Record”-like one available.

After kicking South around for a day, I’ve found that I really like it.  Except… I’m having a hell of a time wrapping my head around some ways to manage data migrations.  Schema migrations feel straight forward and reminds me of rails & active record.  But data migration presents an interesting set of problems for me so far.

South seems to expect that data migrations are only ever going to be run once.  Now the way that I tend to develop with django on new apps is to dump the data from the apps every chance I get and refresh to make sure I’ve not broken anything.  This approach doesn’t work with the way south handles the actual data migrations.

Also,  if data migrations can’t be run more than once, do I need to actually check them in?  I’m a huge fan of tracking all changes, even if they’re short lived.  But with south and these types of migrations, tt feels like you writing run once code that can be thrown in the trash after.  And apparently, the argument could be made for schema changes as well, based upon this thread found on the south mailing list.

So clearly I’ve not gotten my head wrapped around it yet.  I just need read more and look at the way others are using data migrations.

2010
05/11
CATEGORY
development
TAGS




1 comment