In the current django project I’m working to deploy, I’m running into some interesting little hoops I have to jump through in order to effectively get the project sandboxed.  One such case is making xapian available to the sandboxed django (through haystack).

Now to give you some background on my production environment, I’m running with a default install of ubuntu, which includes xapian and the bindings specific to the system version of python.  But because I’ve add “–no-site-packages” to my virtualenv, I can’t actually make use of it.  With a little bit of google fu, I was actually able to find two completely separate solutions on the first search page.

With the first solution, I was given a bit of hope that it would be simple.  This mail thread explains that you could just symbolically link the library into your virtualenv.  Ok, simple enough but I just couldn’t get it to work.  No worries, there was another solution to try.

The second solution really is just a script posted to gist.  Simply put, it installs xapian and the bindings into the sandboxed directory for the virtualenv.

Now in my development environment on my laptop, I run xapian (and practically everything else I use for development) like this.  In a production environment though, I try to not install development tools (compilers and such). Following the steps in this script (or running the script itself) means I need to install those tools.  Not quite what I wanted but c’est la vie.

After some modification to the script, I was able to get things up and running.  Django is happy and able to utilize xapian via haystack.

Here’s the modified script I used for review:


export VENV=$VIRTUAL_ENV
mkdir $VENV/packages && cd $VENV/packages
wget http://oligarchy.co.uk/xapian/1.2.7/xapian-core-1.2.7.tar.gz
wget http://oligarchy.co.uk/xapian/1.2.7/xapian-bindings-1.2.7.tar.gz
tar xzvf xapian-core-1.2.7.tar.gz
tar xzvf xapian-bindings-1.2.7.tar.gz
cd $VENV/packages/xapian-core-1.2.7
./configure --prefix=$VENV && make && make install
export LD_LIBRARY_PATH=$VENV/lib
cd $VENV/packages/xapian-bindings-1.2.7
./configure --prefix=$VENV --with-python --without-ruby --without-perl --without-php --without-tcl
make && make install
python -c "import xapian"


The irony is that it’s been over a year now since I’ve posted anything. It’s not that I don’t have anything to say, it’s just that it’s been an interesting year of things distracting from me actually posting. Honestly I’m not a prolific blogger, but I do have things to say on occasion.

As it turns out, I’ve got a number of drafts sitting in limbo ready to be edited. The plan is to get some of those out to the world and maybe add some new ones.

Let’s see how that goes.

Comment 1: Users don’t have enough foresight or concern to be digging into these problems.  Because the user base is so large on these devices, it becomes next to impossible to educate them all.  Even the most privacy disciplined individuals aren’t focused enough to spend the time focusing on the apps they install on their smart device.

Which leads me to comment 2.  When discussing issues like this, you need to address it with the developers (or company).  The great part about this is that most mobile development shops are just a couple of folks, making initial contact easier.  The tradeoff is that they may not have the resources to immediately address the need.  And in the case of the iTunes app store, there is review time as well.

And comment 3: the idea of putting together a group specifically focused on mobile device app analysis has been floating around in my head for a couple of years.  Given the complexity of these apps, you’d need start with the most popular apps and dig into understand the use cases and trends that developers are exhibiting.  But the idea does have some traction given the concerns of the researchers presenting in that article.

My case in support of this statement is very simple.  In blogging you want to get a point a across that is:

• Clear
• Concise
• Personal

I’m going to admit that I have a problem since I can spend upwards of 3-4 hours writing and rewriting a couple of paragraphs just to get one post out.  I’m completely ineffective at getting exactly the things I’ve outlined above.

Problem is that I’ve got things to say…..

Rugged software is on the rise as hot topic.  As a manifesto, it makes some bold statements.  In practical terms, it’s not seen any traction yet.

Right after DC18, I did some digging to find out more and if you look at the Rugged site, there’s not much. Two interesting pieces do come up and I wanted to make sure they get out to the world.  This first is balanced overview of both sides of the Rugged discussion, as of June 2010:

http://www.infoq.com/news/2010/06/rugged-software-manifesto

http://chaordicmind.com/blog/2010/08/07/empty-slogans-and-their-effectiveness-examples-and-statistics/

My initial impression is nice idea, but needs to be applied before we can understand whether this type of approach is effective.

So the app was approved yesterday afternoon and was available in the store by 5pm pst.  So, rock on!

If you’re just downloading the app, then awesome.  You’ll get the updated version (1.1) for dc18.

If you have the dc17 version of the app (1.0 or 1.0.1), then you can just go into the app store and check for updates.

All Your Data…

Make sure to have good network connectivity and do an update of the conference data.  To do this, you need to do the following:

1. From the defcon 18 menu, navigate to either the Talks (knowledge) or Events lists.
2. From the top of the list, Pull to Update (like the tweetie app feature).

Give it a minute to actually do the update, since this is a new feature and we’ve not worked out the performance issues yet.

IF YOU FIND A BUG IN THE DATA OR KNOW OF AN UPDATE, direct message @dtjedi or send mail to feedback [at] group6 [dot] net.

Bugs (and Workarounds)

Well, it was bound to happen that some bugs would ship, but I’m a little embarrassed by the number of bugs that got through the development process.  Better planning and more eyes on the app before shipping is still the best bet for finding this stuff early.  I plan to write a post mortem like I did last year, but for now here are the known bugs and their workarounds.

Update Failures

With 1.1, the ability to update data was added to the app.  In the talks or events listings, you can navigate all the way to the top of the list and use “pull to update” functionality to get the latest data.  There are two known issues:

Update Screen Failed Message

If you use the “pull to update” functionality, have network connectivity and the update fails, you will see an update failed screen.  If you navigate away from this screen where the talk or event list should be and then come back, the update failed screen is still there.

Workaround: You need to kill the app and restart.  Bad, but needs to happen until the next update.  NOTE: iOS 4.0 users need to actually kill the app in the multitasking menu.  Otherwise, the app will return to it’s previous state when you launch it.

Update Failure When Offline

If you use the “pull to update” functionality and have no network connectivity (ie, airport mode), the interface will lock up with the list pulled down and the update status message spinning.  The app is locked up and you can’t actually do anything else.

Workaround: You need to kill the app and restart.  NOTE: iOS 4.0 users need to actually kill the app in the multitasking menu.  Otherwise, the app will return to it’s previous state when you launch it.

Cosmetic

If you navigate to a talk detail screen from any list besides the talk list screen, it will display the speakers heading as “organizers”.  No workaround, but will be fixed after con.

This one on top ideas hit it out of the park for me.  Having had a couple of stellar startup ideas that never came to fruition, I can totally vouch that these ideas are usually the one that stays on top of everything else in my brain.  But I’ve also fallen victim to the circumstances that prevent them from actually being developed into real projects.

Two key passages that I find interesting are:

Tim Bray, Google’s Android Evangelist, went off on a rather enthusiastic but somewhat silly Twitter rant a few days ago about openness and the “curated experience” of the iPhone. It’s clear that Google sees “openness” as a competitive advantage over Apple and has made it their battle cry in the mobile space.

And:

Here’s the reality of the Android situation now: if you buy an Android phone, it will most likely be locked down by your carrier, possibly also with some features disabled. Or, to use Tim Bray’s term, the reality is that most Android phones that get bought are a “curated experience”. 

At one point late last year, I briefly had a Nexus One in my possession and I absolutely hated it.  The user experience was horrible.  The interface was like using a 1990′s linux window manager and 90% of the apps that I tried crashed within the first minute of usage.

More recently, some friends have picked up one of the HTC android phones with verizon service.  When I had a chance to play with one of these phones, I noted the fact that the user experience had greatly improved.  This was in no small part due to the changes that verizon had made to the default android user interface, providing a better user experience.

If android phones are going to the “curated” experience as described above, I think it’s only a winning situation for android users and google.

Diaspora

There’s been a lot made of the project called Diaspora.  I think between the good marketing and the general distaste for facebook right now, these guys are capitalizing on the right thing at the right time

However, here’s the thing about the project: it’s niche at best.  Facebook may be the biggest privacy disaster we’ve seen, but only a fraction of it’s user base really care about that.  It’s an excellent goal to try and change the way people use social networks (and by extension, the web).  It’s another great goal to help people try and focus enough to care about privacy  The wrong way to do it though is to offer folks an alternative that increases complexity of use.

I know a lot of folks in the security community are excited by the possibility, but most of them have the skill set to tackle the complexity of usage.  The typical facebook user doesn’t.  BTW, I’m not the only one saying this.

Facebook/iPhone Integration

Patently Apple posted the following:

Perhaps “Apple’s new hooks into Facebook,” reported Frommer “will make this sync process part of the phone’s operating system and not just a feature of the Facebook app.” Coincidentally, a recently published Apple patent sheds light on this very subject under the scope of a fuller social networking application concerning various workflows including an “Add Contact” workflow and a “Social Networking” workflow which specifically highlights an exemplary Facebook example.

As a developer, this offers me some interesting possibilities for future apps.  But as a iPhone user, this scares the hell out of me.  With the privacy problems facebook is already dealing with, I’m more than a little concerned with tighter integration between my personal data and my social “network”.

One of my biggest concerns about the android platform is the tight integration with google accounts.  I would never chose to buy an android device AND have it tied to my google account.  It’s hard enough that I’ve made a concession to let them host some of my personal data.  To let facebook onto my iPhone may be a deal breaker for my continued use of it as my personal device.

HTML 5

On a lighter note, check out this very cool use of html 5′s canvas element.  It even loads up on an iPad, though it’s not playable due to gestures not being implemented.

One of the neat things about python is it’s use of doc strings for generating documentation.  Fabric makes some use of docs strings for outputting help but it again feels a bit clunky.  For example, when you use the -l option on fabric, it lists the command verbs (implemented as functions) along with the first line of the doc string associated with a command.

So:

[host:~/Projects/server] user% fab -l

Available commands:
    clean_db                 Deletes existing db, runs syncdb, adds user. NO...
    clean_up                 Clean up byte compiled files and emacs backups.
    dump_data                Dumps the existing data to a file.

And if you wanted more detail, you’d need to add the -d option for a specific command:

[host:~/Projects/server] user% fab -d clean_db
    Displaying detailed information for command 'clean_db':
    Deletes existing db, runs syncdb, adds user. NOTE: DATA LOSS

def help(command):
    from fabric.main import display_command

[host:~/Projects/server] user% fab help:clean_db
    Displaying detailed information for command 'clean_db':
    Deletes existing db, runs syncdb, adds user. NOTE: DATA LOSS